Spark Streaming实时流项目实战笔记十

最新推荐文章于 2024-07-18 06:29:40 发布

huxin9611

最新推荐文章于 2024-07-18 06:29:40 发布

阅读量185

点赞数

CC 4.0 BY-SA版权

分类专栏：学习笔记文章标签： spark streaming

本文链接：https://blog.youkuaiyun.com/huxin9611/article/details/89880045

学习笔记专栏收录该内容

19 篇文章

订阅专栏

本文介绍了Spark Streaming整合Kafka的实战内容，包含Receiver-based和Direct-Approach两种方式。Receiver-based方式需先启动zookeeper和kafka、创建topic，还给出了生产和消费信息的测试命令及spark-submit命令；Direct-Approach提及了Kafka版本选择。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Spark Streaming整合Kafka实战

实战一：Receiver-based

1）先启动zookeeper
2）启动kafka

[hadoop@hadoop000 bin]$ ./kafka-server-start.sh -daemon /home/hadoop/app/kafka_2.11-0.9.0.0/config/server
server-1.properties   server-3.properties   server.properties~
server-2.properties   server-3.properties~  
server-2.properties~  server.properties     
[hadoop@hadoop000 bin]$ ./kafka-server-start.sh -daemon /home/hadoop/app/kafka_2.11-0.9.0.0/config/server.properties
[hadoop@hadoop000 bin]$ jps

3）创建topic

[hadoop@hadoop000 bin]$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic  kafka_streaming_topic
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "kafka_streaming_topic".


[hadoop@hadoop000 bin]$ ./kafka-topics.sh --list --zookeeper localhost:2181
hello_topic
kafka_streaming_topic
my-replicated-topic

4）通过控制台测试本控制台是否能够正常的生产和消费信息

./kafka-console-producer.sh --broker-list localhost:9092 --topic kafka_streaming_topic

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafka_streaming_topic

spark-submit
–class com.imooc.spark.KafkaReceiverWordCount
–master local[2]
–name KafkaReceiverWordCount
–packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0
/home/hadoop/lib/sparktrain-1.0.jar hadoop000:2181 test kafka_streaming_topic 1