kafka0.8和1.0整合sparkstreaming

最新推荐文章于 2020-09-09 19:54:15 发布

转载最新推荐文章于 2020-09-09 19:54:15 发布 · 768 阅读

文章标签：

#kafka

kafka 专栏收录该内容

15 篇文章

订阅专栏

本文介绍如何使用 Apache Spark Streaming 与 Apache Kafka 集成进行实时数据处理。详细对比了针对不同 Kafka 版本（0.8 和 0.10）的 Spark Streaming 集成方案，并提供了各个方案的特性比较，如 API 稳定性、语言支持、SSL/TLS 支持等。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Spark Streaming + Kafka Integration Guide

Apache Kafka is publish-subscribe messaging rethought as a distributed, partitioned, replicated commit log service. Please read the Kafka documentation thoroughly before starting an integration using Spark.

The Kafka project introduced a new consumer api between versions 0.8 and 0.10, so there are 2 separate corresponding Spark Streaming packages available. Please choose the correct package for your brokers and desired features; note that the 0.8 integration is compatible with later 0.9 and 0.10 brokers, but the 0.10 integration is not compatible with earlier brokers.

	spark-streaming-kafka-0-8	spark-streaming-kafka-0-10
Broker Version	0.8.2.1 or higher	0.10.0 or higher
Api Stability	Stable	Experimental
Language Support	Scala, Java, Python	Scala, Java
Receiver DStream	Yes	No
Direct DStream	Yes	Yes
SSL / TLS Support	No	Yes
Offset Commit Api	No	Yes
Dynamic Topic Subscription	No	Yes