java.lang.NoClassDefFoundError:org/apache/kafka/common/message/KafkaKZ4BlockOutStream

问题描述：

在用sparkStreaming消费Kafka时，时不时的在几个batch任务中报错误java.lang.NoClassDefFoundError:org/apache/kafka/common/message/KafkaKZ4BlockOutStream，导致sparkStreaming程序异常终止，就算重启也不行，必须删除checkpoint，然后重启，但是等消费到了这个offset时，又会报同样的错误，导致任务失败（本地自己电脑上跑没问题，提交到集群上就会报此错）。我的环境参数如下：

<scala.version>2.11.11</scala.version>
<spark.version>2.2.0</spark.version>
<scala.binary.version>2.11</scala.binary.version>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_${scala.binary.version}</artifactId>
      <version>${spark.version}</version>
      <scope>${scopetype}</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_${scala.binary.version}</artifactId>
      <version>${spark.version}</version>
      <scope>${scopetype}</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_${scala.binary.version}</artifactId>
      <version>${spark.version}</version>
      <scope>${scopetype}</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_${scala.binary.version}</artifactId>
      <version>${spark.version}</version>
      <scope>${scopetype}</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka-0-8_${scala.binary.version}</artifactId>
      <version>${spark.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.11</artifactId>
      <version>0.8.2.1</version>
    </dependency>

解决办法：

添加jar包kafka-clients-0.8.2.1.jar即可，不同的环境可能解决办法不同！！！！！！！！！！

jar包链接：https://pan.baidu.com/s/1vjPmomdp6I2QQ9S0ZpPAMQ 
提取码：lubx