sparkstreaming整合kafka运行报错

最新推荐文章于 2023-02-21 11:26:01 发布

原创最新推荐文章于 2023-02-21 11:26:01 发布 · 969 阅读

0 ·

CC 4.0 BY-SA版权

报错专栏收录该内容

4 篇文章

订阅专栏

在尝试将SparkStreaming与Kafka整合时，遇到了'java.lang.NoSuchMethodError'异常，具体错误指向io.netty.buffer.PooledByteBufAllocator.defaultNumHeapArena()I方法。错误出现在Spark的NettyRpcEnv初始化过程中，可能与Netty版本冲突或依赖库不兼容有关。

20/11/30 14:24:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(flashtown); groups with view permissions: Set(); users with modify permissions: Set(flashtown); groups with modify permissions: Set()
Exception in thread "main" java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.defaultNumHeapArena()I
   at org.apache.spark.network.util.NettyUtils.createPooledByteBufAllocator(NettyUtils.java:113)
   at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:106)
   at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
   at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:71)
   at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461)
   at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
   at org.apache.spark.SparkEnv$.create(SparkEnv.scala:249)
   at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
   at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
   at org.apache.spark.SparkContext.<init>(SparkContext.scala:424)
   at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:838)
   at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
   at _04StreamingReadFromKafkaOps$.main(_04StreamingReadFromKafkaOps.scala:22)
   at _04StreamingReadFromKafkaOps.main(_04StreamingReadFromKafkaOps.scala)
20/11/30 14:24:49 INFO ShutdownHookManager: Shutdown hook called

Process finished with exit code 1