spark解决 org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow

最新推荐文章于 2025-07-13 14:57:08 发布

原创

最新推荐文章于 2025-07-13 14:57:08 发布 · 1.3w 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#spark #buffer #Kryo #jdbc #thrift

在使用Spark SQL的Thrift JDBC接口查询数据时遇到`org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow`错误。为解决此问题，需要调整`spark.kryoserializer.buffer.max`参数，至少设置为3236381。尝试在`spark-defaults.conf`文件中配置未成功，因为Spark SQL Thrift JDBC不受该配置影响。通过`spark-shell`测试发现配置生效，于是采用启动参数方式解决，启动命令添加相关参数后问题得到解决。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

使用spark sql 的 thrift jdbc接口查询数据时报这个错误

Exception in thread "main" java.sql.SQLException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3107 in stage 308.0 failed 4 times, most recent failure: Lost task 3107.3 in stage 308.0 (TID 620318, XXX): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 1572864, required: 3236381
Serialization trace:
values (org.apache.spark.sql.catalyst.expressions.GenericInternalRow). To avoid this, increase spark.kryoserializer.buffer.max value.
        at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:299)
        a