spark-shell出错_解决方法

最新推荐文章于 2025-11-03 13:02:57 发布

原创最新推荐文章于 2025-11-03 13:02:57 发布 · 4.9k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#spark

spark 专栏收录该内容

57 篇文章

订阅专栏

本文介绍了在Spark集群中遇到的问题及解决方案，特别是在node1节点启动spark-shell时出现的错误。通过调整spark-defaults.conf配置文件，包括设置认证、动态分配、序列化方式等参数，最终解决了问题并实现了集群的正常运行。

输入spark-shell出错（node1节点出错，其他节点正常），出错信息如下：
这里写图片描述
修改配置文件spark-default.conf

修改内容如下

spark.authenticate=false
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.executorIdleTimeout=60
spark.dynamicAllocation.minExecutors=0
spark.dynamicAllocation.schedulerBacklogTimeout=1
spark.eventLog.dir=hdfs://node1:8020/user/spark/applicationHistory
spark.eventLog.enabled=true
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.shuffle.service.enabled=true
spark.shuffle.service.port=7337
spark.executorEnv.PYTHONPATH=/usr/local/bin/python2.7
spark.yarn.historyServer.address=http://node1:18088
spark.yarn.jar=local:/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/spark/lib/spark-assembly.jar
spark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop/lib/native
spark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop/lib/native
spark.yarn.am.extraLibraryPath=/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop/lib/native
spark.yarn.config.gatewayPath=/opt/cloudera/parcels
spark.yarn.config.replacementPath={{HADOOP_COMMON_HOME}}/../../..
spark.master=yarn-client
#spark.master=spark://node1:7077