spark报错整理

ClassNotFound

如果报ClassNotFound，一般是缺少相应的jar包

比如：

Error: Could not find or load main class NodeManager

CLASSPATH应该包括所有hadoop的jar包(或者写入hadoop-env.sh)

Caused by: java.lang.ClassNotFoundException:org.I0Itec.zkclient.serialize.ZkSerializer

缺zkclient-0.3.jar

 

Too many open files

java.net.SocketException:Too many open files

一般是服务器的最大连接数限制了（linux默认是1024）。用ulimit -n 可以查看配置，如果值太小，就修改/etc/security/limits.conf文件，增加* - nofile 10240

修改机器（linux）的最大连接数

 

OutOfMemoryError

java.lang.OutOfMemoryError:unable to create new native thread

查看applicationMaster所在的节点，内存使用率很高。所以换一个节点启动任务，一般在哪提交任务，这个任务的AM就在哪个节点上

 

Container expired since it was unused

[2016-09-16 15:28:54,081] WARN Container marked as failed: container_e09_1473933378250_0012_01_000012 on host: SCSV00461. Exit status: -100. Diagnostics: Container expired since it was unused (org.apache.spark.deploy.yarn.YarnAllocator)

端口8042防火墙没开放