yarn.server.nodemanager.security.NMContainerTokenSecretManager.retrievePassword

在新搭建Hadoop2.6.0集群上运行自己写的程序报以下错误：


hadoop@master:/opt/hadoop$ bin/hadoop jar /home/hadoop/桌面/GRbyMDAS.jar Process
15/04/07 20:20:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/07 20:20:00 INFO pack.MR2: outMR1文件夹存在，已执行删除操作！
15/04/07 20:20:01 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.11:8032
15/04/07 20:20:01 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/04/07 20:20:07 INFO input.FileInputFormat: Total input paths to process : 1
15/04/07 20:20:07 INFO mapreduce.JobSubmitter: number of splits:2
15/04/07 20:20:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1428406153917_0003
15/04/07 20:20:07 INFO impl.YarnClientImpl: Submitted application application_1428406153917_0003
15/04/07 20:20:07 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1428406153917_0003/
15/04/07 20:20:07 INFO mapreduce.Job: Running job: job_1428406153917_0003
15/04/07 20:20:12 INFO mapreduce.Job: Job job_1428406153917_0003 running in uber mode : false
15/04/07 20:20:12 INFO mapreduce.Job:  map 0% reduce 0%
15/04/07 20:20:13 INFO mapreduce.Job: Task Id : attempt_1428406153917_0003_m_000000_1000, Status : FAILED
Container launch failed for container_1428406153917_0003_02_000002 : java.lang.NullPointerException: java.lang.NullPointerException
    at org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager.retrievePassword(NMContainerTokenSecretManager.java:155)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.verifyAndGetContainerTokenIdentifier(ContainerManagerImpl.java:882)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:709)
    at org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
    at org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)    
        ......

再运行示例程序计算PI,也报错。

hadoop@master:/opt/hadoop$ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 10 10
Number of Maps  = 10
Samples per Map = 10
15/04/07 20:38:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
15/04/07 20:38:25 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.11:8032
15/04/07 20:38:26 INFO input.FileInputFormat: Total input paths to process : 10
15/04/07 20:38:26 INFO mapreduce.JobSubmitter: number of splits:10
15/04/07 20:38:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1428406153917_0004
15/04/07 20:38:26 INFO impl.YarnClientImpl: Submitted application application_1428406153917_0004
15/04/07 20:38:26 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1428406153917_0004/
15/04/07 20:38:26 INFO mapreduce.Job: Running job: job_1428406153917_0004
15/04/07 20:38:31 INFO mapreduce.Job: Job job_1428406153917_0004 running in uber mode : false
15/04/07 20:38:31 INFO mapreduce.Job:  map 0% reduce 0%
15/04/07 20:38:32 INFO mapreduce.Job: Task Id : attempt_1428406153917_0004_m_000009_0, Status : FAILED
Container launch failed for container_1428406153917_0004_01_000011 : java.lang.NullPointerException: java.lang.NullPointerException
    at org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager.retrievePassword(NMContainerTokenSecretManager.java:155)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.verifyAndGetContainerTokenIdentifier(ContainerManagerImpl.java:882)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:709)
    at org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
    at org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
        ......

经过一番调查发现是因为配置有误导致错误。

Hadoop2.6.0的yarn-default.xml里面的yarn.nodemanager.resource.memory-mb（Amount of physical memory, in MB, that can be allocated for containers）默认的value是8192，即8GB，而我的集群节点内存只有4个GB，所以要在yarn-site.xml中配置yarn.nodemanager.resource.memory-mb的值为4096.