运行hadoop任务报错org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource requ-优快云博客

本文链接：https://blog.youkuaiyun.com/luxiangzhou/article/details/82012104

本文详细介绍了在YARN集群上调整应用内存配置的过程，解决因虚拟机内存限制导致的任务失败问题。通过修改yarn-site.xml文件中的yarn.app.mapreduce.am.resource.mb参数，将默认的1.5GB内存需求降低至更适配的大小，确保任务能在资源受限的环境下正常运行。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

yarn-site.xml文件中加上

<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>200</value>
</property>

原因：yarn默认是设置1.5G内存来运行任务，但是虚拟机内存太小，没有1.5G，要改小运行任务内存

[root@s100 lxz]# hadoop jar hadoopmaven-0.0.1-SNAPSHOT.jar com.lxz.hadoopmaven.mr.MyMaxTempApp /user/sample.txt /user/sample_out.txt
18/08/22 16:41:07 INFO client.RMProxy: Connecting to ResourceManager at s100/192.168.88.100:8032
18/08/22 16:41:14 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/08/22 16:41:18 INFO input.FileInputFormat: Total input paths to process : 1
18/08/22 16:41:18 INFO mapreduce.JobSubmitter: number of splits:1
18/08/22 16:41:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1534842402074_0001
18/08/22 16:41:24 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1534842402074_0001
Exception in thread "main" java.io.IOException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=1536, maxMemory=1024
   at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:272)
   at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228)
   at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:236)
   at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
   at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:330)
   at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:282)
   at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:580)
   at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218)
   at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419)
   at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)

   at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316)
   at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)