关于hive执行mr时报错Call From hadoop01/192.168.80.128 to 0.0.0.0:10020 failed on connection exception

今天在用hive执行mr时启用42个job，执行到一半的时候突然报了以下的错，以前也没遇到过，不知道是不是job数量太多的原因。错误提示是无法从主机访问到10020端口。

上网查了一下原因：这是因为DataNode 需要访问 MapReduce JobHistory Server，它的默认端口为0.0.0.0:10020 。

报错信息：

java.io.IOException: java.net.ConnectException: Call From hadoop01/192.168.80.128 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:343)
	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:428)
	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:568)
	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:323)
	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:320)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
	at org.apache.hadoop.mapreduce.Job.getStatus(Job.java:338)
	at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:603)
	at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:631)
	at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:289)
	at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549)
	at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437)
	at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1650)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1409)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1192)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

 

解决方法：

1）在/etc/hadoop/mapred-site.xml配置文件中开启jobhistory

<property>

        <name>mapreduce.jobhistory.address</name>

        <!-- 配置实际的主机名和端口-->

        <value>hadoop01:10020</value>

 </property>

 

2）并增加historyserver的heap 的大小，根据具体情况而定。 

在配置文件${HADOOP_DIR}/etc/hadoop/mapred-env.sh下：

export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=2000

 

3）开启historyserver

[root@hadoop01 sbin]# mr-jobhistory-daemon.sh start historyserver