hive写入ES报错 FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

关注

原创最新推荐文章于 2026-03-30 09:33:32 发布 1.3k 阅读 AI 写同款· GEO 优化 ›

使用hive报 return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask解决方法 1.情况两表join 向新表中插入数据时导入其他分区的的数据时正常唯独0701这一天的数据异常 insert overwrite table A partition (log_date= '20210701') select * from employees e left join b where a.log_date= '20210701' and b.log_date= '20210701' 2.分析 3.解决 ... 阅读详情

HIVE整合ES场景：

ES版本：6.5.2

hive版本：2.3.4

插件包下载地址：https://repo.maven.apache.org/maven2/org/elasticsearch/elasticsearch-hadoop-hive/

整合过程网上很多资料，下载与自己ES版本匹配的插件包，然后将elasticsearch-hadoop-hive-6.5.2.jar拷贝到hive安装目录lib下。因为我的hive下已经有

commons-httpclient-3.0.1.jar文件了，所以不用做重复拷贝了。

执行hivesql将表数据写入到elasticsearch报错如下：

2020-05-30 14:08:12】org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:379)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257)
	at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

这里日志显示的不是很明确，到yarn管理页面找到具体的MR任务看日志报找不到es的类错误，日志如下：

Caused by: java.lang.ClassNotFoundException: org.elasticsearch.hadoop.hive.EsHiveInputFormat
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
	... 60 more

2020-05-30 14:56:01,845 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1590680711550_0090_m_000000_0: Error: java.lang.RuntimeException: Failed to load plan: hdfs://SINOTRUK/tmp/hive/hive/c7f92979-e7cc-4a39-beba-d8007ddcc817/hive_2020-05-30_14-55-44_617_4414312671758408146-2/-mr-10002/7db2e153-f315-4f63-9886-0774d2f85bf0/map.xml
	at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:481)
	at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:313)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:394)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:665)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:658)
	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:692)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:175)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:444)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: org.elasticsearch.hadoop.hive.EsHiveInputFormat

因为是执行的是mapreduce任务，显然只是将（elasticsearch-hadoop-hive-6.5.2.jar）和（commons-httpclient-3.0.1.jar）包放在某一个节点是不行的。所以需要将（elasticsearch-hadoop-hive-6.5.2.jar）和（commons-httpclient-3.0.1.jar）包拷贝到每个hadoop安装目录的lib下。重启hadoop和hive即可。

如果没有拷贝（commons-httpclient-3.0.1.jar）包，执行任务会报如下错误：

2020-05-30 15:22:51,047 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1590680711550_0092_m_000000_1: Error: java.lang.ClassNotFoundException: org.apache.commons.httpclient.Credentials
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransportFactory.create(CommonsHttpTransportFactory.java:40)
	at org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:102)
	at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:85)
	at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:61)
	at org.elasticsearch.hadoop.rest.RestClient.<init>(RestClient.java:94)
	at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:615)
	at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173)
	at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:58)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:762)
	at org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:101)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
	at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:137)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
	at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.closeOp(VectorMapOperator.java:900)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

总结：因为hivesql提交后，yarn会把MR任务分发到某个节点执行，如果该节点的hadoop下没有（elasticsearch-hadoop-hive-6.5.2.jar）和（commons-httpclient-3.0.1.jar）这两个包就会抛出异常。

建议：将（elasticsearch-hadoop-hive-6.5.2.jar）和（commons-httpclient-3.0.1.jar）这两个包拷贝到集群的hadoop安装目录的lib下和hive安装目录的lib下。

收录于