2021-08-30 Could not get block locations. Source file XXXXXX

枪枪枪

已于 2023-01-16 22:55:33 修改

阅读量2.8k

点赞数

分类专栏： Spark Hive 文章标签： scala spark big data

于 2021-09-02 14:22:40 首次发布

本文链接：https://blog.youkuaiyun.com/az9996/article/details/120005539

版权

Spark 同时被 2 个专栏收录

35 篇文章

订阅专栏

Hive

8 篇文章

订阅专栏

博客内容描述了一次Spark作业在运行时遇到的问题，具体表现为Sparkjob在处理Hive查询时因stage失败而终止。错误源于特定分区的任务无法在任何节点上运行，导致TaskSet被黑listed。问题的根本原因是一个HiveMapRecordHandler在处理row时抛出了HiveException，进一步追踪发现是由于尝试获取文件块位置时发生了IOException。解决方案是通过调整配置参数，如增大mapred.task.timeout，以避免任务超时，并重新提交任务，最终任务成功执行。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Query Hive on Spark job[0] stages: [0]
Spark job[0] status = RUNNING
--------------------------------------------------------------------------------------
          STAGES   ATTEMPT        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED
--------------------------------------------------------------------------------------
Stage-0                  0       RUNNING  12438         37      100    12301      70
--------------------------------------------------------------------------------------
STAGES: 00/01    [>>--------------------------] 0%    ELAPSED TIME: 989.76 s
--------------------------------------------------------------------------------------
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to: Job aborted due to stage failure:
Aborting TaskSet 0.0 because task 52 (partition 52)
cannot run anywhere due to node and executor blacklist.
Most recent failure:
Lost task 52.1 in stage 0.0 (TID 168, a.b, executor 11): java.lang.RuntimeException: Error processing row: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"row_key":"0197009c7e571a001d28000","zdate":"2021-07-26 00:29:40","vid":81557,"lat":0,"lon":0,"spd":0,"dir":0,"alt":0,"zstate":786432,"alarm":0,"mile":42949671,"fuel":0,"wireless_signal_state":23,"gnss_num":0,"total_fuel":0,"statellite_infor":null,"h_alt":0,"h_speed":0,"h_dir":0,"h_time":0,"h_lat":0,"h_lon":0}
        at org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:146)
        at org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
        at org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
        at org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
        at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$12.apply(AsyncRDDActions.scala:127)
        at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$12.apply(AsyncRDDActions.scala:127)
        at org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2232)
        at org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:2232)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:121)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"row_key":"0197009c7e571a001d28000","zdate":"2021-07-26 00:29:40","vid":81557,"lat":0,"lon":0,"spd":0,"dir":0,"alt":0,"zstate":786432,"alarm":0,"mile":42949671,"fuel":0,"wireless_signal_state":23,"gnss_num":0,"total_fuel":0,"statellite_infor":null,"h_alt":0,"h_speed":0,"h_dir":0,"h_time":0,"h_lat":0,"h_lon":0}
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:494)
        at org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:133)
        ... 18 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Could not get block locations. Source file "test.demo1.hive-staging_hive_2021-08-30_19-16-24_549_3376039352418297486-1/_task_tmp.-ext-10002/zday=2021-07-26/_tmp.000052_1" - Aborting...block==null
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:803)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882)
        at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
        at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:146)
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:484)
        ... 19 more
Caused by: java.io.IOException: Could not get block locations. Source file "test.demo1.hive-staging_hive_2021-08-30_19-16-24_549_3376039352418297486-1/_task_tmp.-ext-10002/zday=2021-07-26/_tmp.000052_1" - Aborting...block==null
        at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1477)
        at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
        at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)


Blacklisting behavior can be configured via spark.blacklist.*.

参考资料：
https://blog.youkuaiyun.com/lookqlp/article/details/88336851

mapred.task.timeout 200000 The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string.

set mapred.task.timeout=6000000;

暂且先修改资料中提到的参数后重新提交任务

任务顺利执行，成功