Spark数据写入Kudu，报Caused by: java.lang.RuntimeException: PendingErrors overflowed. Failed to write at..

最新推荐文章于 2024-06-20 08:04:52 发布

原创

最新推荐文章于 2024-06-20 08:04:52 发布 · 1.3k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#spark

本文记录了一次使用Apache Spark向Kudu表写入数据时遇到的错误，并详细分析了错误原因，涉及到的问题包括写入超时和服务不可用等。通过重启Kudu节点解决了问题。

错误如下：

2021-02-01 17:11:13 ERROR TaskSetManager:73 - Task 0 in stage 4.0 failed 1 times; aborting job
2021-02-01 17:11:13 INFO  TaskSchedulerImpl:57 - Removed TaskSet 4.0, whose tasks have all completed, from pool 
2021-02-01 17:11:13 INFO  TaskSchedulerImpl:57 - Cancelling stage 4
2021-02-01 17:11:13 INFO  TaskSchedulerImpl:57 - Killing all running tasks in stage 4: Stage cancelled
2021-02-01 17:11:13 INFO  DAGScheduler:57 - ResultStage 4 (foreachPartition at KuduContext.scala:350) failed in 141.587 s due to Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4, localhost, executor driver): java.lang.RuntimeException: PendingErrors overflowed. Failed to write at least 1000 rows to Kudu; Sample errors: Timed out: cannot complete before timeout: Batch{operations=256, tablet="4a02e65bac264694b14faeee409987d1" [0x00000002, 0x00000003), ignoredErrors=[], rpc=KuduRpc(method=Write, tablet=4a02e65bac264694b14faeee409987d1, attempt=23, TimeoutTracker(timeout=30000, elapsed=28936), Trace Summary(28936 ms): Sent(23), Received(23), Delayed(23), MasterRefresh(0), AuthRefresh(0), Truncated: false
 Sent: (49d6bbea2c404778a65f3738a72c6ae9, [ Write, 23 ])
 Received: (49d6bbea2c404778a65f3738a72c6ae9, [ SERVICE_UNAVAILABLE, 23 ])
 Delayed: (UNKNOWN, [ Write, 23 ]))}Timed out: cannot complete before timeout: Batch{operations=256, tablet="4a02e65bac264694b14faeee409987d1" [0x00000002, 0x00000003), ignoredErrors=[], rpc=KuduRpc(method=Write, tablet=4a02e65bac264694b14faeee409987d1, attempt=23, TimeoutTracker(timeout=30000, elapsed=28936), Trace Summary(28936 ms): Sent(23), Received(23), Delayed(23), MasterRefresh(0), AuthRefresh(0), Truncated: false
 Sent: (49d6bbea2c404778a65f3738a72c6ae9, [ Write, 23 ])
 Received: (49d6bbea2c404778a65f3738a72c6ae9, [ SERVICE_UNAVAILABLE, 23 ])
 Delayed: (UNKNOWN, [ Write, 23 ]))}Timed out: cannot complete before timeout: Batch{operations=256, tablet="4a02e65bac264694b14faeee409987d1" [0x00000002, 0x00000003), ignoredErrors=[], rpc=KuduRpc(method=Write, tablet=4a02e65bac264694b14faeee409987d1, attempt=23, TimeoutTracker(timeout=30000, elapsed=28936), Trace Summary(28936 ms): Sent(23), Received(23), Delayed(23), MasterRefresh(0), AuthRefresh(0), Truncated: false
 Sent: (49d6bbea2c404778a65f3738a72c6ae9, [ Write, 23 ])
 Received: (49d6bbea2c404778a65f3738a72c6ae9, [ SERVICE_UNAVAILABLE, 23 ])
 Delayed: (UNKNOWN, [ Write, 23 ]))}Timed out: cannot complete before timeout: Batch{operations=256, tablet="4a02e65bac264694b14faeee409987d1" [0x00000002, 0x00000003), ignoredErrors=[], rpc=KuduRpc(method=Write, tablet=4a02e65bac264694b14faeee409987d1, attempt=23, TimeoutTracker(timeout=30000, elapsed=28936), Trace Summary(28936 ms): Sent(23), Received(23), Delayed(23), MasterRefresh(0), AuthRefresh(0), Truncated: false
 Sent: (49d6bbea2c404778a65f3738a72c6ae9, [ Write, 23 ])
 Received: (49d6bbea2c404778a65f3738a72c6ae9, [ SERVICE_UNAVAILABLE, 23 ])
 Delayed: (UNKNOWN, [ Write, 23 ]))}Timed out: cannot complete before timeout: Batch{operations=256, tablet="4a02e65bac264694b14faeee409987d1" [0x00000002, 0x00000003), ignoredErrors=[], rpc=KuduRpc(method=Write, tablet=4a02e65bac264694b14faeee409987d1, attempt=23, TimeoutTracker(timeout=30000, elapsed=28936), Trace Su