hive执行任务报错Execution failed with exit status: 3

最新推荐文章于 2022-12-11 09:32:07 发布

转载最新推荐文章于 2022-12-11 09:32:07 发布 · 343 阅读

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/weiqiang-liu/p/4691675.html

文章标签：

本文记录了一次使用Hive进行MapJoin操作的过程，详细展示了随着处理行数的增加，哈希表大小及内存使用情况的变化，并提出了通过设置Hive参数来避免自动转换MapJoin的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

2014-12-18 03:21:03 Starting to launch local task to process map join; maximum memory = 1065484288

2014-12-18 03:21:08 Processing rows: 200000 Hashtable size: 199999 Memory usage: 112049704 percentage: 0.105

2014-12-18 03:21:09 Processing rows: 300000 Hashtable size: 299999 Memory usage: 160367688 percentage: 0.151

2014-12-18 03:21:10 Processing rows: 400000 Hashtable size: 399999 Memory usage: 209294088 percentage: 0.196

2014-12-18 03:21:11 Processing rows: 500000 Hashtable size: 499999 Memory usage: 257089944 percentage: 0.241

2014-12-18 03:21:12 Processing rows: 600000 Hashtable size: 599999 Memory usage: 305440536 percentage: 0.287

2014-12-18 03:21:14 Processing rows: 700000 Hashtable size: 699999 Memory usage: 347305664 percentage: 0.326

2014-12-18 03:21:14 Processing rows: 800000 Hashtable size: 799999 Memory usage: 403916624 percentage: 0.379

2014-12-18 03:21:16 Processing rows: 900000 Hashtable size: 899999 Memory usage: 452238592 percentage: 0.424

2014-12-18 03:21:16 Processing rows: 1000000 Hashtable size: 999999 Memory usage: 499593552 percentage: 0.469

2014-12-18 03:21:18 Processing rows: 1100000 Hashtable size: 1099999 Memory usage: 547966320 percentage: 0.514

2014-12-18 03:21:19 Processing rows: 1200000 Hashtable size: 1199999 Memory usage: 593792800 percentage: 0.557

2014-12-18 03:21:21 Processing rows: 1300000 Hashtable size: 1299999 Memory usage: 641564688 percentage: 0.602

2014-12-18 03:21:21 Processing rows: 1400000 Hashtable size: 1399999 Memory usage: 690130432 percentage: 0.648

2014-12-18 03:21:21 Processing rows: 1500000 Hashtable size: 1499999 Memory usage: 737340976 percentage: 0.692

2014-12-18 03:21:24 Processing rows: 1600000 Hashtable size: 1599999 Memory usage: 793258352 percentage: 0.745

2014-12-18 03:21:25 Processing rows: 1700000 Hashtable size: 1699999 Memory usage: 841009952 percentage: 0.789

2014-12-18 03:21:25 Processing rows: 1800000 Hashtable size: 1799999 Memory usage: 887464680 percentage: 0.833

2014-12-18 03:21:28 Processing rows: 1900000 Hashtable size: 1899999 Memory usage: 934581288 percentage: 0.877

2014-12-18 03:21:28 Processing rows: 2000000 Hashtable size: 1999999 Memory usage: 984062056 percentage: 0.924

Execution failed with exit status: 3

解决方法：

set hive.auto.convert.join=false;关闭自动转化MapJoin，默认为true;

set hive.ignore.mapjoin.hint=false; 关闭忽略mapjoin的hints（不忽略，hints有效），默认为true(忽略hints)。

转载于:https://www.cnblogs.com/weiqiang-liu/p/4691675.html