解决hive数据仓库执行Map任务时，Map任务中的容器(container)超过了虚拟内存限制导致任务失败的问题

解决Hive Map任务容器超虚拟内存限制问题

原创

已于 2024-12-20 17:51:52 修改 · 1.9k 阅读

33 ·

CC 4.0 BY-SA版权

文章标签：

#数据仓库 #hive #hadoop

于 2024-12-12 20:18:13 首次发布

具体的错误演示大概如下所示：

我们先创建一个表

create table if not exists dept( deptno int,

dname string, loc int

)

row format delimited fields terminated by '\t';

我们向表中插入几条数据，然后就会报错如下图

insert into table dept values(1,'accounting',1700),(0,'researc',180),(2,'sale',190),(0,'operations',1700);

错误信息如下：

hive (default)> insert into table dept values(1,'accounting',1700),(0,'researc',180),(2,'sale',190),(0,'operations',1700);
Query ID = zq_20241212191427_1aeca309-eeb5-405b-8ce7-99c6b8bb31d8
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1734001217809_0001, Tracking URL = http://hadoop102:8088/proxy/application_1734001217809_0001/
Kill Command = /opt/module/hadoop-3.1.3/bin/mapred job -kill job_1734001217809_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2024-12-12 19:14:46,565 Stage-1 map