hive执行sql:stage-1 map = 0 reduce = 0

在执行Hive连表操作时遇到stage-1 map = 0 reduce = 0的问题,可能是由于笛卡尔积、YARN资源不足或者MapReduce任务输出错误等原因。解决方案包括检查SQL语句、调整YARN资源配置、验证MapReduce任务的正确性以及考虑升级Hive版本。

执行hive连表时,出现这种情况:

WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = root_20180824151434_b0b979fe-7a85-4ac6-b924-33dd40897a28
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1535114415498_0379, Tracking URL = http://master:8088/proxy/application_1535114415498_0379/
Kill Command = /opt/hadoop-2.7.4/bin/hadoop job  -kill job_1535114415498_0379
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2023-04-24 15:14:57,985 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:15:58,823 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:16:59,579 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:18:00,262 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:19:00,912 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:20:01,566 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:21:02,214 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:22:02,841 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:23:03,480 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:24:04,120 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:25:04,711 Stage-1 map = 0%,  reduce = 0%
2023-04-24 15:26:05,313 Stage-1 map = 0%,  reduce = 0%

控制台一直输出 stage-1 map = 0 reduce = 0
一般有以下几种原因:

        一、连表语句有误,产生笛卡尔积

        二、Yarn资源不足,增加Yarn资源内存

        解决方式:
        修改yarn-site.xml中的yarn.nodemanager.resource.memory-mb值大小

<property>
      <name>yarn.nodemanager.resource.memory-mb</name>
      <value>4096</value>
</property>

        三、其他 

        当 Hive 执行 SQL 时,它会将 SQL 语句转换为 MapReduce 任务,并且在第一阶段 (Map 阶段) 只会执行 Map 任务,不会执行 Reduce 任务。如果 Map 阶段的输出没有被正确地合并,则可能会导致第一阶段的 Map 任务输出为 0。因此,如果 Hive 执行 SQL 时显示“stage-1 map = 0 reduce = 0”,则表示第一阶段的 Map 任务没有产生任何输出,可能是因为输出被错误地合并或丢失了。

要解决这个问题,可以尝试以下方法:

  1. 检查 Hive 查询是否使用了正确的排序顺序,以确保 Map 阶段可以正确地合并输出。

  2. 检查 Hive 查询是否使用了正确的分组或过滤条件,以确保 Map 阶段可以正确地处理数据并产生输出。

  3. 检查 Hive 查询是否使用了正确的 MapReduce 框架,例如是否使用了正确的 MapReduce 版本或自定义的 MapReduce 函数。

  4. 检查 Hive 查询是否使用了正确的数据类型,以确保 Map 阶段可以正确地处理数据并产生输出。

  5. 检查 Hive 查询是否使用了正确的连接类型,以确保 Map 阶段可以正确地连接数据并产生输出。

  6. 如果以上方法都无法解决问题,则可以尝试升级 Hive 版本或使用不同的 MapReduce 框架。

hive> LOAD DATA INPATH '/user/hive/external_data/machine_data_20251027.txt' > INTO TABLE ods.Basemachine > PARTITION (etldate='2025-10-27'); Loading data to table ods.basemachine partition (etldate=2025-10-27) Partition ods.basemachine{etldate=2025-10-27} stats: [numFiles=1, numRows=0, totalSize=635, rawDataSize=0] OK Time taken: 0.537 seconds hive> LOAD DATA INPATH '/user/hive/external_data/machine_data_20251102.txt' > INTO TABLE ods.Basemachine > PARTITION (etldate='2025-11-02'); Loading data to table ods.basemachine partition (etldate=2025-11-02) Partition ods.basemachine{etldate=2025-11-02} stats: [numFiles=1, numRows=0, totalSize=601, rawDataSize=0] OK Time taken: 0.377 seconds hive> SET hive.cli.print.header=true; hive> SET hive.resultset.use.unique.column.names=false; hive> SHOW PARTITIONS ods.Basemachine; OK partition etldate=2025-10-27 etldate=2025-11-02 Time taken: 0.078 seconds, Fetched: 2 row(s) hive> SELECT > etldate, > COUNT(*) AS total_records, > COUNT(DISTINCT BaseMachineID) AS unique_machines > FROM ods.Basemachine > GROUP BY etldate; Query ID = root_20251103172246_ad44da2d-d2b0-4ef2-b28d-e2680ff82506 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1762153875239_0008, Tracking URL = http://node01:8088/proxy/application_1762153875239_0008/ Kill Command = /home/qxy27/Bigdata/servers/hadoop-2.7.4/bin/hadoop job -kill job_1762153875239_0008 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2025-11-03 17:22:56,227 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:23:56,562 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:24:56,861 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:25:57,113 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:26:57,327 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:27:57,550 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:28:57,760 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:29:57,988 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:30:58,202 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:31:58,418 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:32:58,615 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:33:58,835 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:34:59,045 Stage-1 map = 0%, reduce = 0% 2025-11-03 17:35:59,253 Stage-1 map = 0%, reduce = 0%
11-04
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值