hive hive.optimize.ppd=false导致map数量很大

hive官方https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView

Prior to Hive 0.6.0, lateral view did not support the predicate push-down optimization. In Hive 0.5.0 and earlier, if you used a WHERE clause your query may not have compiled. A workaround was to add set hive.optimize.ppd=false; before your query. The fix was made in Hive 0.6.0; see https://issues.apache.org/jira/browse/HIVE-1056: Predicate push down does not work with UDTF's.
在0.5.0版本前使用hive.optimize.ppd=false是能解决where条件不识别的。但我在使用0.9.0版本时使用了该参数,导致where不识别。
不使用参数日志:

Starting Job = job_201312261829_7430, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201312261829_7430
Kill Command = /home/hadoop/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=master:8021 -kill job_201312261829_7430
Hadoop job information for Stage-1: number of mappers: 90; number of reducers: 287

使用该参数日志:

Starting Job = job_201312261829_7433, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201312261829_7433
Kill Command = /home/hadoop/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=master:8021 -kill job_201312261829_7433
Hadoop job information for Stage-1: number of mappers: 4363; number of reducers: 999


所以在新版本的hive中使用hive.optimize.ppd=true;默认值即可
-- set job name set tez.job.name=market.market_w_user_pool_feature; set hive.execution.engine=tez; set tez.queue.name=phl_dw; set paimon.hadoop-load-default-config =false; -- 20250609 增加性能调优参数-开启向量化 谓词下推 小表mapjoin SET hive.vectorized.execution.enabled=true; SET hive.vectorized.execution.reduce.enabled=true; SET hive.optimize.ppd=true; -- 谓词下推 SET hive.optimize.ppd.storage=true; SET hive.auto.convert.join=true; SET hive.mapjoin.smalltable.filesize=104857600; SET hive.auto.convert.join.noconditionaltask.size=251658240; --小文件合并 set hive.merge.mapfiles=true; set hive.merge.mapredfiles=true; set hive.merge.size.per.task=256000000; set hive.merge.smallfiles.avgsize=32000000; --数据倾斜 set hive.map.aggr=true; set hive.groupby.skewindata=true; set hive.optimize.skewjoin=true; set hive.skewjoin.key = 100000; --tez set tez.am.resource.memory.mb=6144; set tez.task.resource.memory.mb=10240; -- set hive.tez.java.opts=8192; set hive.tez.container.max.java.heap.fraction=0.8; set tez.task.resource.cpu.vcores=3; set tez.runtime.io.sort.mb=4096; set tez.runtime.unordered.output.buffer.size-mb=1024; set tez.runtime.skew.detect=true; set tez.runtime.skew.maxsplit=10; drop table if exists dw_tmp.tmp_market_list_pool_v2; create table dw_tmp.tmp_market_list_pool_v2 stored as orc as select m.apply_source source_name ,u.user_uuid ,u.user_phone ,p.plain_phone_md5 user_phone_md5 ,u.user_source ,u.user_channel_code ,u.user_status ,u.user_create_at as register_time from dwb_paimon.dwb_c_user as u left join dwd.dwd_w_user_risk_source_mapping_paimon as m on u.user_source=m.user_source left join privacy.privacy_c_plain_paimon as p on p.plain_user_id=u.user_id where u.user_status='ACTIVE' ; 该hive on tez报错: ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 3, vertexId=vertex_1746754059482_90650_1_01, diagnostics=[Task failed, taskId=task_1746754059482_90650_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_e08_1746754059482_90650_01_000003 finished with diagnostics set to [Container failed, exitCode=1. [2025-08-04 20:25:19.841]Exception from container-launch. Container id: container_e08_1746754059482_90650_01_000003 Exit code: 1 Exception message: Launch container failed Shell error output: Nonzero exit code=1, error message='Invalid argument number' Shell output: main : command provided 1 main : run as user is omm main : requested yarn user is deploy Getting exit code file... Creating script paths... Writing pid file... Writing to tmp file /srv/BigData/data2/nm/localdir/nmPrivate/application_1746754059482_90650/container_e08_1746754059482_90650_01_000003/container_e08_1746754059482_90650_01_000003.pid.tmp Writing to cgroup task files... Creating local dirs... Launching container... [2025-08-04 20:25:19.842]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class 8192
最新发布
08-05
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值