.hive-staging_hive临时文件处理

.hive-staging_hive临时文件不删除原因:
通过spark-sql、hive-sql、hue等提交select或者insert overwrite等sql到hive时,会产生该目录,用于临时存放执行结果,比如insert overwrite会将结果暂存到该目录下,待任务结束,将结果复制到hive表中。hql任务执行失败时,这些临时文件和目录不会被自动删除掉,直到有相关的hql执行成功时,才会自动删掉。

解决办法:
hive-site.xml 服务高级配置代码段(安全阀)和 客户端高级配置代码段(安全阀),增加配置:

<property>
    <name>hive.insert.into.multilevel.dirs</name>
    <value>true</value>
    <description>允许生成多级目录</description>
</property>
<property>
    <name>hive.exec.stagingdir</name>
    <value>/tmp/hive/staging/.hive-staging</value>
    <description>临时文件暂放目录</description>
</property>
Query ID = root_20250402142135_3ca1da7e-f563-4a4a-90af-d3f314ba82e6 Total jobs = 3 Launching Job 1 out of 3 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1743574293623_0001, Tracking URL = http://master:8088/proxy/application_1743574293623_0001/ Kill Command = /root/software/hadoop-3.3.0/bin/mapred job -kill job_1743574293623_0001 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2025-04-02 14:22:39,211 Stage-1 map = 0%, reduce = 0% 2025-04-02 14:22:54,895 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.48 sec 2025-04-02 14:23:02,452 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.82 sec MapReduce Total cumulative CPU time: 3 seconds 820 msec Ended Job = job_1743574293623_0001 Stage-4 is selected by condition resolver. Stage-3 is filtered out by condition resolver. Stage-5 is filtered out by condition resolver. Moving data to directory hdfs://master:9820/user/hive/warehouse/student/.hive-staging_hive_2025-04-02_14-21-35_316_3330971039988947438-1/-ext-10000 Loading data to table default.student MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.82 sec HDFS Read: 15213 HDFS Write: 241 SUCCESS Total MapReduce CPU Time Spent: 3 seconds 820 msec OK _col0 _col1 Time taken: 100.794 seconds
最新发布
04-03
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值