记一次hive执行报错问题

文章讲述了在数据中台上运行时遇到的关于Hive任务错误,主要涉及锁管理问题,特别是共享锁和排它锁导致的冲突。提出了解决方案,即在代码中设置`hive.support.concurrency`为false以避免并发问题。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

问题描述:在数据中台上运行一段的时间后,有个别任务出现这样的报错:

ERROR : FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired. retry after some time
org.apache.hadoop.hive.ql.lockmgr.LockException: Locks on the underlying objects cannot be acquired. retry after some time
	at org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager.acquireLocks(DummyTxnManager.java:190)
	at org.apache.hadoop.hive.ql.Driver.acquireLocksAndOpenTxn(Driver.java:1244)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1557)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1334)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)
	at org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
运行作业出现错误,请联系管理员:java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor149.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.dtwave.dipper.spi.plugin.impl.DefaultPluginExecutor.execute(DefaultPluginExecutor.java:26)
	at com.dtwave.dipper.executor.proxy.executor.TaskExecutor.execute(TaskExecutor.java:126)
	at com.dtwave.dipper.executor.proxy.executor.TaskExecutor.run(TaskExecutor.java:55)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Error while processing statement: FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired. retry after some time
	at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:381)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:260)
	at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:487)
	at com.dtwave.dipper.hive.Hive2TaskRunnerPlugin.handleSql(Hive2TaskRunnerPlugin.java:541)
	at com.dtwave.dipper.hive.Hive2TaskRunnerPlugin.normalRun(Hive2TaskRunnerPlugin.java:299)
	at com.dtwave.dipper.spi.plugin.runner.SQLTaskRunnerPlugin.doRun(SQLTaskRunnerPlugin.java:68)
	at com.dtwave.dipper.spi.plugin.runner.BasicTaskRunnerPlugin.execute(BasicTaskRunnerPlugin.java:88)
	... 11 more

作业运行失败(Failed)

引发问题的原因:FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired. retry after some time

Hive中有两种锁模式,分别为:共享锁(S)和排它锁(X)。
多个共享锁(S)可以同时获取,排它锁(X)会阻塞其它所有锁。

如果select一张表,这张表则会进入shared模式,增加、插入、删除、修改数据和修改表名等操作都会在shared锁被释放之后再执行,会一直等待。

如果插入、删除、修改数据则进入Exclusive锁模式,进入排他锁模式之后不允许增删改操作,会报错

ERROR : FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired. retry after some time org.apache.hadoop.hive.ql.lockmgr.LockException: Locks on the underlying objects cannot be acquired. retry after some time

释放掉Exclusive锁模式即可操作。

采取的措施:在代码里面添加:set hive.support.concurrency=false; 即可

参照:https://blog.youkuaiyun.com/jekongoo/article/details/124998077

执行HiveSQL建表命令报错时,可能有几个原因导致: 1. 列名重复:根据引用中的内容,建表语句中指定了两个相同的列名,这可能会导致建表失败。请确保列名的唯一性并重试建表命令。 2. 元数据服务器配置错误:根据引用中的内容,当Hive执行建表命令时,可能会连接到远程的元数据服务器,而不是默认的本地数据库。如果你的MySQL数据库不在本地,你需要在Hive的配置文件中配置元数据服务器的地址。确保配置正确后,重启Hive服务并再次尝试执行建表命令。 3. Orc文件版本不兼容:根据引用中的内容,当使用CDH-Hive2.1.1版本读取Orc文件时,可能会报错。可能是由于Orc文件的版本与Hive不兼容导致的。你可以尝试替换Orc文件或升级Hive版本来解决这个问题。 综上所述,建议你检查列名是否重复,配置正确的元数据服务器地址,并考虑升级Hive版本或替换Orc文件来解决建表报错问题。<span class="em">1</span><span class="em">2</span><span class="em">3</span> #### 引用[.reference_title] - *1* [一次hive建表报错](https://blog.youkuaiyun.com/weixin_39445556/article/details/103200509)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 33.333333333333336%"] - *2* [hive执行建表命令报错](https://blog.youkuaiyun.com/weixin_45567175/article/details/102716671)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 33.333333333333336%"] - *3* [cdh-hive2.1.1版本orc格式表读取包数组越界替换包](https://download.youkuaiyun.com/download/weinierzui/85404988)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 33.333333333333336%"] [ .reference_list ]
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值