今天偶尔发现一个CDH集群中的Hive MetaStore Server发生了异常,于是检查相关日志,具体日志为Hive MetaStore Server所在节点的/var/log/hive/hadoop-cmf-hive-HIVEMETASTORE-sbh01.esgyn.cn.log.out,日志报错如下,
2019-10-31 06:22:51,467 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-5-thread-199]: 199: source:172.31.232.22 get_table : db=cloudera_manager_metastore_canary_test_db_hive_HIVEME
TASTORE_89c6545c32a1b6da390011bba5c4799a tbl=CM_TEST_TABLE
2019-10-31 06:22:51,467 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-199]: ugi=hue ip=172.31.232.22 cmd=source:172.31.232.22 get_table : db=cloudera_
manager_metastore_canary_test_db_hive_HIVEMETASTORE_89c6545c32a1b6da390011bba5c4799a tbl=CM_TEST_TABLE
2019-10-31 06:22:51,470 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-5-thread-199]: Retrying HMSHandler after 2000 ms (attempt 4 of 10) with error: javax.jdo.JDOException: E
xception thrown when executing query
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:275)
at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1217)
at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1024)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce