CDH HIVE报错引发的一系列血案

今天偶尔发现一个CDH集群中的Hive MetaStore Server发生了异常，于是检查相关日志，具体日志为Hive MetaStore Server所在节点的/var/log/hive/hadoop-cmf-hive-HIVEMETASTORE-sbh01.esgyn.cn.log.out，日志报错如下，

2019-10-31 06:22:51,467 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-5-thread-199]: 199: source:172.31.232.22 get_table : db=cloudera_manager_metastore_canary_test_db_hive_HIVEME
TASTORE_89c6545c32a1b6da390011bba5c4799a tbl=CM_TEST_TABLE
2019-10-31 06:22:51,467 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-199]: ugi=hue        ip=172.31.232.22        cmd=source:172.31.232.22 get_table : db=cloudera_
manager_metastore_canary_test_db_hive_HIVEMETASTORE_89c6545c32a1b6da390011bba5c4799a tbl=CM_TEST_TABLE
2019-10-31 06:22:51,470 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-5-thread-199]: Retrying HMSHandler after 2000 ms (attempt 4 of 10) with error: javax.jdo.JDOException: E
xception thrown when executing query
        at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
        at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:275)
        at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1217)
        at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1024)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMet