安装CDH时出现Unexpected error. Unable to verify database connection问题的解决思路

置顶 dongyuan234

已于 2023-03-31 19:16:33 修改

阅读量6.1k

点赞数 5

CC 4.0 BY-SA版权

分类专栏： Cloudera 文章标签： Cloudera CDH CM

于 2019-07-18 12:27:34 首次发布

本文链接：https://blog.youkuaiyun.com/dongyuan234/article/details/96427885

Cloudera 专栏收录该内容

1 篇文章

订阅专栏

本文介绍了解决CDH安装过程中HUE数据库连接失败的问题。通过分析主节点和Agent节点的日志，最终定位到perl命令缺失导致的错误，并通过安装相关软件包解决了问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

出现Unexpected error. Unable to verify database connection问题的解决思路

软件版本：cm-5.13.0

问题场景

在安装CDH过程中，在测试数据库连接环节，验证HUE数据库连接失败，报Unexpected error. Unable to verify database connection错误

问题分析

1. 分析主节点日志

tail -f /opt/cm-5.13.0/log/cloudera-scm-server/cloudera-scm-server.log

日志内容

2019-07-18 11:06:42,477 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=122 name=null host=eb49c2ab-fc46-44f6-b92c-2871c2baa942/dcc-dn1.dcc.com
2019-07-18 11:06:42,477 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=123 name=null host=eb49c2ab-fc46-44f6-b92c-2871c2baa942/dcc-dn1.dcc.com
2019-07-18 11:06:43,829 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection'
2019-07-18 11:06:43,829 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 1
2019-07-18 11:06:43,829 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand:     self._setup(name)
  File "/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup
    self._wrapped = Settings(settings_module)
  File "/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module
    __import__(name)
  File "/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/desktop/core/src/desktop/settings.py", line 325, in <module>
    "PASSWORD" : desktop.conf.get_database_password(),
  File "/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/desktop/core/src/desktop/conf.py", line 1632, in get_database_password
    password = DATABASE.PASSWORD_SCRIPT.get()
  File "/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/desktop/core/src/desktop/lib/conf.py", line 154, in get
    return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True)
  File "/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/desktop/core/src/desktop/lib/conf.py", line 270, in get_value
    return self._coerce_type(raw_val, prefix)
  File "/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/desktop/core/src/desktop/lib/conf.py", line 290, in _coerce_type
    return self.type(raw)
  File "/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/desktop/core/src/desktop/lib/conf.py", line 721, in coerce_password_from_script
    raise subprocess.CalledProcessError(p.returncode, script)
subprocess.CalledProcessError: Command '{{CMF_CONF_DIR}}/altscript.sh sec-2-password' returned non-zero exit status 127

2019-07-18 11:06:43,829 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 150(HueTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.
2019-07-18 11:06:44,674 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'HiveTestDatabaseConnection'
2019-07-18 11:06:44,674 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 0
2019-07-18 11:06:44,674 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: + '[' -z /usr/java/jdk1.8.0_144 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_144 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_144
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ JAVA=/usr/java/jdk1.8.0_144/bin/java
+ [[ -z /opt/cm-5.13.0/share/cmf ]]
+ MGMT_CLASSPATH='/opt/cm-5.13.0/share/cmf/lib/*'
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/*'
++ pwd
+ MGMT_CLASSPATH='/opt/cm-5.13.0/run/cloudera-scm-agent/process/124-HIVE-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/*'
+ MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true '
+ exec /usr/java/jdk1.8.0_144/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/opt/cm-5.13.0/run/cloudera-scm-agent/process/124-HIVE-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties
Thu Jul 18 11:06:43 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

2019-07-18 11:06:44,700 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'RepMgrTestDatabaseConnection'
2019-07-18 11:06:44,700 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 0
2019-07-18 11:06:44,700 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: + '[' -z /usr/java/jdk1.8.0_144 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_144 ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_144
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ JAVA=/usr/java/jdk1.8.0_144/bin/java
+ [[ -z /opt/cm-5.13.0/share/cmf ]]
+ MGMT_CLASSPATH='/opt/cm-5.13.0/share/cmf/lib/*'
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/*'
++ pwd
+ MGMT_CLASSPATH='/opt/cm-5.13.0/run/cloudera-scm-agent/process/125-MGMT.REPORTSMANAGER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/*'
+ MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true '
+ exec /usr/java/jdk1.8.0_144/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/opt/cm-5.13.0/run/cloudera-scm-agent/process/125-MGMT.REPORTSMANAGER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.13.0/share/cmf/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties
Thu Jul 18 11:06:43 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

定位到验证HUE数据库连接相关日志，通过日志只是看到提示subprocess.CalledProcessError: Command ‘{{CMF_CONF_DIR}}/altscript.sh sec-2-password’ returned non-zero exit status 127，并无具体原因。

2. 分析Agent节点日志

远程到需要安装HUE的Agent节点，查看agent日志

tail -f /opt/cm-5.13.0/log/cloudera-scm-agent/cloudera-scm-agent.log

定位到HUE相关日志

[18/Jul/2019 11:06:42 +0000] 30413 MainThread util         INFO     Using generic audit plugin for process HUE-test-db-connection
[18/Jul/2019 11:06:42 +0000] 30413 MainThread util         INFO     Creating metadata plugin for process HUE-test-db-connection
[18/Jul/2019 11:06:42 +0000] 30413 MainThread util         INFO     Using specific metadata plugin for process HUE-test-db-connection
[18/Jul/2019 11:06:42 +0000] 30413 MainThread util         INFO     Using generic metadata plugin for process HUE-test-db-connection
[18/Jul/2019 11:06:42 +0000] 30413 MainThread agent        INFO     [127-HUE-test-db-connection] Instantiating process
[18/Jul/2019 11:06:42 +0000] 30413 MainThread process      INFO     [127-HUE-test-db-connection] Updating process: True {}
[18/Jul/2019 11:06:42 +0000] 30413 MainThread process      INFO     First time to activate the process [127-HUE-test-db-connection].
[18/Jul/2019 11:06:42 +0000] 30413 MainThread agent        INFO     Created /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection
[18/Jul/2019 11:06:42 +0000] 30413 MainThread agent        INFO     Chowning /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection to hue (978) hue (975)
[18/Jul/2019 11:06:42 +0000] 30413 MainThread agent        INFO     Chmod'ing /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection to 0751
[18/Jul/2019 11:06:42 +0000] 30413 MainThread agent        INFO     Created /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/logs
[18/Jul/2019 11:06:42 +0000] 30413 MainThread agent        INFO     Chowning /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/logs to hue (978) hue (975)
[18/Jul/2019 11:06:42 +0000] 30413 MainThread agent        INFO     Chmod'ing /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/logs to 0751
[18/Jul/2019 11:06:42 +0000] 30413 MainThread process      INFO     [127-HUE-test-db-connection] Refreshing process files: None
[18/Jul/2019 11:06:42 +0000] 30413 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.13.3-1.cdh5.13.3.p0.2'}, [u'cdh'], [u'cdh-plugin', u'hue-plugin']
[18/Jul/2019 11:06:42 +0000] 30413 MainThread parcel       INFO     The following requested parcels are not available: {}
[18/Jul/2019 11:06:42 +0000] 30413 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH
[18/Jul/2019 11:06:42 +0000] 30413 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.13.3-1.cdh5.13.3.p0.2'}
[18/Jul/2019 11:06:42 +0000] 30413 MainThread __init__     INFO     Extracted 3 files and 0 dirs to /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection.
[18/Jul/2019 11:06:42 +0000] 30413 MainThread process      INFO     [127-HUE-test-db-connection] Launching process. one-off True, command hue/hue.sh, args [u'is_db_alive']
[18/Jul/2019 11:06:42 +0000] 30413 MainThread agent        INFO     Triggering supervisord update.
[18/Jul/2019 11:06:42 +0000] 30413 MainThread process      INFO     Begin audit plugin refresh

通过日志发现，测试HUE数据库连接的相关日志记录到/opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/logs目录中，该目录每次测试会发生变化，根据日志文件查找。

进入到日志目录并查看错误日志文件

cd /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/logs
ls
access.log  error.log  is_db_alive.log  stderr.log  stdout.log
vi stderr.log

查找stderr.log日志中的错误信息

+ PYTHON_EGG_CACHE=/opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/.python-eggs
+ perl -pi -e 's#{{JAVA_HOME}}#/usr/java/jdk1.8.0_144#g' /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hue.ini
/opt/cm-5.13.0/lib64/cmf/service/hue/hue.sh:行95: perl: 未找到命令
+ perl -pi -e 's#{{HBASE_HOME}}#/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hbase#g' /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hue.ini
/opt/cm-5.13.0/lib64/cmf/service/hue/hue.sh:行96: perl: 未找到命令
+ perl -pi -e 's#{{PIG_SCRIPT_HOME}}#/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/pig/../..#g' /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hue.ini
/opt/cm-5.13.0/lib64/cmf/service/hue/hue.sh:行97: perl: 未找到命令
+ perl -pi -e 's#{{HADOOP_CONF_DIR}}#/opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hadoop-conf#g' /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hue.ini
/opt/cm-5.13.0/lib64/cmf/service/hue/hue.sh:行98: perl: 未找到命令
+ perl -pi -e 's#{{HUE_CONF_DIR}}#/opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection#g' /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hue.ini
/opt/cm-5.13.0/lib64/cmf/service/hue/hue.sh:行99: perl: 未找到命令
+ [[ -d /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hbase-conf ]]
+ [[ -d /opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/sqoop2-conf ]]
+ '[' '' -ge 4 ']'
/opt/cm-5.13.0/lib64/cmf/service/hue/hue.sh: 第 108 行:[: : 期待整数表达式
+ perl -pi -e 's#{{CDH_MR1_HOME}}#/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hadoop-0.20-mapreduce#g' '/opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hadoop-conf/*'
/opt/cm-5.13.0/lib64/cmf/service/hue/hue.sh:行116: perl: 未找到命令
+ perl -pi -e 's#{{CDH_MR2_HOME}}#/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hadoop-mapreduce#g' '/opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hadoop-conf/*'
/opt/cm-5.13.0/lib64/cmf/service/hue/hue.sh:行117: perl: 未找到命令
+ perl -pi -e 's#{{CDH_HADOOP_HOME}}#/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hadoop#g' '/opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection/hadoop-conf/*'
/opt/cm-5.13.0/lib64/cmf/service/hue/hue.sh:行118: perl: 未找到命令
+ replace_conf_dir
+ echo CONF_DIR=/opt/cm-5.13.0/run/cloudera-scm-agent/process/127-HUE-test-db-connection

3、问题修复

通过错误日志发现未找到perl命令，安装相关软件包

yum install php-devel php-pear httpd-devel

4、验证

重新测试，验证通过。

总结

1、分析主节点cloudera-scm-server的相关日志，看能否定位到相关问题

2、分析Agent节点的cloudera-scm-agent，找到错误日志记录位置

3、分析Agent节点错误日志，定位问题

4、修复问题并验证