CDH 5.8 namenode增加内存后重启后碰到的问题

最新推荐文章于 2022-07-21 12:02:42 发布

原创最新推荐文章于 2022-07-21 12:02:42 发布 · 1.2k 阅读

0 ·

CC 4.0 BY-SA版权

hadoop 专栏收录该内容

3 篇文章

订阅专栏

本文详细介绍了在Cloudera集群中增加内存的具体步骤，包括停机与开机顺序、服务重启流程及验证方法。同时，针对增加内存后出现的mapreduce错误和sqoop抽取问题，提供了有效的解决策略。

增加内存步骤

停机顺序：

停止所有服务(spark程序，flume)

停止Cloudera Management Service

停止CDH agent 248/249： service cloudera-scm-agent stop

停掉CDH 服务 248 ：service cloudera-scm-server stop

停掉spark 249 /opt/spark2/spark-2.2.0-bin-hadoop2.6/sbin/stop-all.sh

备份mysql

停掉 249上的slave mysql

停掉 248上master mysql

停服务器

增加内存

开机顺序：

启动服务器

启动248和249mysql

启动spark 249 /opt/spark2/spark-2.2.0-bin-hadoop2.6/sbin/start-all.sh

启动CDH服务248 service cloudera-scm-server start

启动CDH agent service cloudera-scm-agent start

启动 Cloudera Management Service

启动所有服务

验证：

hive select count(*) from table_name;

报mapreduce错误

验证sqoop抽取：

碰到如下问题

18/10/16 19:47:19 INFO mapreduce.Job: map 0% reduce 0%

18/10/16 19:47:19 INFO mapreduce.Job: Job job_1539689333685_0019 failed with state FAILED due to: Application application_1539689333685_0019 failed 2 times due to AM Container for appattempt_1539689333685_0019_000002 exited with exitCode: -1000

For more detailed output, check application tracking page:http://nm2:8088/proxy/application_1539689333685_0019/Then, click on links to logs of each attempt.

Diagnostics: Application application_1539689333685_0019 initialization failed (exitCode=255) with output: main : command provided 0

main : run as user is nobody

main : requested yarn user is test

Can't create directory /disk01/yarn/nm/usercache/test/appcache/application_1539689333685_0019 - Permission denied

Can't create directory /disk02/yarn/nm/usercache/test/appcache/application_1539689333685_0019 - Permission denied

Can't create directory /disk03/yarn/nm/usercache/test/appcache/application_1539689333685_0019 - Permission denied

Can't create directory /disk04/yarn/nm/usercache/test/appcache/application_1539689333685_0019 - Permission denied

Can't create directory /disk05/yarn/nm/usercache/test/appcache/application_1539689333685_0019 - Permission denied

Can't create directory /disk06/yarn/nm/usercache/test/appcache/application_1539689333685_0019 - Permission denied

Can't create directory /disk07/yarn/nm/usercache/test/appcache/application_1539689333685_0019 - Permission denied

Can't create directory /disk08/yarn/nm/usercache/test/appcache/application_1539689333685_0019 - Permission denied

Did not create any app directories

解决办法，删除集群所有datanode中usercache的文件

1、移动到/tmp目录下

mkdir -p /tmp/user

for i in {01..08}; do mkdir -p /tmp/user/disk$i/; done

for i in {01..08}; do mv /disk$i/yarn/nm/usercache/* /tmp/user/disk$i/; done

2、直接删除

for i in {01..08}; do rm -rf /disk$i/yarn/nm/usercache/* ; done

删除后问题解决