cloudera--CDH Performance Management

最新推荐文章于 2023-12-11 20:29:57 发布

wandy0211

最新推荐文章于 2023-12-11 20:29:57 发布

阅读量170

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/wjandy0211/article/details/90229396

本文档提供了Cloudera集群的性能管理指南，包括禁用tuned服务、透明大页、调整vm.swappiness参数等操作，以及针对MapReduce配置的最佳实践，如心跳发送、任务间隔、JVM启动优化等，旨在提升集群效率。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

集群管理指南描述了如何使用Cloudera Manager在Cloudera企业部署中配置和管理集群。Cloudera Enterprise Hadoop管理员管理资源、主机、高可用性以及备份和恢复配置。Cloudera Manager管理控制台是管理员用来监视和管理集群的主要工具。您还可以将Cloudera Manager API用于集群管理任务。

Performance Management

https://www.cloudera.com/documentation/enterprise/6/6.0/topics/cdh_admin_performance.html#cdh_performance

Disable the tuned Service

Ensure that the tuned service is started:
```
systemctl start tuned
```
Turn the tuned service off:
```
tuned-adm off
```
Ensure that there are no active profiles:
```
tuned-adm list
```
The output should contain the following line:
```
No current active profile
```

Shutdown and disable the tuned service:

systemctl stop tuned
systemctl disable tuned

Disabling Transparent Hugepages

To see whether transparent hugepages are enabled, run the following commands and check the output:

$ cat defrag_file_pathname
$ cat enabled_file_pathname

[always] never means that transparent hugepages is enabled.
always [never] means that transparent hugepages is disabled.

To disable transparent hugepages on reboot, add the following commands to the /etc/rc.d/rc.local file on all cluster hosts:

RHEL/CentOS 7.x:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

RHEL/CentOS 6.x

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

Setting the vm.swappiness Linux Kernel Parameter

To view your current setting for vm.swappiness, run:

cat /proc/sys/vm/swappiness

To set vm.swappiness to 1, run:

sudo sysctl -w vm.swappiness=1

Improving Performance in Shuffle Handler and IFile Reader

Best Practices for MapReduce Configuration

Send a heartbeat as soon as a task finishes

<property>
    <name>mapreduce.tasktracker.outofband.heartbeat</name>
    <value>true</value>
</property>

Reduce the interval for JobClient status reports on single node systems

<property>
    <name>jobclient.progress.monitor.poll.interval</name>
    <value>10</value>
</property>

Tune the JobTracker heartbeat interval

Tuning the minimum interval for the TaskTracker-to-JobTracker heartbeat to a smaller value may improve MapReduce performance on small clusters.

<property>
    <name>mapreduce.jobtracker.heartbeat.interval.min</name>
    <value>10</value>
</property>

Start MapReduce JVMs immediately

Map . Reduce .slowstart.completed.maps属性指定在调度任何Reduce任务之前必须完成的任务中映射任务的比例。对于需要快速周转的小型作业，将此值设置为0可以提高性能;较大的值(高达50%)可能适合较大的作业。

<property>
    <name>mapred.reduce.slowstart.completed.maps</name>
    <value>0</value>
</property>

Decrease Reserve Space

Cloudera recommends reducing the root user block reservation from 5% to 1% for the DataNode volumes. To set reserved space to 1% with the tune2fs command:

# tune2fs -m 1 /dev/sde1