Hadoop Kernel tunning

Hadoop性能调优配置

最新推荐文章于 2025-09-13 12:54:57 发布

转载最新推荐文章于 2025-09-13 12:54:57 发布 · 58 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/huaxiaoyao/p/5050033.html

文章标签：

#大数据

本文介绍了如何通过配置文件limits.conf和sysctl.conf来优化Hadoop集群的性能，包括设置文件描述符数量、内存锁定限制及调整虚拟内存策略等。

/etc/security/limits.conf

@ochadoop soft nofile 102642
@ochadoop hard nofile 102642
@ochadoop soft nproc unlimited
@ochadoop hard nproc unlimited
@ochadoop soft memlock unlimited
@ochadoop hard memlock unlimited

/etc/sysctl.conf

#尽可能少地让应用把内容交换到硬盘
echo "vm.swappiness = 0" >> /etc/sysctl.conf
#可以让应用在请求内存的时候使用交换
echo 'vm.overcommit_memory = 1' >> /etc/sysctl.conf

Mount options:disabling file access time

The real problem with maintaining access time (or
atime as it’s commonly called) is that every time a file is read, the metadata needs to be
updated. That is, for each read, there’s also a mandatory write. This is relatively expensive at scale and can negatively impact the overall performance of Hadoop

sample:

LABEL=/ / ext3 noatime,nodiratime 1 1
LABEL=/data/1 /data/1 ext3 noatime,nodiratime 1 2
LABEL=/data/2 /data/2 ext3 noatime,nodiratime 1 2
LABEL=/data/3 /data/3 ext3 noatime,nodiratime 1 2
LABEL=/data/4 /data/4 ext3 noatime,nodiratime 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0