Mac OS下搭建Hadoop3.2.1

1、参考链接汇总

• macOS Hadoop 3.1.0的安装及简单使用 https://www.jianshu.com/p/2c3b04ac498d
• mac下编译hadoop3.0.*版本的native lib https://blog.youkuaiyun.com/qq_14811559/article/details/80308434
• protobuf-2.5.0.tar.gz的下载与安装 https://blog.youkuaiyun.com/will5451/article/details/62885105
• 下载Hadoop3.2.1源码 https://github.com/apache/hadoop/archive/branch-3.2.1.zip
• mac 下编译hadoop3.1.2 https://blog.youkuaiyun.com/u011666720/article/details/99674659
• Mac下SSH免密登录localhost https://blog.youkuaiyun.com/tianqishu11/article/details/78787721
• 执行ssh-add时报错Could not open a connection to your authentication agent https://blog.youkuaiyun.com/Dior_wjy/article/details/79035214

 

2、踩坑

• hadoop启动报错：Unable to load native-hadoop library for your platform

// 参考链接，采用编译源码的方式
https://blog.youkuaiyun.com/u011666720/article/details/99674659
// 修改配置文件 hadoop-env.sh
export HADOOP_OPTS="-Djava.library.path=/usr/local/Cellar/hadoop/3.2.1_1/lib/native"

 

• hadoop-hdfs-native-client 源码编译时失败

// 注释 hadoop-branch-3.2.1/hadoop-hdfs-project/pom.xml 文件中以下代码，重新编译
<module>hadoop-hdfs-native-client</module>

 

• 源码编译命令

mvn clean && mvn package -Pdist,native -DskipTests -Dtar

 

• Apache Hadoop Pipes 源码编译时失败

sudo vim ~/.bash_profile
// 加入以下代码
export OPENSSL_ROOT_DIR=/usr/local/Cellar/openssl/1.0.2t
export OPENSSL_INCLUDE_DIR=/usr/local/Cellar/openssl/1.0.2t/include
source ~/.bash_profile  // 重新载入配置文件，使其立即生效

 

• mac ssh免密登陆localhost，否则hadoop启动报错Permission denied (publickey,password,keyboard-interactive).

// 执行
ssh-add
// 报错 Could not open a connection to your authentication agent.  
// 继续执行
eval `ssh-agent`
ssh-add ~/.ssh/id_rsa
// 输入密码后提示 Identity added: /Users/xiaoanna/.ssh/id_rsa (xiaoanna@mafengwo.com) 成功！
ssh localhost
// 每次重新登录后需再次ssh-add，所以ssh-add后直接进入hadoop目录，执行
./sbin/start-dfs.sh

 

• 启动结果如下

➜  3.2.1_1 ./sbin/start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [xiaoannadeMacBook-Pro.local]
➜  3.2.1_1 ./sbin/start-yarn.sh
Starting resourcemanager
Starting nodemanagers
➜  3.2.1_1 jps
65331 DataNode
65462 SecondaryNameNode
59542
65869 ResourceManager
65965 NodeManager
65230 NameNode
351 Elasticsearch
66078 Jps