阿里云搭建hadoop伪分布式

购买机器

地址：https://www.aliyun.com/product/ecs?spm=5176.12825654.eofdhaal5.2.e9392c4acVBgPk

使用 root登录机器

创建用户

1. $ useradd hadoop
2. $ vi /etc/sudoers
3.  
4. # 找到root ALL=(ALL) ALL，添加
5. hadoop ALL=(ALL) NOPASSWD:ALL

创建目录

在hadoop用户的根目录下创建目录

1. $ mkdir app data lib maven_repos software script source tmp
2.  
3. # app 最终软件安装的目录
4. # data 测试数据
5. # lib 开发的jar
6. # maven_repos Maven本地仓库
7. # software 软件
8. # script 脚本 flume hive project
9. # source 源码
10. # tmp 临时文件夹

下载 JDK解压文件

jdk安装

• 下载

1. https://download.oracle.com/otn/java/jdk/8u221-b11/230deb18db3e4014bb8e3e8324f81b43/jdk-8u221-linux-x64.tar.gz

• 配置、解压文件

1. $ mkdir /usr/java
2. $ tar -zxvf ./jdk-8u221-linux-x64.tar.gz -C /usr/java/
3. #配置环境变量
4. $ sudo vim /etc/profile
5.  
6. export JAVA_HOME=/usr/java/jdk1.8.0_221
7. export PATH=$JAVA_HOME/bin:$PATH
8.  
9.  
10. $ source /etc/profile
11. #验证
12. $ java
13. Usage: java [-options] class [args...]
14. (to execute a class)
15. or java [-options] -jar jarfile [args...]
16. (to execute a jar file)
17. where options include:
18. -d32 use a 32-bit data model if available
19. -d64 use a 64-bit data model if available
20. -server to select the "server" VM
21. The default VM is server,
22. because you are running on a server-class machine.
23.  
24.  
25. -cp <class search path of directories and zip/jar files>
26. -classpath <class search path of directories and zip/jar files>
27. A : separated list of directories, JAR archives,
28. and ZIP archives to search for class files.
29. -D<name>=<value>
30. set a system property
31. -verbose:[class|gc|jni]
32. enable verbose output
33. -version print product version and exit
34. -version:<value>
35. Warning: this feature is deprecated and will be removed
36. in a future release.
37. require the specified version to run
38. -showversion print product version and continue
39. -jre-restrict-search | -no-jre-restrict-search
40. Warning: this feature is deprecated and will be removed
41. in a future release.
42. include/exclude user private JREs in the version search
43. -? -help print this help message
44. -X print help on non-standard options
45. -ea[:<packagename>...|:<classname>]
46. -enableassertions[:<packagename>...|:<classname>]
47. enable assertions with specified granularity
48. -da[:<packagename>...|:<classname>]
49. -disableassertions[:<packagename>...|:<classname>]
50. disable assertions with specified granularity
51. -esa | -enablesystemassertions
52. enable system assertions
53. -dsa | -disablesystemassertions
54. disable system assertions
55. -agentlib:<libname>[=<options>]
56. load native agent library <libname>, e.g. -agentlib:hprof
57. see also, -agentlib:jdwp=help and -agentlib:hprof=help
58. -agentpath:<pathname>[=<options>]
59. load native agent library by full pathname
60. -javaagent:<jarpath>[=<options>]
61. load Java programming language agent, see java.lang.instrument
62. -splash:<imagepath>
63. show splash screen with specified image

配置免密登录

1. $ ssh-keygen -t rsa
2. Generating public/private rsa key pair.
3. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
4. Created directory '/home/hadoop/.ssh'.
5. Enter passphrase (empty for no passphrase):
6. Enter same passphrase again:
7. Your identification has been saved in /home/hadoop/.ssh/id_rsa.
8. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
9. The key fingerprint is:
10. SHA256:8ZhB5+0Mu1WPhe85fijGI08Dctp/V+wjLXxtR1x+5Vw hadoop@bigdata
11. The key's randomart image is:
12. +---[RSA 2048]----+
13. | . . |
14. | . o . . |
15. | o o . o .|
16. | * = . =E|
17. | S = + .**|
18. | = + .@|
19. | . o.+ .*=|
20. | .o==++O|
21. | +o++++|
22. +----[SHA256]-----+
23.  
24. $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
25. $ chmod 0600 ~/.ssh/authorized_keys
26. $ ssh bigdata
27. The authenticity of host 'bigdata (172.26.19.x)' can't be established.
28. ECDSA key fingerprint is SHA256:yoQ6N2QsIscrIYJiSJ7MnHIWdQ8T/zAaTIcXB0zZJNY.
29. ECDSA key fingerprint is MD5:ba:ed:b5:0a:c2:f0:0a:52:e1:8d:76:1f:4d:b0:72:28.
30. Are you sure you want to continue connecting (yes/no)? yes
31. Warning: Permanently added 'bigdata,172.26.19.x' (ECDSA) to the list of known hosts.
32. Last login: Mon Aug 26 22:26:45 2019 from 124.64.74.89
33.  
34. Welcome to Alibaba Cloud Elastic Compute Service !
35. $ ssh localhost date
36. The authenticity of host 'localhost (127.0.0.1)' can't be established.
37. ECDSA key fingerprint is SHA256:yoQ6N2QsIscrIYJiSJ7MnHIWdQ8T/zAaTIcXB0zZJNY.
38. ECDSA key fingerprint is MD5:ba:ed:b5:0a:c2:f0:0a:52:e1:8d:76:1f:4d:b0:72:28.
39. Are you sure you want to continue connecting (yes/no)? yes
40. Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
41. Mon Aug 26 22:56:14 CST 2019

下载 HADOOP解压文件

安装

• 下载

1. http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.15.1.tar.gz

• 配置、解压文件

1. $ tar -zxvf hadoop-2.6.0-cdh5.15.1.tar.gz -C ../app/
2. #创建软连接
3. $ ln -s /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/ hadoop
4. # 配置当前用户的环境变量
5. $ vim ~/.bash_profile
6. export HADOOP_HOME=/home/hadoop/app/hadoop
7. export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
8.  
9. $ source ~/.bash_profile

• hadoop的基本配置

1. $ vim /home/hadoop/app/hadoop/etc/hadoop/core-site.xml
2.  
3. <configuration>
4.  
5. <property>
6. <name>fs.defaultFS</name>
7. <value>hdfs://172.26.19.217:9000</value> # 配置自己机器的IP
8. </property>
9. <property>
10. <name>hadoop.tmp.dir</name>
11. <value>/home/hadoop/data/hadoop/tmp</value>
12. </property>
13.  
14. </configuration>
15.  
16. $ vim /home/hadoop/app/hadoop/etc/hadoop/hdfs-site.xml
17.  
18. <configuration>
19. <property>
20. <name>dfs.replication</name>
21. <value>1</value>
22. </property>
23.  
24. </configuration>

• 启动和验证

1. $ hdfs namenode -format
2. 19/08/26 23:18:47 INFO namenode.NameNode: STARTUP_MSG:
3. /************************************************************
4. STARTUP_MSG: Starting NameNode
5. STARTUP_MSG: user = hadoop
6. STARTUP_MSG: host = bigdata/172.26.19.x
7. STARTUP_MSG: args = [-format]
8. STARTUP_MSG: version = 2.6.0-cdh5.15.1
9.  
10. $ start-dfs.sh
11. Starting namenodes on [bigdata]
12. bigdata: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-namenode-bigdata.out
13. localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-datanode-bigdata.out
14. Starting secondary namenodes [0.0.0.0]
15. 0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-secondarynamenode-bigdata.out
16. condarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-secondarynamenode-bigdata.out
17. $ jps
18. 3970 Jps
19. 3588 NameNode
20. 3861 SecondaryNameNode
21. 3708 DataNode

配置 yarn

1. $ cd /home/hadoop/app/hadoop/etc/hadoop
2.  
3. $ cp mapred-site.xml.template mapred-site.xml
4.  
5. $ vim mapred-site.xml
6. <configuration>
7. <property>
8. <name>mapreduce.framework.name</name>
9. <value>yarn</value>
10. </property>
11. </configuration>
12.  
13. $ vim yarn-site.xml
14. <configuration>
15. <property>
16. <name>yarn.nodemanager.aux-services</name>
17. <value>mapreduce_shuffle</value>
18. </property>
19. </configuration>
20.  
21. $ start-yarn.sh
22. starting yarn daemons
23. starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-resourcemanager-bigdata.out
24. localhost: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-nodemanager-bigdata.out
25.  
26. $ jps
27. 4161 NodeManager
28. 4066 ResourceManager
29. 3588 NameNode
30. 3861 SecondaryNameNode
31. 4454 Jps
32. 3708 DataNode

阿里云安全组设置

浏览器测试是否可以访问

输入ip地址:50070

• 参考文档:http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.15.1/hadoop-project-dist/hadoop-common/SingleCluster.html