Hadoop集群测试3_disabling erasure coding for path-优快云博客

本文链接：https://blog.youkuaiyun.com/m0_73706260/article/details/127683336

思维导图

（一）通过UI界面查看Hadoop运行状态

1、hadoop2和hadoop3端口区别表

2、查看HDFS集群状态

3、查看YARN集群状态

（二）Hadoop集群初体验 —— 词频统计

6、在YARN集群UI界面查看程序运行状态

思维导图

（一）通过UI界面查看Hadoop运行状态

Hadoop集群正常启动后，它默认开放了两个端口9870和8088，分别用于监控HDFS集群和YARN集群。通过UI界面可以方便地进行集群的管理和查看，只需要在本地操作系统的浏览器输入集群服务的IP和对应的端口号即可访问。

1、hadoop2和hadoop3端口区别表

2、查看HDFS集群状态

在浏览器里访问http://master:9870

不能通过主机名master加端口9870的方式，以级不能通过IP地址加端口号的方式访问：http://192.168.218.219:9870
原因一：在于没有在hosts文件里IP与主机名的映射修改宿主机C:\Windows\System32\drivers\etc\hosts文件

增加hadoop集群主机名与IP地址的映射

原因二：
也可能是防火墙没有关闭，要关闭防火墙：
命令：systemctl status firewalld（打开防火墙设置）
关闭防火墙：命令：systemctl stop firewalld 命令：systemctl disable firewalld

关闭完成后：

此时，访问http://master:9870，从图中可以看出HDFS集群状态显示正常。

单击导航条上的【Datanodes】，查看数据节点信息

点开【Utilities】下拉菜单，选择【Browse the file system】

此时HDFS上什么都没有

在HDFS上创建一个目录BigData，执行命令：hdfs dfs -mkdir /BigData

在Hadoop WebUI界面查看刚才创建的目录

3、查看YARN集群状态

访问http://master:8088/cluster，从图中可以看出YARN集群状态显示正常。

单击[About]链接

（二）Hadoop集群初体验 —— 词频统计

1、启动Hadoop集群

在master虚拟机上执行命令：start-all.sh

2、在虚拟机上准备文件

在master虚拟机上创建test.txt文件
命令：vim test.txt(输入内容，：wq退出保存）
查看test.txt文件，命令：cat test.txt

3、文件上传到HDFS指定目录

上传test.txt文件到HDFS的/BigData目录
命令：hdfs dfs -put test.txt /BigData

利用HDFS命令查看文件是否上传成功
命令：hdfs dfs -ls /BigData

利用Hadoop WebUI查看文件是否上传成功

4、运行词频统计程序的jar包

查看Hadoop自带示例的jar包
执行命令：hadoop jar ./hadoop-mapreduce-examples-3.3.4.jar wordcount /BigData/test.txt /wc_result

2022-11-04 10:35:00,843 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at master/192.168.1.100:8032
2022-11-04 10:35:01,740 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1667521610609_0001
2022-11-04 10:35:02,161 INFO input.FileInputFormat: Total input files to process : 1
2022-11-04 10:35:02,300 INFO mapreduce.JobSubmitter: number of splits:1
2022-11-04 10:35:02,632 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1667521610609_0001
2022-11-04 10:35:02,632 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-11-04 10:35:02,866 INFO conf.Configuration: resource-types.xml not found
2022-11-04 10:35:02,867 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2022-11-04 10:35:03,327 INFO impl.YarnClientImpl: Submitted application application_1667521610609_0001
2022-11-04 10:35:03,370 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1667521610609_0001/
2022-11-04 10:35:03,371 INFO mapreduce.Job: Running job: job_1667521610609_0001
2022-11-04 10:35:11,543 INFO mapreduce.Job: Job job_1667521610609_0001 running in uber mode : false
2022-11-04 10:35:11,545 INFO mapreduce.Job: map 0% reduce 0%
2022-11-04 10:35:17,652 INFO mapreduce.Job: map 100% reduce 0%
2022-11-04 10:35:30,773 INFO mapreduce.Job: map 100% reduce 100%
2022-11-04 10:35:31,798 INFO mapreduce.Job: Job job_1667521610609_0001 completed successfully
2022-11-04 10:35:32,086 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=98
FILE: Number of bytes written=552153
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=190
HDFS: Number of bytes written=60
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3517
Total time spent by all reduces in occupied slots (ms)=11049
Total time spent by all map tasks (ms)=3517
Total time spent by all reduce tasks (ms)=11049
Total vcore-milliseconds taken by all map tasks=3517
Total vcore-milliseconds taken by all reduce tasks=11049
Total megabyte-milliseconds taken by all map tasks=3601408
Total megabyte-milliseconds taken by all reduce tasks=11314176
Map-Reduce Framework
Map input records=5
Map output records=16
Map output bytes=154
Map output materialized bytes=98
Input split bytes=100
Combine input records=16
Combine output records=8
Reduce input groups=8
Reduce shuffle bytes=98
Reduce input records=8
Reduce output records=8
Spilled Records=16
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=151