大数据平台运维之MapReduce

Mapreduce

12.在集群节点中/usr/hdp/2.4.3.0-227/hadoop-mapreduce/目录下，存在一个案例JAR                           包hadoop-mapreduce-examples.jar。运行JAR包中的PI程序来进行计算圆周率π的近似值，要求运行5次Map任务，每个Map任务的投掷次数为5，运行完成后输出结果为。

[root@master ~]# hadoop jar/usr/hdp/2.4.3.0-227/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.1.2.4.3.0-227.jarpi 5 5

WARNING: Use "yarn jar" to launch YARNapplications.

Number of Maps = 5

Samples per Map = 5

Wrote input for Map #0

Wrote input for Map #1

Wrote input for Map #2

Wrote input for Map #3

Wrote input for Map #4

Starting Job

17/05/07 03:25:16 INFO impl.TimelineClientImpl:Timeline service address: http://slaver1:8188/ws/v1/timeline/

17/05/07 03:25:16 INFO client.RMProxy: Connecting toResourceManager at slaver1/10.0.0.15:8050

17/05/07 03:25:17 INFO input.FileInputFormat: Totalinput paths to process : 5

17/05/07 03:25:17 INFO mapreduce.JobSubmitter: numberof splits:5

17/05/07 03:25:18 INFO mapreduce.JobSubmitter:Submitting tokens for job: job_1494125392913_0001

17/05/07 03:25:19 INFO impl.YarnClientImpl: Submittedapplication application_1494125392913_0001

17/05/07 03:25:19 INFO mapreduce.Job: The url to trackthe job: http://slaver1:8088/proxy/application_1494125392913_0001/

17/05/07 03:25:19 INFO mapreduce.Job: Running job:job_1494125392913_0001

17/05/07 03:25:30 INFO mapreduce.Job: Jobjob_1494125392913_0001 running in uber mode : false

17/05/07 03:25:30 INFO mapreduce.Job:  map 0% reduce 0%

17/05/07 03:25:36 INFO mapreduce.Job:  map 40% reduce 0%

17/05/07 03:25:41 INFO mapreduce.Job:  map 60% reduce 0%

17/05/07 03:25:42 INFO mapreduce.Job:  map 80% reduce 0%

17/05/07 03:25:45 INFO mapreduce.Job:  map 100% reduce 0%

17/05/07 03:25:48 INFO mapreduce.Job:  map 100% reduce 100%

17/05/07 03:25:49 INFO mapreduce.Job: Jobjob_1494125392913_0001 completed successfully

17/05/07 03:25:49 INFO mapreduce.Job: Counters: 49

        FileSystem Counters

               FILE: Number of bytes read=116

               FILE: Number of bytes written=819237

               FILE: Number of read operations=0

               FILE: Number of large read operations=0

               FILE: Number of write operations=0

               HDFS: Number of bytes read=1300

               HDFS: Number of bytes written=215

               HDFS: Number of read operations=23

               HDFS: Number of large read operations=0

               HDFS: Number of write operations=3

        JobCounters

               Launched map tasks=5

               Launched reduce tasks=1

               Data-local map tasks=5

               Total time spent by all maps in occupied slots (ms)=50808

               Total time spent by all reduces in occupied slots (ms)=10839

               Total time spent by all map tasks (ms)=16936

               Total time spent by all reduce tasks (ms)=3613

                Total vcore-seconds taken by all maptasks=16936

               Total vcore-seconds taken by all reduce tasks=3613

               Total megabyte-seconds taken by all map tasks=26013696

               Total megabyte-seconds taken by all reduce tasks=5549568

       Map-Reduce Framework

               Map input records=5

               Map output records=10

               Map output bytes=90

               Map output materialized bytes=140

               Input split bytes=710

               Combine input records=0

               Combine output records=0

               Reduce input groups=2

               Reduce shuffle bytes=140

               Reduce input records=10

               Reduce output records=0

               Spilled Records=20

                Shuffled Maps =5

               Failed Shuffles=0

               Merged Map outputs=5

               GC time elapsed (ms)=450

               CPU time spent (ms)=4330

               Physical memory (bytes) snapshot=5840977920

                Virtual memory (bytes) snapshot=19436744704

               Total committed heap usage (bytes)=5483528192

        ShuffleErrors

               BAD_ID=0