MapReduce编程开发之求平均成绩

    MapReduce计算平均成绩是一个常见的算法，本省思路很简单，就是将每个人的成绩汇总，然后做除法，在map阶段，是直接将姓名做key,分数作为value输出。在shuffle阶段，会将每个人的所有成绩做汇总，数据结构变为<name,<score1,score2...>>这样子，我们在reduce阶段就通过分数这个value-list来结算平均分。average = sum(score)/courseCount，即平均分等于分数总和除以课程数。

mapreduce代码：

package com.xxx.hadoop.mapred;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

/**
 * 求平均成绩
 *
 */
public class AverageScoreApp {

	public static class Map extends Mapper<Object, Text, Text, IntWritable>{
		@Override
		protected void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context)
				throws IOException, InterruptedException {
			//成绩的结构是:
			// 张三	80
			// 李四	82
			// 王五	86
			StringTokenizer tokenizer = new StringTokenizer(value.toString(), "\n");
			while(tokenizer.hasMoreElements()) {
				StringTokenizer lineTokenizer = new StringTokenizer(tokenizer.nextToken());
				String name = lineTokenizer.nextToken(); //姓名
				String score = lineTokenizer.nextToken();//成绩
				context.write(new Text(name), new IntWritable(Integer.parseInt(score)));
			}
		}
	}
	
	public static class Reduce extends Reducer<Text, IntWritable, Text, DoubleWritable>{
		@Override
		protected void reduce(Text key, Iterable<IntWritable> values,
				Reducer<Text, IntWritable, Text, DoubleWritable>.Context context)
				throws IOException, InterruptedException {
			//reduce这里输入的数据结构是：
			// 张三 <80,85,90>
			// 李四 <82,88,94>
			// 王五 <86,80,92>
			int sum = 0;//所有课程成绩总分
			double average = 0;//平均成绩
			int courseNum = 0; //课程数目
			for(IntWritable score:values) {
				sum += score.get();
				courseNum++;
			}
			average = sum/courseNum;
			context.write(new Text(key), new DoubleWritable(average));
		}
	}
	
	public static void main(String[] args) throws Exception{
		String input="/user/root/averagescore/input",
			  output="/user/root/averagescore/output";
		System.setProperty("HADOOP_USER_NAME", "root");
		Configuration conf = new Configuration();
		conf.set("fs.defaultFS", "hdfs://192.168.56.202:9000");
		Job job = Job.getInstance(conf);
		job.setJarByClass(AverageScoreApp.class);
		job.setMapperClass(Map.class);
		job.setReducerClass(Reduce.class);
		
		job.setMapOutputKeyClass(Text.class);
		job.setMapOutputValueClass(IntWritable.class);
		
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(DoubleWritable.class);
		
		FileInputFormat.addInputPath(job, new Path(input));
		FileOutputFormat.setOutputPath(job, new Path(output));
		
		System.exit(job.waitForCompletion(true)?0:1);
	}

}

准备学生成绩数据：

控制台打印信息：

2019-08-31 15:50:26 [INFO ]  [main]  [org.apache.hadoop.conf.Configuration.deprecation] session.id is deprecated. Instead, use dfs.metrics.session-id
2019-08-31 15:50:26 [INFO ]  [main]  [org.apache.hadoop.metrics.jvm.JvmMetrics] Initializing JVM Metrics with processName=JobTracker, sessionId=
2019-08-31 15:50:27 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-08-31 15:50:27 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2019-08-31 15:50:27 [INFO ]  [main]  [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] Total input paths to process : 3
2019-08-31 15:50:27 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] number of splits:3
2019-08-31 15:50:27 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] Submitting tokens for job: job_local83653871_0001
2019-08-31 15:50:27 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] The url to track the job: http://localhost:8080/
2019-08-31 15:50:27 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Running job: job_local83653871_0001
2019-08-31 15:50:27