MapReduce编程开发之数据排序

    MapReduce的数据排序，其实没有很复杂的实现，默认在shuffle阶段，MapReduce就帮我们将数据排好序了，我们在Map和Reduce阶段，无需做额外的操作。

    MapReduce在shuffle阶段，默认帮我们将数据做了排序，并且是做了合并，相同的数据归为一组了。在reduce的时候，我们需要注意的是：因为相同的键做了合并，为了将所有的数据输出，还需要遍历每一个键，如果values集合长度大于1，那么就有可能是重复的数据。另外为了记录数据的顺序，我们的程序里面设定了一个全局静态变量lineNum，记录所有集合的长度总和，同时也是数据的输出顺序。

mapreduce程序：

package com.xxx.hadoop.mapred;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

/**
 * 数据排序
 *
 */
public class DataSortApp {
	
	public static class Map extends Mapper<Object, Text, IntWritable, IntWritable>{
		protected void map(Object key, Text value, 
				Context context) throws IOException ,InterruptedException {
			StringTokenizer tokenizer = new StringTokenizer(value.toString(),"\n");
			while(tokenizer.hasMoreElements()) {
				IntWritable data = new IntWritable(Integer.parseInt(tokenizer.nextToken()));
				context.write(data, new IntWritable(1));
			}
		}
	}
	
	public static class Reduce extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable>{
		private static IntWritable lineNum = new IntWritable(1);
		@SuppressWarnings("unused")
		protected void reduce(IntWritable key, Iterable<IntWritable> values, 
				Context context) throws IOException ,InterruptedException {
			for(IntWritable val:values) {
				context.write(lineNum, key);
				lineNum.set(lineNum.get()+1);
			}
		}
	}

	public static void main(String[] args) throws Exception{
		String input="/user/root/datasort/input",
		       output="/user/root/datasort/output";
		System.setProperty("HADOOP_USER_NAME", "root");
		Configuration conf = new Configuration();
		conf.set("fs.defaultFS", "hdfs://192.168.56.202:9000");
		
		FileSystem fs = FileSystem.get(conf);
		boolean exists = fs.exists(new Path(output));
		if(exists) {
			fs.delete(new Path(output), true);
		}
		
		Job job = Job.getInstance(conf);
		job.setJarByClass(DataSortApp.class);
		job.setMapperClass(Map.class);
		job.setReducerClass(Reduce.class);
		job.setMapOutputKeyClass(IntWritable.class);
		job.setMapOutputValueClass(IntWritable.class);
		job.setOutputKeyClass(IntWritable.class);
		job.setOutputValueClass(IntWritable.class);
		
		FileInputFormat.addInputPath(job, new Path(input));
		FileOutputFormat.setOutputPath(job, new Path(output));
		
		System.exit(job.waitForCompletion(true)?0:1);
	}

}

准备数据：

运行程序，控制台打印信息：

2019-08-31 22:22:35 [INFO ]  [main]  [org.apache.hadoop.conf.Configuration.deprecation] session.id is deprecated. Instead, use dfs.metrics.session-id
2019-08-31 22:22:35 [INFO ]  [main]  [org.apache.hadoop.metrics.jvm.JvmMetrics] Initializing JVM Metrics with processName=JobTracker, sessionId=
2019-08-31 22:22:35 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-08-31 22:22:35 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] Total input paths to process : 3
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] number of splits:3
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] Submitting tokens for job: job_local1111900714_0001
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] The url to track the job: http://localhost:8080/
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Running job: job_local1111900714_0001
2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter set in config null
2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitte