MapReduce编程开发之数据排序

最新推荐文章于 2024-04-02 17:38:23 发布
原创 最新推荐文章于 2024-04-02 17:38:23 发布 · 1.9k 阅读 · 15 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#mapreduce #数据排序

    MapReduce的数据排序，其实没有很复杂的实现，默认在shuffle阶段，MapReduce就帮我们将数据排好序了，我们在Map和Reduce阶段，无需做额外的操作。

    MapReduce在shuffle阶段，默认帮我们将数据做了排序，并且是做了合并，相同的数据归为一组了。在reduce的时候，我们需要注意的是：因为相同的键做了合并，为了将所有的数据输出，还需要遍历每一个键，如果values集合长度大于1，那么就有可能是重复的数据。另外为了记录数据的顺序，我们的程序里面设定了一个全局静态变量lineNum，记录所有集合的长度总和，同时也是数据的输出顺序。

mapreduce程序：

package com.xxx.hadoop.mapred;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

/**
 * 数据排序
 *
 */
public class DataSortApp {
	
	public static class Map extends Mapper<Object, Text, IntWritable, IntWritable>{
		protected void map(Object key, Text value, 
				Context context) throws IOException ,InterruptedException {
			StringTokenizer tokenizer = new StringTokenizer(value.toString(),"\n");
			while(tokenizer.hasMoreElements()) {
				IntWritable data = new IntWritable(Integer.parseInt(tokenizer.nextToken()));
				context.write(data, new IntWritable(1));
			}
		}
	}
	
	public static class Reduce extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable>{
		private static IntWritable lineNum = new IntWritable(1);
		@SuppressWarnings("unused")
		protected void reduce(IntWritable key, Iterable<IntWritable> values, 
				Context context) throws IOException ,InterruptedException {
			for(IntWritable val:values) {
				context.write(lineNum, key);
				lineNum.set(lineNum.get()+1);
			}
		}
	}

	public static void main(String[] args) throws Exception{
		String input="/user/root/datasort/input",
		       output="/user/root/datasort/output";
		System.setProperty("HADOOP_USER_NAME", "root");
		Configuration conf = new Configuration();
		conf.set("fs.defaultFS", "hdfs://192.168.56.202:9000");
		
		FileSystem fs = FileSystem.get(conf);
		boolean exists = fs.exists(new Path(output));
		if(exists) {
			fs.delete(new Path(output), true);
		}
		
		Job job = Job.getInstance(conf);
		job.setJarByClass(DataSortApp.class);
		job.setMapperClass(Map.class);
		job.setReducerClass(Reduce.class);
		job.setMapOutputKeyClass(IntWritable.class);
		job.setMapOutputValueClass(IntWritable.class);
		job.setOutputKeyClass(IntWritable.class);
		job.setOutputValueClass(IntWritable.class);
		
		FileInputFormat.addInputPath(job, new Path(input));
		FileOutputFormat.setOutputPath(job, new Path(output));
		
		System.exit(job.waitForCompletion(true)?0:1);
	}

}

准备数据：

运行程序，控制台打印信息：

2019-08-31 22:22:35 [INFO ]  [main]  [org.apache.hadoop.conf.Configuration.deprecation] session.id is deprecated. Instead, use dfs.metrics.session-id
2019-08-31 22:22:35 [INFO ]  [main]  [org.apache.hadoop.metrics.jvm.JvmMetrics] Initializing JVM Metrics with processName=JobTracker, sessionId=
2019-08-31 22:22:35 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-08-31 22:22:35 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] Total input paths to process : 3
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] number of splits:3
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] Submitting tokens for job: job_local1111900714_0001
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] The url to track the job: http://localhost:8080/
2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Running job: job_local1111900714_0001
2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter set in config null
2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] Waiting for map tasks
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1111900714_0001_m_000000_0
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@42324622
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/datasort/input/1.txt:0+14
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 32; bufvoid = 104857600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1111900714_0001_m_000000_0 is done. And is in the process of committing
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1111900714_0001_m_000000_0' done.
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1111900714_0001_m_000000_0
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1111900714_0001_m_000001_0
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@1e445343
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/datasort/input/2.txt:0+13
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 32; bufvoid = 104857600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1111900714_0001_m_000001_0 is done. And is in the process of committing
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1111900714_0001_m_000001_0' done.
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1111900714_0001_m_000001_0
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1111900714_0001_m_000002_0
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@3bd4bc73
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/datasort/input/3.txt:0+7
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 16; bufvoid = 104857600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1111900714_0001_m_000002_0 is done. And is in the process of committing
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1111900714_0001_m_000002_0' done.
2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1111900714_0001_m_000002_0
2019-08-31 22:22:37 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] map task executor complete.
2019-08-31 22:22:37 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] Waiting for reduce tasks
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1111900714_0001_r_000000_0
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@5cb84ffa
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.ReduceTask] Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@37f7c67f
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] MergerManager: memoryLimit=1265788544, maxSingleShuffleLimit=316447136, mergeThreshold=835420480, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2019-08-31 22:22:37 [INFO ]  [EventFetcher for fetching Map Completion Events]  [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] attempt_local1111900714_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2019-08-31 22:22:37 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Job job_local1111900714_0001 running in uber mode : false
2019-08-31 22:22:37 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job]  map 100% reduce 0%
2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1111900714_0001_m_000000_0 decomp: 42 len: 46 to MEMORY
2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 42 bytes from map-output for attempt_local1111900714_0001_m_000000_0
2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 42, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->42
2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1111900714_0001_m_000001_0 decomp: 42 len: 46 to MEMORY
2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 42 bytes from map-output for attempt_local1111900714_0001_m_000001_0
2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 42, inMemoryMapOutputs.size() -> 2, commitMemory -> 42, usedMemory ->84
2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1111900714_0001_m_000002_0 decomp: 22 len: 26 to MEMORY
2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 22 bytes from map-output for attempt_local1111900714_0001_m_000002_0
2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 22, inMemoryMapOutputs.size() -> 3, commitMemory -> 84, usedMemory ->106
2019-08-31 22:22:37 [INFO ]  [EventFetcher for fetching Map Completion Events]  [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] EventFetcher is interrupted.. Returning
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Merging 3 sorted segments
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 3 segments left of total size: 88 bytes
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merged 3 segments, 106 bytes to disk to satisfy reduce memory limit
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 1 files, 106 bytes from disk
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 0 segments, 0 bytes from memory into reduce
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Merging 1 sorted segments
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 1 segments left of total size: 96 bytes
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.conf.Configuration.deprecation] mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task:attempt_local1111900714_0001_r_000000_0 is done. And is in the process of committing
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task attempt_local1111900714_0001_r_000000_0 is allowed to commit now
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] Saved output of task 'attempt_local1111900714_0001_r_000000_0' to hdfs://192.168.56.202:9000/user/root/datasort/output/_temporary/0/task_local1111900714_0001_r_000000
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] reduce > reduce
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1111900714_0001_r_000000_0' done.
2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1111900714_0001_r_000000_0
2019-08-31 22:22:37 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] reduce task executor complete.
2019-08-31 22:22:38 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job]  map 100% reduce 100%
2019-08-31 22:22:38 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Job job_local1111900714_0001 completed successfully
2019-08-31 22:22:38 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Counters: 35
	File System Counters
		FILE: Number of bytes read=4001
		FILE: Number of bytes written=1098948
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=109
		HDFS: Number of bytes written=55
		HDFS: Number of read operations=37
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=10
	Map-Reduce Framework
		Map input records=10
		Map output records=10
		Map output bytes=80
		Map output materialized bytes=118
		Input split bytes=366
		Combine input records=0
		Combine output records=0
		Reduce input groups=10
		Reduce shuffle bytes=118
		Reduce input records=10
		Reduce output records=10
		Spilled Records=20
		Shuffled Maps =3
		Failed Shuffles=0
		Merged Map outputs=3
		GC time elapsed (ms)=8
		Total committed heap usage (bytes)=1486880768
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=34
	File Output Format Counters 
		Bytes Written=55

查看运行结果：

    数据排序也有点词频统计的感觉，只不过我们无需关注词的次数具体是多少，但是我们需要利用这个次数来输出重复的记录。

MapReduce案例9——多个数字文件的数据排序并添加序号(添加可并行方法）
jin6872115的博客
03-17 3406
题目：数字排序并加序号源数据： 2 32 654 32 15 756 65223 5956 22 650 92 26 54 6 最张结果： 1 2 2 6 3 15 4 22 5 26 6 32 7 32 8 54 9 92 10 650 11 654 12 756 13 5956 14 65223 一定要考虑 当数据量一大的时候， 你的实现思路能否使用。解题思路：当有多...
【hadoop学习之路】MapReduce实现数字排序
onion23的博客
05-20 4322
参考视频：【狂野大数据】一天搞定大数据之MapReduce 目录 1. MapReduce 运行流程 2. 规划自定义的 MapReduce 程序 3. 代码实现 1. MapReduce 运行流程 MapReduce主要经过三个阶段： ① Map阶段 1）以<k1, v1>格式读取文本文件，其中k1为文本偏移量，通常定义为数字类型（如LongWritable），v2为文本，通常定义为Text类型 2）自定义语句，用于处理<k1, v1>及生成<k2, v.
idea编写mapreduceTopN程序,运用hdfs上的数据测试并将结果存到hdfs
Late_whale的博客 作者：代码搬运工老李的二师兄
12-10 1444
问题：选出蜀国中五虎将 原始数据如下： 序号 姓名 武力值 国家 1 刘备 68 蜀国 2 马超 90 蜀国 3 黄忠 91 蜀国 4 魏延 76 蜀国 5 姜维 92 蜀国 6 关羽 96 蜀国 7 严颜 78 蜀国 8 孟达 64 蜀国 9 张飞 88 蜀国 10马谡 76 蜀国 11 赵云 95 蜀国 12 法正 88 蜀国 预期结果如下： 6 关羽 96 蜀国 11 赵云 95 蜀国 5 ...
MapReduce程序之数据排序
weixin_34320724的博客
03-08 797
[toc] MapReduce程序之数据排序 需求 下面有三个文件： yeyonghao@yeyonghaodeMacBook-Pro:~/data/input/sort$ cat file1.csv 2 32 654 32 15 756 65223 yeyonghao@yeyonghaodeMacBook-Pro:~/data/input/sort$ cat file2.csv 5956 22...
mapreduce编程（一）－二次排序
热门推荐
和大黄的博客
09-17 1万+
mr自带的例子中的源码SecondarySort，我重新写了一下，基本没变。 这个例子中定义的map和reduce如下，关键是它对输入输出类型的定义：（java泛型编程）  public static class Map extends Mapper public st
大数据实验四-MapReduce编程实践
04-03
### 大数据实验四-MapReduce编程实践 #### 一、实验内容与目的 ##### 实验内容概述 本次实验的主要内容是使用MapReduce框架来实现WordCount词频统计功能，即统计HDFS（Hadoop Distributed File System）系统中多个...
基于 Hadoop 平台，使用 MapReduce 编程，统计NBA球员五项数据.zip
03-13
总结起来，这个项目涵盖了 Hadoop 分布式系统的基础知识，包括 MapReduce 模型、数据存储（HDFS）、资源管理（YARN）以及数据处理和分析。通过实际操作，我们可以深入了解大数据处理的流程，并学习如何利用 Hadoop ...
实验项目 MapReduce 编程
04-18
实验项目“MapReduce 编程”旨在让学生深入理解并熟练运用MapReduce编程模型，这是大数据处理领域中的核心技术之一。实验内容涵盖了从启动全分布模式的Hadoop集群到编写、运行和分析MapReduce应用程序的全过程。 ...
MapReduce案例编程开发，详情参考压缩文件展开图
最新发布
12-14
基础实战主要是指通过实际案例来掌握MapReduce编程开发的技能。在MapReduce编程模型中，Map函数负责处理输入数据，输出键值对，然后Reduce函数对所有具有相同键的值进行合并处理。 MapReduce案例编程开发通常涉及...
MapReduce编程实例：单词计数
01-07
本节介绍如何编写基本的 MapReduce 程序实现数据分析。本节代码是基于 Hadoop 2.7.3 开发的。 任务准备 单词计数（WordCount）的任务是对一组输入文档中的单词进行分别计数。假设文件的量比较大，每个文档又包含...
mapreduce的顺序流程
qq_38976693的博客
09-21 1160
MapReduce 详解
草原孤狼的专栏
04-26 2075
函数式编程概念 　　MapReduce程序是设计用来并行计算大规模海量数据的，这需要把工作流分划到大量的机器上去，如果组件(component)之间可以任意的共享数据，那这个模型就没法扩展到大规模集群上去了（数百或数千个节点），用来保持节点间数据的同步而产生的通信开销会使得系统在大规模集群上变得不可靠和效率低下。 　　实际上，所有在MapReduce上的数据元素都是不可变的，这就意味着它们
MapReduce的编程开发——排序
Comet_sgf的程序员之路
01-30 2001
本文章主要概述了利用MapReduce进行排序的流程以及实验代码，例举了普通排序、二次排序以及倒排索引排序三种排序情况
Hadoop MapReduce数据流_亮仔_新浪博客
jingquanliang的专栏
07-03 177
转载一篇文章，感觉很不错，原文出处：http://www.linuxidc.com/Linux/2012-02/54485p2.htm Hadoop的核心组件在一起工作时如下图所示： 图4.4高层MapReduce工作流水线 　　MapReduce的输入一般来自HDFS中的文件，这些文件分布存储在集群内的节点上。运行一个MapReduce程序会在集群的许多节点甚至所有节点上运行...
头歌-第1关：MapReduce的编程开发-排序
2301_79646631的博客
04-02 4150
在 MapReduce 的 Shuffle 的过程中执行了三次排序，分别是： map 中的溢写阶段：根据分区以及 key 进行快速排序。map 中合并溢写文件：将同一分区的多个溢写文件进行归并排序，合成一个大的溢写文件。reduce 输入阶段：将同一分区，来自不同 map task 的数据文件进行归并排序。在 MapReduce 整个过程中，默认是会对输出的键值对按照 key 进行排序的，而且是使用快速排序。map 输出的排序的，其实也就是上面的溢写过程中的排序。
MapReduce之基本数据类的排序
在记录自己编程工作与编程生活的同时，给那些困于无法入门编程世界的朋友提供一点帮助，在有限的生命中给有需要的人尽一点绵薄之力。
05-17 981
MapReduce之基本数据类的排序一、需求说明二、测试数据三、编程思路四、实现步骤四、打包上传到集群中运行 一、需求说明 要求：将各个部门降序排列，利用MapReduce实现将emp.csv中的部门倒序排列 二、测试数据 员工信息表：下载地址 表字段说明： 三、编程思路 因在MapReduce中基本数据类型（如int）默认是升序排序的，因此我们只需要写一个类继承IntWritable.Comparator，重写compare方法即可 四、实现步骤 在Idea或eclipse中创建
java mapreduce实例_Mapreduce实例——排序
weixin_42465953的博客
02-26 993
原理Map、Reduce任务中Shuffle和排序的过程图如下：流程分析：1.Map端：(1)每个输入分片会让一个map任务来处理，默认情况下，以HDFS的一个块的大小(默认为64M)为一个分片，当然我们也可以设置块的大小。map输出的结果会暂且放在一个环形内存缓冲区中(该缓冲区的大小默认为100M，由io.sort.mb属性控制)，当该缓冲区快要溢出时(默认为缓冲区大小的80%，由io.sort...
mapreduce之数据排序
weixin_55885866的博客
05-06 191
/写入的目录应该是空的，否则报错。//reduce输出value。//2.创建Job，设置入口。//6.写出结果到hdfs。//5.进行reducer计。//1.连接hadoop。//4.进行mapper计算。//reduce输出key。//3.读取hdfs的数。//map输出value。//从小到大输出每个数字。//map输出 key。//每行 <成绩，1>
MapReduce编程模型在大数据处理中的应用
"大数据开发之走进MapReduce" MapReduce是大数据处理领域中的一个核心编程模型，由Google提出并广泛应用于Hadoop框架中。它为处理和生成大规模数据集提供了一种分布式计算方法。MapReduce模型主要由两个阶段组成：...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

luffy5459

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值