Hadoop WordCount

最新推荐文章于 2024-11-19 22:39:34 发布

原创最新推荐文章于 2024-11-19 22:39:34 发布 · 315 阅读

0 ·

CC 4.0 BY-SA版权

Hadoop 专栏收录该内容

24 篇文章

订阅专栏

本文介绍了一个基于Hadoop的单词计数程序实现，通过MapReduce框架处理文本文件中的单词并进行计数。该程序包括主类MainFunc、映射类Map及归约类Reduce，演示了如何设置Hadoop作业、读取输入文件、定义映射和归约操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

版本一：

默认使用Split类方法去做，这样做就是默认一行一行去解析，map阶段 key传入的为行标，value传入的为这一行的值

这种情况只适合于有超多行，但每一行不太长的情况

下面贴代码:

文本文件words.txt

aaa bbb
ccc ddd eee fff

文本文件words2.txt

aa
bb aaaa aaa
ccc casdasd 11 2
3 11 222 3 1
2

MainClass:

package wordcnt;

import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class MainFunc extends Configured implements Tool {

	public static void main(String[] args) throws Exception {
		int ret = ToolRunner.run(new MainFunc(), args);
		System.exit(ret);
	}

	@Override
	public int run(String[] args) throws Exception {
		// TODO Auto-generated method stub

		Job job = new Job(getConf());
		job.setJarByClass(MainFunc.class);
		job.setJobName("wordcount");

		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(IntWritable.class);

		job.setMapperClass(Map.class);
		job.setReducerClass(Reduce.class);

		job.setInputFormatClass(TextInputFormat.class);
		job.setOutputFormatClass(TextOutputFormat.class);

		FileInputFormat.setInputPaths(job, new Path("src/wordcnt/words.txt"),
				new Path("src/wordcnt/words2.txt"));
		FileOutputFormat.setOutputPath(job, new Path("rst"));

		boolean success = job.waitForCompletion(true);

		return success ? 0 : 1;
	}

}

Map:

package wordcnt;

import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class Map extends Mapper<LongWritable, Text, Text, IntWritable> {

	private final static IntWritable one = new IntWritable(1);
	private Text word = new Text();

	public void map(LongWritable key, Text value, Context context)
			throws IOException, InterruptedException {
		String line = value.toString();
		StringTokenizer tokenizer = new StringTokenizer(line);
		while (tokenizer.hasMoreTokens()) {
			word.set(tokenizer.nextToken());
			context.write(word, one);
		}

	}

}

Reduce:

package wordcnt;

import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {

	public void reduce(Text key, Iterable<IntWritable> values, Context context)
			throws IOException, InterruptedException {
		int sum = 0;

		for (IntWritable val : values) {
			sum += val.get();
		}
		context.write(key, new IntWritable(sum));
	}

}

结果:

1	1
11	2
2	2
222	1
3	2
aa	1
aaa	2
aaaa	1
bb	1
bbb	1
casdasd	1
ccc	2
ddd	1
eee	1
fff	1