wordcount完整总结

最新推荐文章于 2022-03-07 22:55:47 发布

weixin_40839731

最新推荐文章于 2022-03-07 22:55:47 发布

阅读量1k

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/weixin_40839731/article/details/82715758

本文详细介绍使用Hadoop实现WordCount的过程，包括在IDEA中创建Maven项目、编写WordCountMapper和WordCountReducer类、Driver类的设置，以及如何在虚拟机上运行打包后的jar文件。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.idea创建maven项目(百度查找)

2.编写wordcount代码

WordCountMapper类

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class WordCountMapper extends Mapper <LongWritable,Text, Text, LongWritable>{
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        //获得一行数据
        String line = value.toString();
        //切分
        String[] words = line.split(" ");
        //循环遍历单词
        for (String word : words) {
            context.write(new Text(word), new LongWritable(1));
        }
    }
}

WordCountReducer类

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

public class WordCountReducer extends Reducer<Text, LongWritable, Text, LongWritable> {

    @Override
    protected void reduce(Text key, Iterable<LongWritable> value,Context context)
            throws IOException, InterruptedException {
        // TODO Auto-generated method stub
        //计数
        int count = 0;
        //值累加
        for (LongWritable l : value) {
            count += l.get();
        }
        context.write(key, new LongWritable(count));
    }
}

Driver类

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;
public class Driver {
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        // 配置信息
        Configuration conf = new Configuration();
        //跑一个job
        Job job = Job.getInstance(conf);
        //设置程序入口
        job.setJarByClass(Driver.class);
        //指定mapper
        job.setMapperClass(WordCountMapper.class);
        job.setMapOutputKeyClass(Text .class);
        job.setMapOutputValueClass(LongWritable .class);
        //指定reducer
        job.setReducerClass(WordCountReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);

        //指定输入路径和输出路径
        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        //提交作业
            job.waitForCompletion(true);
    }

}

3.idea打包成jar包

https://www.cnblogs.com/blog5277/p/5920560.html

4.虚拟机运行wordcount

hadoop jar 打成的jar包名输入路径输出路径

举例: hadoop jar /aa.txt /output