IDEA编写Hadoop WordCount实例

最新推荐文章于 2024-06-11 20:12:19 发布

原创最新推荐文章于 2024-06-11 20:12:19 发布 · 484 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop

Hadoop 专栏收录该内容

1 篇文章

订阅专栏

本文档介绍了如何在IntelliJ IDEA中使用Maven创建Hadoop项目，并实现WordCount应用程序。首先，配置pom.xml文件，引入Hadoop相关依赖。接着，编写WordCount的Mapper和Reducer类。最后，通过main方法设置作业配置并提交任务。文章还提供了运行项目的Maven打包及执行命令。

IDEA编写Hadoop WordCount实例

使用Idea下Maven创建项目

pom文件配置

hadoop.version对于集群版本

    <properties>
        <!-- 开发阶段配置文件-->
        <hadoop.version>2.7.3</hadoop.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common </artifactId>
            <version >${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version >${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version >${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.10</version>
        </dependency>
    </dependencies>

WordCount代码

public class WordCountApp {
    public static class MyMapper extends Mapper<Object, Text, Text, IntWritable>{
        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            while (itr.hasMoreTokens()){
                word.set(itr.nextToken());
                context.write(word, one);
            }
        }
    }

    public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable>{
        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable value : values) {
                sum += value.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }


    public static void main(String[] args) throws Exception {
        String INPUT_PATH = "hdfs://master:9000/input/1.txt";
        String OUTPUT_PATH =  "hdfs://master:9000/output";

        Configuration conf = new Configuration();
        final FileSystem fileSystem = FileSystem.get(new URI(INPUT_PATH), conf);
        if(fileSystem.exists(new Path(OUTPUT_PATH))){
            fileSystem.delete(new Path(OUTPUT_PATH), true);
        }

        Job job = Job.getInstance(conf, "WordCountApp");

        job.setJarByClass(WordCountApp.class);

        job.setMapperClass(MyMapper.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        job.setReducerClass(MyReducer.class);
        job.setOutputKeyClass(Text.class) ;
        job.setOutputValueClass(IntWritable.class);

        job.setInputFormatClass(TextInputFormat.class);
        Path inputPath =new Path(INPUT_PATH) ;
        FileInputFormat.addInputPath(job, inputPath);

        job.setOutputFormatClass(TextOutputFormat.class);
        Path outputPath = new Path(OUTPUT_PATH);
        FileOutputFormat.setOutputPath(job, outputPath);

        System.exit(job.waitForCompletion(true)? 0: 1);
    }
}

可以直接使用Maven直接进行打包
在这里插入图片描述

运行命令

hadoop jar 包名.jar com.*.hadoop.需要运行类名