Hadoop自己编写程序WordCount.java 手动编译程序

最新推荐文章于 2022-07-24 14:52:48 发布

原创最新推荐文章于 2022-07-24 14:52:48 发布 · 1.7k 阅读

6 ·

CC 4.0 BY-SA版权

大数据处理技术专栏收录该内容

5 篇文章

订阅专栏

本文详细介绍如何在Hadoop环境下实现WordCount程序，包括环境变量配置、Java程序编写、编译打包、运行及结果查看等步骤，适用于初学者快速上手。

假设：Hadoop已安装好。Hadoop安装步骤：https://blog.youkuaiyun.com/LiuHuan_study/article/details/84347262

第一步：配置环境变量

export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64/
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
export CLASSPATH=.:${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.7.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-2.7.7.jar:${CLASSPATH}

可以以上配置放在cat ~/.bashrc中，每次启动服务器时，环境变量都有效。

或者放在一个启动文件vi /root/hadoop-env.sh中，每次服务器重启都需要，先执行这个脚本文件

或者，每次启动Hadoop时，手动执行一遍上诉命令

第二步：编写WordCount.java程序

WordCount.java

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
      }
    }
  }


  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }


  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);

  }

}

第三步：借助Xshell工具上传到服务器，然后进行编译

上传命令：rz

我上传的路径为：/home/

编译：

#编译，打包jar文件。因为编译后可能会生成多个.class文件，所以需要加星号

javac WordCount.java

jar cvf WordCount.jar WordCount*class

第四步：使用WordCount.jar包，计算某一文档或者某一路径下（该路径下只有文档，没有文件夹）所有文档中单词的个数

/LiuHuan统计的路径；/output统计结果输出路径。

hadoop jar /home/WordCount.jar WordCount /LiuHuan /output

#或者

hadoop jar /home/WordCount.jar WordCount /LiuHuan/test.txt /output

注意：保证路径/output在HDFS中不存在，否则会报错。

第五步：查看统计结果

[root@localhost home]# hdfs dfs -cat /output/part-r-00000
Huan    2
Liu    2
hello    1
world!    1