下载气象数据集部分数据，写一个Map-Reduce作业，求每年的最低温度

最新推荐文章于 2024-05-30 08:02:02 发布

weixin_33755554

最新推荐文章于 2024-05-30 08:02:02 发布

阅读量601

点赞数 1

CC 4.0 BY-SA版权

文章标签：大数据 java python

原文链接：https://my.oschina.net/zc741520/blog/368330

本文介绍了一个使用Hadoop进行温度数据处理的MapReduce程序。通过设置HADOOP_PREFIX环境变量配置Hadoop环境，并提供了完整的Java代码实现，包括Mapper和Reducer类，用于找出特定年份的最低气温。

2019独角兽企业重金招聘Python工程师标准>>>

环境：CentOS6.6 Hadoop1.2.1

1.设置 HADOOP_PREFIX 环境变量

[grid@hadoop1 ~]$ cat .bash_profile 
HADOOP_PREFIX=/home/grid/hadoop-1.2.1
PATH=$PATH:$HOME/bin:$HADOOP_PREFIX/bin
export HADOOP_PREFIX PATH

2.准备天气数据

3.解压合并

4.上传到HDFS

5.编写JAVA文件

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
public class MinTemperatureMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private static final int MISSING = -9999;
    @Override
    public void map(LongWritable key, Text value, Context context)
            throws IOException, InterruptedException {
        String line = value.toString();
        String year = line.substring(0, 4);
        int airTemperature = Integer.parseInt(line.substring(13, 19).trim());
        if (airTemperature != MISSING) {
            context.write(new Text(year), new IntWritable(airTemperature));
        }
    }
}

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
public class MinTemperatureReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    @Override
    public void reduce(Text key, Iterable<IntWritable> values, Context context)
            throws IOException, InterruptedException {
        int minValue = Integer.MAX_VALUE;
        for (IntWritable value : values) {
            minValue = Math.min(minValue, value.get());
        }
        context.write(key, new IntWritable(minValue));
    }
}

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.Job;
public class MinTemperature {
    public static void main(String[] args) throws Exception {
        if (args.length != 2) {
            System.err.println("Usage: MinTemperature <input path> <output path>");
            System.exit(-1);
        }
        Job job = new Job();
        job.setJarByClass(MinTemperature.class);
        job.setJobName("Min temperature");
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        job.setMapperClass(MinTemperatureMapper.class);
        job.setCombinerClass(MinTemperatureReducer.class);
        job.setReducerClass(MinTemperatureReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

6.编译并打成JAR包，删除编译后的class文件