使用MapReduce计算HDFS中每年最高温度-优快云博客

该博客详细介绍了如何在Hadoop环境下利用MapReduce计算大数据中每年的最大温度。首先，需要安装Hadoop、IDEA和Maven。接着，数据预处理生成wc.txt文件，包含年份和对应温度，存储在HDFS的/input/目录下。然后，通过Java编写MapReduce程序，包括Mapper和Reducer两个部分。Mapper负责解析数据并发送键值对，Reducer则找出每个年份的最大温度。最后，执行MapReduce任务得到结果。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

前提：

安装hadoop、IDEA、maven

具体操作：（1）准备数据 wc.txt(格式-12345年份12345温度12345，放在hdfs中的/input/中)：

123452015123453012345
123452015123453212345
123452016123453812345
123452016123453812345
123452021123453612345
123452021123453812345
123452021123453612345

（2）Java代码、pom：

POM:

<dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.6.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.6.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.6.5</version>
        </dependency>
    </dependencies>

Mapper：

package com.demo.mapper;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
public class MaxTemperatureMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        String year = line.substring(5, 9);
        int airTemperature = Integer.valueOf(line.substring(14, 16));
        context.write(new Text(year), new IntWritable(airTemperature));
    }
}

Reducer：

package com.demo.reduer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
public class MaxTemperatureReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    @Override
    protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int maxValue = Integer.MIN_VALUE;
        for (IntWritable value : values) {
            maxValue = Math.max(maxValue, value.get());
        }
        context.write(key, new IntWritable(maxValue));
    }
}

（3）执行与结果

hadoop jar newWordCount.jar com.demo.MaxTemperature /input/wc.txt /output/

2015	32
2016	38
2021	38

MR计算每年最大温度示例