Hadoop采样器实现全排序（报错java.io.EOFException）

最新推荐文章于 2021-05-10 12:28:56 发布

转载最新推荐文章于 2021-05-10 12:28:56 发布 · 280 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/flyingcr/p/10326958.html

文章标签：

#java #大数据

本文介绍使用Hadoop实现全排序分区的过程，包括Mapper、Reducer类的编写，以及如何通过内置全排序分区类进行自动分区。同时，文章还分享了解决序列文件创建及随机采样对象创建中遇到的问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

利用采样器，mapreducer自动将数据按照从大到小的顺序，根据数据分布的概率，自动分区到不同的区域，之前我们是手动设置分区的范围，将数据分区到不同的分区

点击打开链接

下面我们采用Hadoop内置类-全排序分区类进行自动分区

1、mapper类

package com.cr.wordcount;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class MaxTempMapper extends Mapper<LongWritable, IntWritable, LongWritable, IntWritable> {

    @Override
    protected void map(LongWritable key, IntWritable value, Context context) throws IOException, InterruptedException {
        context.write(key, value);
    }
}

2、reducer类

package com.cr.wordcount;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

public class MaxTempReducer extends Reducer<LongWritable,IntWritable,LongWritable,IntWritable> {
    @Override
    protected void reduce(LongWritable key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {

        int max = Integer.MIN_VALUE;
        for(IntWritable iw :values){
            max = max > iw.get()? max : iw.get();
        }
        context.write(key,new IntWritable(max));

    }
}

3、全排序采样器主类

这里有两个需要注意的地方就是先创建随机采样对象，然后再写入分区文件，然后设置全排序分区类

另外一个地方需要注意的是job.getConfiguration()注意这里的conf 不是之前的new conf() 是通过job.getConfiguration()

package com.cr.wordcount;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.partition.InputSampler;
import org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner;

import java.io.IOException;

public class MaxTempApp {
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        //单例作业
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS","file:///");
        Job job = Job.getInstance(conf);

        //设置job的各种属性
        job.setJobName("MaxTempApp");                 //设置job名称
        job.setJarByClass(MaxTempApp.class);              //设置搜索类
        job.setInputFormatClass(SequenceFileInputFormat.class);

        //设置输入路径
        FileInputFormat.addInputPath(job,new Path((args[0])));
        //设置输出路径
        FileOutputFormat.setOutputPath(job,new Path(args[1]));

        job.setMapperClass(MaxTempMapper.class);               //设置mapper类
        job.setReducerClass(MaxTempReducer.class);               //设置reduecer类

        job.setMapOutputKeyClass(LongWritable.class);            //设置之map输出key
        job.setMapOutputValueClass(IntWritable.class);   //设置map输出value
        job.setOutputKeyClass(LongWritable.class);               //设置mapreduce 输出key
        job.setOutputValueClass(IntWritable.class);      //设置mapreduce输出value
        //创建随机采样对象
        /**
         * RandomSampler
         * 1:每个key被选中的概率
         * 6000：抽取样本的总数
         * 3：最大采样切片数 分区数
         */
        InputSampler.Sampler<LongWritable,IntWritable> sampler =
                new InputSampler.RandomSampler<LongWritable, IntWritable>(1,6000,3);

        job.setNumReduceTasks(3);                         //设置reduce个数

        //将sample数据写入分区文件
        /**
         * job.getConfiguration()注意这里的conf 不是之前的new conf() 是通过job.getConfiguration()
         */
        TotalOrderPartitioner.setPartitionFile(job.getConfiguration(),new Path("D:/sample/par.list"));
        //设置全排序分区类
        job.setPartitionerClass(TotalOrderPartitioner.class);

        InputSampler.writePartitionFile(job,sampler);
        job.waitForCompletion(true);

    }
}

4、创建序列文件作为数据输入

package com.cr.wordcount;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.junit.Test;

import java.io.FileWriter;
import java.io.IOException;
import java.util.Random;

public class SequenceFile {

    /**
     * 写入文件
     *
     * @throws IOException
     */
    @Test
    public void save() throws IOException {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "file:///");
        FileSystem fs = FileSystem.get(conf);
        Path path = new Path("D:\\sequence\\1.seq");
        org.apache.hadoop.io.SequenceFile.Writer writer = org.apache.hadoop.io.SequenceFile.createWriter(fs, conf, path, IntWritable.class, IntWritable.class);
        for (int i = 0; i < 6000; i++) {

            int year = 2000 + new Random().nextInt(50);
            int temp = 10 + new Random().nextInt(30);
            writer.append(new IntWritable(year),new IntWritable(temp));
        }
    }
}

报错java.io.EOFException，还未解决

Exception in thread "main" java.io.EOFException
	at java.io.DataInputStream.readFully(DataInputStream.java:197)
	at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
	at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
	at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2436)
	at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2568)
	at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:72)
	at org.apache.hadoop.mapreduce.lib.partition.InputSampler$RandomSampler.getSample(InputSampler.java:222)
	at org.apache.hadoop.mapreduce.lib.partition.InputSampler.writePartitionFile(InputSampler.java:320)
	at com.cr.wordcount.MaxTempApp.main(MaxTempApp.java:57)

有想法的盆友能否提点意见呢

好了，上述问题已经解决，吐舌头

问题出在了以下几个地方

1、序列文件的问题,最后结尾的时候没有关闭write（），导致生成的序列文件有问题
2、创建随机采样对象的时候应该将longwritable改成intwritable，因为我读取的是序列文件，类型应该都是int类型
3、这里的mappper和mapreducer的输出类型也应该都是intwritable，而不是longwritable
4、相应的mapper和reducer里面的输入和输出也都应该是intwritable

真的搞得头大，以后一定要注意这些细节

5、运行结果

共产生3个分区文件，每个分区的范围自动生成，按照年份从小到大顺序生成

part-r-00000

part-r-00001

part-r-00002

转载于:https://www.cnblogs.com/flyingcr/p/10326958.html