Hadoop的序列化机制

最新推荐文章于 2022-12-23 21:25:57 发布

sun_0128

最新推荐文章于 2022-12-23 21:25:57 发布

阅读量928

点赞数 3

分类专栏： Hadoop 文章标签： mapreduce hadoop 大数据 java

本文链接：https://blog.youkuaiyun.com/sun_0128/article/details/107041082

版权

Hadoop 专栏收录该内容

32 篇文章

订阅专栏

文章目录

一.什么是序列化和反序列化
二.Hadoop的序列化
三.Hadoop的序列化案例

一.什么是序列化和反序列化

序列化:将对象转化为字节流,以便在网络上传输或者写在磁盘磁盘上进行永久存储
反序列化:将字节流转回成对象
序列化在分布式数据处理的两个领域经常出现: 进程间通信和永久储存
Hadoop中多个节点进程间通信是通过远程过程调用(Remote Procedure Call,RPC) 实现的

二.Hadoop的序列化

Hadoop的序列化并不采用Java的序列化,而是采用自己的序列化机制.在hadoop序列化机制中,用户可以复用对象,减少了Java对象的分配和回收,提高了应用效率
Hadoop通过Writable接口实现序列化机制但没有提供比较功能,所以和Java的Comparable接口合并,提供了一个WritableComparable接口

三.Hadoop的序列化案例

功能需求:先读一个流量统计文件到MapReduce中统记某手机号上行总流量,下行总流量,和总流量和,最后都输出到一个文件中(对象形式保存)
所需的文件链接: https://pan.baidu.com/s/13dqp3rk_sgq-3pRqVQg0OQ 提取码: abu3
流量文件内容如下:

1363157985066 	13726230503	00-FD-07-A4-72-B8:CMCC	120.196.100.82	i02.c.aliimg.com		24	27	2481	24681	200
1363157995052 	13826544101	5C-0E-8B-C7-F1-E0:CMCC	120.197.40.4			4	0	264	0	200
1363157991076 	13926435656	20-10-7A-28-CC-0A:CMCC	120.196.100.99			2	4	132	1512	200
1363154400022 	13926251106	5C-0E-8B-8B-B1-50:CMCC	120.197.40.4			4	0	240	0	200
1363157993044 	18211575961	94-71-AC-CD-E6-18:CMCC-EASY	120.196.100.99	iface.qiyi.com	视频网站	15	12	1527	2106	200
1363157995074 	84138413	5C-0E-8B-8C-E8-20:7DaysInn	120.197.40.4	122.72.52.12		20	16	4116	1432	200
1363157993055 	13560439658	C4-17-FE-BA-DE-D9:CMCC	120.196.100.99			18	15	1116	954	200
1363157995033 	15920133257	5C-0E-8B-C7-BA-20:CMCC	120.197.40.4	sug.so.360.cn	信息安全	20	20	3156	2936	200
1363157983019 	13719199419	68-A1-B7-03-07-B1:CMCC-EASY	120.196.100.82			4	0	240	0	200
1363157984041 	13660577991	5C-0E-8B-92-5C-20:CMCC-EASY	120.197.40.4	s19.cnzz.com	站点统计	24	9	6960	690	200
1363157973098 	15013685858	5C-0E-8B-C7-F7-90:CMCC	120.197.40.4	rank.ie.sogou.com	搜索引擎	28	27	3659	3538	200
1363157986029 	15989002119	E8-99-C4-4E-93-E0:CMCC-EASY	120.196.100.99	www.umeng.com	站点统计	3	3	1938	180	200
1363157992093 	13560439658	C4-17-FE-BA-DE-D9:CMCC	120.196.100.99			15	9	918	4938	200
1363157986041 	13480253104	5C-0E-8B-C7-FC-80:CMCC-EASY	120.197.40.4			3	3	180	180	200
1363157984040 	13602846565	5C-0E-8B-8B-B6-00:CMCC	120.197.40.4	2052.flash2-http.qq.com	综合门户	15	12	1938	2910	200
1363157995093 	13922314466	00-FD-07-A2-EC-BA:CMCC	120.196.100.82	img.qfc.cn		12	12	3008	3720	200
1363157982040 	13502468823	5C-0A-5B-6A-0B-D4:CMCC-EASY	120.196.100.99	y0.ifengimg.com	综合门户	57	102	7335	110349	200
1363157986072 	18320173382	84-25-DB-4F-10-1A:CMCC-EASY	120.196.100.99	input.shouji.sogou.com	搜索引擎	21	18	9531	2412	200
1363157990043 	13925057413	00-1F-64-E1-E6-9A:CMCC	120.196.100.55	t3.baidu.com	搜索引擎	69	63	11058	48243	200
1363157988072 	13760778710	00-FD-07-A4-7B-08:CMCC	120.196.100.82			2	2	120	120	200
1363157985066 	13560436666	00-FD-07-A4-72-B8:CMCC	120.196.100.82	i02.c.aliimg.com		24	27	2481	24681	200
1363157993055 	13560436666	C4-17-FE-BA-DE-D9:CMCC	120.196.100.99			18	15	1116	954	200

代码如下:
1)流量统计实体类

package hadoop.hdfs.flowcount;

import org.apache.hadoop.io.Writable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

/**
 * @author sunyong
 * @date 2020/07/01
 * @description
 */
 //实现Writable接口才可序列化和反序列化
public class FlowBean implements Writable {
    private long upFlow;//上行流量 文件的倒数第三列
    private long downFlow;//下载流量 文件的倒数第二列
    private long sumFlow;//总流量 自定义的

    /**
     * 序列化
     * @param dataOutput
     * @throws IOException
     */
    @Override
    public void write(DataOutput dataOutput) throws IOException {
        dataOutput.writeLong(upFlow);
        dataOutput.writeLong(downFlow);
        dataOutput.writeLong(sumFlow);
    }

    /**
     * 反序列化
     * 注意:序列化和反序列化的顺序需要保持一致
     * @param dataInput
     * @throws IOException
     */
    @Override
    public void readFields(DataInput dataInput) throws IOException {
        this.upFlow = dataInput.readLong();
        this.downFlow = dataInput.readLong();
        this.sumFlow = dataInput.readLong();
    }

    public FlowBean() {
    }

    public FlowBean(long upFlow, long downFlow, long sumFlow) {
        this.upFlow = upFlow;
        this.downFlow = downFlow;
        this.sumFlow = sumFlow;
    }

    public long getUpFlow() {
        return upFlow;
    }

    public void setUpFlow(long upFlow) {
        this.upFlow = upFlow;
    }

    public long getDownFlow() {
        return downFlow;
    }

    public void setDownFlow(long downFlow) {
        this.downFlow = downFlow;
    }

    public long getSumFlow() {
        return sumFlow;
    }

    public void setSumFlow(long sumFlow) {
        this.sumFlow = sumFlow;
    }
    //自己创建一个set方法,用于map和reduce设置
    public void set(long upFlow,long downFlow){
        this.downFlow=downFlow;
        this.upFlow=upFlow;
        this.sumFlow=upFlow+downFlow;
    }
    //tostring方法方便统计
    @Override
    public String toString() {
        return "FlowBean{" +
                "upFlow=" + upFlow +
                ", downFlow=" + downFlow +
                ", sumFlow=" + sumFlow +
                '}';
    }
}

2)Mapper类

package hadoop.hdfs.flowcount;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

/**
 * @author sunyong
 * @date 2020/07/01
 * @description
 */
 //定义输入键为LongWritable,值为 Text,输出为键是Text,值是FlowBean
public class FlowMapper extends Mapper<LongWritable, Text,Text,FlowBean> {
    Text k = new Text();
    FlowBean v = new FlowBean();
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        //1.将文本转化成字符串
        String line = value.toString();
        //2.将字符串切割
        String[] fields = line.split("\\s+");
        //3.执行我们的业务逻辑
        //取出手机号
        String phoneNum = fields[1];
        //取出上行流量和下行流量
        long upFlow=Long.parseLong(fields[fields.length-3]);
        long downFlow=Long.parseLong(fields[fields.length-2]);
        //4.map端输出
        k.set(phoneNum);
        v.set(upFlow,downFlow);
        //注意写的是对象
        context.write(k,v);
    }
}

3)Reduce统合所有数据

package hadoop.hdfs.flowcount;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

/**
 * @author sunyong
 * @date 2020/07/01
 * @description
 */
  //定义输入键为Text,值为 FlowBean,输出为 键是Text,值是FlowBean
public class FlowReducer extends Reducer< Text,FlowBean,Text,FlowBean> {
    FlowBean v = new FlowBean();
    @Override
    protected void reduce(Text key, Iterable<FlowBean> values, Context context) throws IOException, InterruptedException {
       //reduce的输入大概是这样的:("13560439658",FlowBean对象((918,4938),(1116,954)))
        //创建两个初始值用于累加操作
        long sum_upFlow = 0;
        long sum_downFlow=0;
        //执行累加操作
        for (FlowBean flowBean : values) {
            sum_upFlow+=flowBean.getUpFlow();
            sum_downFlow+=flowBean.getDownFlow();
        }
        //将结果写出
        v.set(sum_upFlow,sum_downFlow);
        //写对象
        context.write(key,v);
    }
}

4)Driver类执行Job

package hadoop.hdfs.flowcount;

import hadoop.mapreduce.WCDriver;
import hadoop.mapreduce.WCMapper;
import hadoop.mapreduce.WCReducer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;

/**
 * @author sunyong
 * @date 2020/07/01
 * @description
 */
public class FlowDriver {
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        //1.创建配置文件,创建Job
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf,"wordcount");

        //2.设置jar的位置,加载当前类
        job.setJarByClass(FlowDriver.class);

        //3.设置map和reduce的位置
        job.setMapperClass(FlowMapper.class);
        job.setReducerClass(FlowReducer.class);

        //4.设置map输出端的key,value类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(FlowBean.class);

        //5.设置reduce输出的key,value类型
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(FlowBean.class);

        //6.设置输入和输出路径
        FileInputFormat.setInputPaths(job,new Path("F:\\sunyong\\Java\\codes\\javaToHdfs\\phone_data.txt"));
        FileOutputFormat.setOutputPath(job,new Path("flow"));

        //7.提交程序运行
        boolean result = job.waitForCompletion(true);
        System.exit(result?0:1);
    }
}

5)执行结果如下:
6)打开统计文件如下(里面是一个键为手机号,值为FlowBean对象):

13480253104	FlowBean{upFlow=180, downFlow=180, sumFlow=360}
13502468823	FlowBean{upFlow=7335, downFlow=110349, sumFlow=117684}
13560436666	FlowBean{upFlow=3597, downFlow=25635, sumFlow=29232}
13560439658	FlowBean{upFlow=2034, downFlow=5892, sumFlow=7926}
13602846565	FlowBean{upFlow=1938, downFlow=2910, sumFlow=4848}
13660577991	FlowBean{upFlow=6960, downFlow=690, sumFlow=7650}
13719199419	FlowBean{upFlow=240, downFlow=0, sumFlow=240}
13726230503	FlowBean{upFlow=2481, downFlow=24681, sumFlow=27162}
13760778710	FlowBean{upFlow=120, downFlow=120, sumFlow=240}
13826544101	FlowBean{upFlow=264, downFlow=0, sumFlow=264}
13922314466	FlowBean{upFlow=3008, downFlow=3720, sumFlow=6728}
13925057413	FlowBean{upFlow=11058, downFlow=48243, sumFlow=59301}
13926251106	FlowBean{upFlow=240, downFlow=0, sumFlow=240}
13926435656	FlowBean{upFlow=132, downFlow=1512, sumFlow=1644}
15013685858	FlowBean{upFlow=3659, downFlow=3538, sumFlow=7197}
15920133257	FlowBean{upFlow=3156, downFlow=2936, sumFlow=6092}
15989002119	FlowBean{upFlow=1938, downFlow=180, sumFlow=2118}
18211575961	FlowBean{upFlow=1527, downFlow=2106, sumFlow=3633}
18320173382	FlowBean{upFlow=9531, downFlow=2412, sumFlow=11943}
84138413	FlowBean{upFlow=4116, downFlow=1432, sumFlow=5548}