Yarn核心——Protobuf

最新推荐文章于 2022-04-12 13:08:00 发布

夹克牛

最新推荐文章于 2022-04-12 13:08:00 发布

阅读量638

点赞数

分类专栏： Yarn 文章标签： Yarn

本文链接：https://blog.youkuaiyun.com/u010651137/article/details/77769515

版权

Yarn 专栏收录该内容

3 篇文章

订阅专栏

本文介绍YARN如何利用Protobuf进行RPC通信，并通过示例详细展示了Protobuf的定义、编译及序列化过程。从student.proto文件的定义到序列化后的数据传输，完整地演示了整个流程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Yarn将Protobuf用在RPC通信中，基于Protobuf的方式进行通信。Yarn RPC中的所有参数都采用Protocol Buffer进行序列化和反序列化，体积小，速度快。

(1)Proto文件 student.proto

package tutorial;
option java_package="com.jackniu.hadoop_yarn.com.proto";
option java_outer_classname="StudentProtos";

message Student{
    required int32 ID=1;
    required string name=2;
    required string sex=3;

        message StudentPhone{
        required  string number=1;
        optional int32 type=2;
        }
    repeated StudentPhone phones=4;

}

(2) 编译

bin/protoc  --java_out=javacode  code/student.proto

(3) 编译完成部分代码

package com.jackniu.hadoop_yarn.com.proto;

public final class StudentProtos {
  private StudentProtos() {}
  public static void registerAllExtensions(
      com.google.protobuf.ExtensionRegistry registry) {
  }
  public interface StudentOrBuilder
      extends com.google.protobuf.MessageOrBuilder {

    // required int32 ID = 1;
    /**
     * <code>required int32 ID = 1;</code>
     */
    boolean hasID();
    /**
     * <code>required int32 ID = 1;</code>
     */
    int getID();

(4) 使用

package com.jackniu.hadoop_yarn.com.proto;

/**
 * Created by JackNiu on 2017/9/1.
 */
public class ProtocolBufferExample {
    public static void main(String[] args) {
        StudentProtos.Student student = StudentProtos.Student.newBuilder().setName("jack").setID(1).setSex("2")
                .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("12345").setType(1))
                .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("67890").setType(2))
                .build();

        System.out.println(student);
        // 在发送端发送
     }
}

ID: 1
name: "jack"
sex: "2"
phones {
  number: "12345"
  type: 1
}
phones {
  number: "67890"
  type: 2
}

(5) 发送和读取

package com.jackniu.hadoop_yarn.com.proto;

import org.apache.hadoop.mapred.FileOutputFormat;

import java.io.FileInputStream;
import java.io.FileOutputStream;

/**
 * Created by JackNiu on 2017/9/1.
 */
public class ProtocolBufferExample {
    public static void main(String[] args)  throws  Exception{
        StudentProtos.Student student = StudentProtos.Student.newBuilder().setName("jack").setID(1).setSex("2")
                .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("12345").setType(1))
                .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("67890").setType(2))
                .build();

        System.out.println(student);
        //在发送端发送
        FileOutputStream output = new FileOutputStream("example.txt");
        student.writeTo(output);
        output.close();

        FileInputStream inputStream = new FileInputStream("example.txt");
        StudentProtos.Student receive_student = StudentProtos.Student.parseFrom(inputStream);
        System.out.println(receive_student);
     }
}

输出: example.txt
     jack      2" 
12345    "   
67890

控制台输出
ID: 1
name: "jack"
sex: "2"
phones {
  number: "12345"
  type: 1
}
phones {
  number: "67890"
  type: 2
}

ID: 1
name: "jack"
sex: "2"
phones {
  number: "12345"
  type: 1
}
phones {
  number: "67890"
  type: 2
}