Facebook Wire Protocal Buffer （protobuf）简介

The Corner

Square Engineering Blog

 

Introducing Wire Protocol Buffers

August23, 2013

What is Wire?

Wire is a new, open-sourceimplementation of Google's Protocol Buffers. It'smeant for Android devices but can be used on anything that runs Java languagecode.

AtSquare, we use Protocol Buffers extensively. Protocol Buffers are a language-and platform-neutral schema for describing and transmitting data. Developerscan use the same schemas across diverse environments, including theenvironments we care about at Square, such as Java and Ruby servers and Androidand iOS devices.

ProtocolBuffer .proto sourcefiles are human-readable and can contain comments, so it's easy to documentyour schema right where it is defined. Protocol Buffers define a compact binarywire format that allows schemas to evolve without breaking existing clients.Typically, a code generator is used that reads .proto files, and emits source code inthe language of your choice. This approach helps to speed development since theresulting code is expressed in the same language as the rest of yourapplication, allowing tools such as IDE autocompletion to do their job fully.

Wire Features

As webegan to run into limitations of the standard Protocol Buffer implementation inour Android apps, we made a wish list of the features we wanted for a futureimplementation:

·        Messages should contain a minimumnumber of generated methods

·        Messages should be clean,developer-friendly data objects:

·        They should be highly readable

·        They should be deeply immutable

·        They should have meaningful equals, hashCode, and toString methods

·        They should support the chained Builder pattern

·        They should inherit documentationfrom the .proto sourcefiles

·        Protocol Buffer enums should maponto Java enums

·        Ideally, everything should be buildableusing Java-based tools

Before wedecided to build a new library we looked at several alternatives, including therecent NanoAndroid Protocol Buffer library.While Nano Protocol Buffers generate very few methods, they didn't meet ourother goals. Ultimately, we decided to create our own library from scratch,built on Square's ProtoParser and JavaWriter libraries.

It'shandy to be able to use generated Protocol Buffer classes as full-fledged dataobjects in your app. By including equals, hashCode, and toString methods,messages can participate in Java collections. Since the generated code is cleanand compact, stepping into it in the debugger is not a problem. And becausecomments in your .proto files arecopied into the generated Java source code, the documentation for your messagesand fields is right there in your IDE.

Reducing the Method Count

In thepast, Android developers attempting to use Protocol Buffers have paid a steepprice. The standard Protocol Buffer implementation (protoc) generates at least nine methods for each optional orrequired field in your schema (variants of get, set, has, and clear), and atleast eighteen methods for repeated fields!

Havingall this flexibility is great in non-constrained environments — whatever methodyou need is probably just a few auto-completed keystrokes away. But in Androidenvironments the Dalvik bytecode format imposes a hard limit of 64Kmethods in a single application. For the Square Register app, generatedProtocol Buffer code was taking up a large chunk of our method space. Byswitching to Wire, we removed almost 10,000 methods from our application andgained a lot of breathing room to add new features.

Wire Example

Considerthe classic Person protocolbuffer definition:

messagePerson{

  // The customer's full name.

  requiredstringname=1;

  // The customer's ID number.

  requiredint32id=2;

  // Email address for the customer.

  optionalstringemail=3;

 

  enumPhoneType{

   MOBILE=0;

   HOME=1;

   WORK=2;

  }

 

  messagePhoneNumber{

   // The user's phone number.

   requiredstringnumber=1;

   // The type of phone stored here.

   optionalPhoneTypetype=2[default=HOME];

  }

 

  // A list of the user's phone numbers.

  repeatedPhoneNumberphone=4;

}

The Person class Wire generates is below(the complete generated code is
here):

publicfinalclassPersonextendsMessage{

  /** The customer's full name. */

  @ProtoField(tag=1,type=STRING,label=REQUIRED)

  publicfinalStringname;

 

  /** The customer's ID number. */

  @ProtoField(tag=2,type=INT32,label=REQUIRED)

  publicfinalIntegerid;

 

  /** Email address for the customer. */

  @ProtoField(tag=3,type=STRING)

  publicfinalStringemail;

 

  /**  A list of the user's phonenumbers. */

  @ProtoField(tag=4,label=REPEATED)

  publicfinalList<PhoneNumber>phone;

 

  privatePerson(Builderbuilder){

   super(builder);

   this.name=builder.name;

   this.id=builder.id;

   this.email=builder.email;

   this.phone=immutableCopyOf(builder.phone);

  }

 

  @Overridepublicbooleanequals(Objectother){

   if(!(otherinstanceofPerson))returnfalse;

   Persono=(Person)other;

   returnequals(name,o.name)

       &&equals(id,o.id)

       &&equals(email,o.email)

       &&equals(phone,o.phone);

  }

 

  @OverridepublicinthashCode(){

   intresult=hashCode;

   if(result==0){

     result=name!=null?name.hashCode():0;

     result=result*37+(id!=null?id.hashCode():0);

     result=result*37+(email!=null?email.hashCode():0);

     result=result*37+(phone!=null?phone.hashCode():0);

     hashCode=result;

   }

   returnresult;

  }

 

  publicstaticfinalclassBuilderextendsMessage.Builder<Person>{

   // not shown

  }

}

How it Works

Aninstance of a message class can only be created by a corresponding nested Builder class. Wire generates a singlemethod per field in each builder in order to support chaining:

Personperson=newPerson.Builder()

   .name("Omar")

   .id(1234)

   .email("omar@wire.com")

   .phone(Arrays.asList(newPhoneNumber.Builder()

       .number("410-555-0909")

       .type(PhoneType.MOBILE)

       .build()))

   .build();

Wirereduces the number of generated methods by using a public final field for each message field.Arrays are wrapped, so message instances are deeply immutable. Each field isannotated with a @ProtoField annotationproviding metadata that Wire needs to perform serialization anddeserialization:

@ProtoField(tag=1,type=STRING,label=REQUIRED)

publicfinalStringname;

Use thesefields directly to access your data:

if(person.phone!=null){

  for(PhoneNumberphone:person.phone)

   if(phone.type==PhoneType.MOBILE){

     sendSms(person.name,phone.number,message);

     break;

   }

  }

}

The codeto serialize and deserialize the Person wecreated above looks like this:

byte[]data=person.toByteArray();

Wirewire=newWire();

PersonnewPerson=wire.parseFrom(data,Person.class);

Somefeatures, such as serialization, deserialization, and the toString method are implemented usingreflection. Wire caches reflection information about each message class thefirst time it is encountered for better performance.

Instandard Protocol Buffers, you would call person.hasEmail() to see if an email address hasbeen set. In Wire, you simply check if person.email == null. For repeated fields such asphone, Wire alsorequires your app to get or set a List of PhoneNumber instances all at once, whichsaves a lot of methods.

Wiresupports additional features such as extensions and unknown fields. At present,it lacks support for some advanced features including custom options, services,and runtime introspection of schemas. These can be added in the future as usecases for them on constrained devices emerge.

Wiredeliberately does not support the obsolete 'groups' feature.

Try it out!

Weencourage you to try Wire, contribute to the code, and let us know how it worksin your apps

Wire Protocol Buffers

“A man got to have a code!” - Omar Little

As ourteams and programs grow, the variety and volume of data also grows. Successwill turn your simple data models into complex ones! Whether your applicationis storing data to disk or transmitting it over a network, the structure andinterpretation of that data should be clear. Consumers work best with data theyunderstand!

Schemasdescribe and document data models. If you have data, you should have a schema.

ProtocolBuffers

Google'sProtocol Buffers are built around a great schema language:

• It's cross platform and language independent. Whatever programming language you use, you'll be able to use proto schemas with your application.
• Proto schemas are backwards-compatible and future-proof. You can evolve your schema as your application loses old features and gains new ones.
• It's focused. Proto schemas describe your data models. That's it.

Here's asample message definition:

syntax ="proto2";

 

packagesquareup.dinosaurs;

 

optionjava_package = "com.squareup.dinosaurs";

 

import"squareup/geology/period.proto";

 

messageDinosaur {

  // Common name of this dinosaur, like"Stegosaurus".

  optionalstringname = 1;

 

  // URLs with images of this dinosaur.

  repeatedstringpicture_urls = 2;

 

  optionalsquareup.geology.Periodperiod = 5;

}

Andhere's an enum definition:

syntax ="proto2";

 

packagesquareup.geology;

 

optionjava_package = "com.squareup.geology";

 

enumPeriod {

  // 145.5 million years ago — 66.0million years ago.

  CRETACEOUS = 1;

 

  // 201.3 million years ago — 145.0million years ago.

  JURASSIC = 2;

 

  // 252.17 million years ago — 201.3million years ago.

  TRIASSIC = 3;

}

Thisschema language is Protocol Buffers' best feature. You might even use it purelyfor documentation purposes, such as to describe a JSON API.

Protocol Buffers also defines acompact binary encoding of messages that conform to the schema. This encodingis fast to encode, fast to decode, small to transmit, and small to store. Thebinary encoding uses numeric tags from the schema, like the 5 for period above.

Forexample, let's encode this dinosaur:

{

  name:"Stegosaurus",

  period: JURASSIC

}

Theencoded value is just 15 bytes:

Hex  Description

 0a  tag: name(1), field encoding:LENGTH_DELIMITED(2). 1 << 3 | 2

 0b  "Stegosaurus".length()

 53  'S'

 74  't'

 65  'e'

 67  'g'

 6f  'o'

 73  's'

 61  'a'

 75  'u'

 72  'r'

 75  'u'

 73  's'

 28  tag: period(5), field encoding: VARINT(0). 5<< 3 | 0

 02  JURASSIC(2)

Why Wire?

TheProtocol Buffers schema language and binary encoding are both defined byGoogle. Wire is an independent implementation from Square that's specificallydesigned for Android and Java.

For eachmessage type defined in the schema, Wire generates an immutable model class andits builder. The generated code looks like code you'd write by hand: it'sdocumented, formatted, and simple. Wire's APIs should feel at home toprogrammers who like Effective Java.

Thatsaid, there are some interesting design decisions in Wire:

• Wire messages declare public final fields instead of the usual getter methods. This cuts down on both code generated and code executed. Less code is particularly beneficial for Android programs.
• Wire avoids case mapping. A field declared as picture_urls in a schema yields a Java fieldpicture_urls and not the conventional pictureUrls camel case. Though the name feels awkward at first, it's fantastic whenever you use grep or more sophisticated search tools. No more mapping when navigating between schema, Java source code, and data. It also provides a gentle reminder to calling code that proto messages are a bit special.
• Primitive types are always boxed. If a field is absent, its value is null. This is used for naturally optional fields, such as a dinosaur whose period is unknown. A field may also be null due to schema evolution: if tomorrow we add a carnivore boolean to our message definition, today's data won’t have a value for that field.

Here's the compact generated codefor the Dinosaur message defined above:

// Code generated by Wire protocol buffer compiler, donot edit.

// Source file: squareup/dinosaurs/dinosaur.proto at 9:1

packagecom.squareup.dinosaurs;

 

importcom.squareup.geology.Period;

importcom.squareup.wire.Message;

importcom.squareup.wire.ProtoAdapter;

importcom.squareup.wire.WireField;

importjava.util.List;

importokio.ByteString;

 

publicfinalclassDinosaurextendsMessage<Dinosaur, Dinosaur.Builder> {

  publicstaticfinalProtoAdapter<Dinosaur>ADAPTER=ProtoAdapter.newMessageAdapter(Dinosaur.class);

 

  privatestaticfinallong serialVersionUID =0L;

 

  publicstaticfinalStringDEFAULT_NAME="";

 

  publicstaticfinalPeriodDEFAULT_PERIOD=Period.CRETACEOUS;

 

  /**

   * Common name ofthis dinosaur, like "Stegosaurus".

   */

  @WireField(

      tag=1,

      adapter="com.squareup.wire.ProtoAdapter#STRING"

  )

  publicfinalString name;

 

  /**

   * URLs withimages of this dinosaur.

   */

  @WireField(

      tag=2,

      adapter="com.squareup.wire.ProtoAdapter#STRING",

      label=WireField.Label.REPEATED

  )

  publicfinalList<String> picture_urls;

 

  @WireField(

      tag=5,

      adapter="com.squareup.geology.Period#ADAPTER"

  )

  publicfinalPeriod period;

 

  publicDinosaur(Stringname, List<String>picture_urls, Periodperiod) {

    this(name, picture_urls, period, ByteString.EMPTY);

  }

 

  publicDinosaur(Stringname, List<String>picture_urls, Periodperiod, ByteStringunknownFields) {

    super(unknownFields);

    this.name = name;

    this.picture_urls = immutableCopyOf("picture_urls", picture_urls);

    this.period = period;

  }

 

  @Override

  publicBuildernewBuilder() {

    Builder builder =newBuilder();

    builder.name = name;

    builder.picture_urls = copyOf("picture_urls", picture_urls);

    builder.period = period;

    builder.addUnknownFields(unknownFields());

    return builder;

  }

 

  @Override

  publicbooleanequals(Objectother) {

    if (other ==this) returntrue;

    if (!(other instanceofDinosaur)) returnfalse;

    Dinosaur o = (Dinosaur) other;

    return equals(unknownFields(), o.unknownFields())

        && equals(name, o.name)

        && equals(picture_urls, o.picture_urls)

        && equals(period, o.period);

  }

 

  @Override

  publicinthashCode() {

    int result =super.hashCode;

    if (result ==0) {

      result = unknownFields().hashCode();

      result = result *37+ (name !=null? name.hashCode() :0);

      result = result *37+ (picture_urls !=null? picture_urls.hashCode() :1);

      result = result *37+ (period !=null? period.hashCode() :0);

      super.hashCode = result;

    }

    return result;

  }

 

  publicstaticfinalclassBuilderextendscom.squareup.wire.Message.Builder<Dinosaur, Builder> {

    publicString name;

 

    publicList<String> picture_urls;

 

    publicPeriod period;

 

    publicBuilder() {

      picture_urls = newMutableList();

    }

 

    /**

     * Common nameof this dinosaur, like "Stegosaurus".

     */

    publicBuildername(Stringname) {

      this.name = name;

      returnthis;

    }

 

    /**

     * URLs withimages of this dinosaur.

     */

    publicBuilderpicture_urls(List<String>picture_urls) {

      checkElementsNotNull(picture_urls);

      this.picture_urls = picture_urls;

      returnthis;

    }

 

    publicBuilderperiod(Periodperiod) {

      this.period = period;

      returnthis;

    }

 

    @Override

    publicDinosaurbuild() {

      returnnewDinosaur(name, picture_urls, period,buildUnknownFields());

    }

  }

}

The Javacode to create and access proto models is compact and readable:

Dinosaur stegosaurus =newDinosaur.Builder()

    .name("Stegosaurus")

    .period(Period.JURASSIC)

    .build();

 

System.out.println("My favoritedinosaur existed in the "+ stegosaurus.period +"period.");

Each type has a corresponding ProtoAdapter that can encode a message tobytes and decode bytes back into a message.

Dinosaur stegosaurus =...

byte[] stegosaurusBytes =Dinosaur.ADAPTER.encode(stegosaurus);

 

byte[] tyrannosaurusBytes =...

Dinosaur tyrannosaurus =Dinosaur.ADAPTER.decode(tyrannosaurusBytes);

When accessing a field, use Wire.get() to replace null values with thecorresponding default:

Period period =Wire.get(stegosaurus.period, Dinosaur.DEFAULT_PERIOD);

This isequivalent to the following:

Period period = stegosaurus.period != null ?stegosaurus.period : Dinosaur.DEFAULT_PERIOD;

GeneratingCode With Wire

Wire's compiler is available viaa Maven plugin. Put .proto sources in your project's src/main/protodirectory,then use the plugin to generate .java files. The plugin willautomatically add the generated Java code to your project's source roots.

<build>

  <plugins>

    <plugin>

      <groupId>com.squareup.wire</groupId>

      <artifactId>wire-maven-plugin</artifactId>

      <version>${project.version}</version>

      <executions>

        <execution>

          <phase>generate-sources</phase>

          <goals>

            <goal>generate-sources</goal>

          </goals>

          <configuration>

            <protoFiles>

              <param>squareup/dinosaurs/dinosaur.proto</param>

              <param>squareup/geology/period.proto</param>

            </protoFiles>

          </configuration>

        </execution>

      </executions>

    </plugin>

  </plugins>

</build>

Wire can read .proto files from the local file systemand from within .jar files.

Thecompiler can optionally prune your schema to a subset of root types and theirtransitive dependencies. This is useful when sharing a schema between projects:a Java service and Android app may each use a subset of a larger shared schema.

If you don't use Maven, thecompiler also has a command line interface. Just substitute wire-compiler-VERSION-jar-with-dependencies.jar with the path to your jar.

% java -jarwire-compiler-VERSION-jar-with-dependencies.jar \

   --proto_path=src/main/proto \

    --java_out=out\

   squareup/dinosaurs/dinosaur.proto \

   squareup/geology/period.proto

Writing com.squareup.dinosaurs.Dinosaur to out

Writing com.squareup.geology.Period to out

If you use Proguard, then youneed to add keep rules. The simplest option is totell Proguard not to touch the Wire runtime library and your generated protocolbuffers (of course these simple rules will miss opportunities to shrink andoptimize the code):

-keep class com.squareup.wire.** { *; }

-keep class com.yourcompany.yourgeneratedcode.** { *; }

Get Wire

The wire-runtime package contains runtime supportlibraries that must be included in applications that use Wire-generated code.

WithMaven:

<dependency>

  <groupId>com.squareup.wire</groupId>

  <artifactId>wire-runtime</artifactId>

  <version>1.8.0</version>

</dependency>

WithGradle:

compile 'com.squareup.wire:wire-runtime:1.8.0'

Snapshots of the developmentversion are available in Sonatype's snapshots repository.

Unsupported

Wire doesnot support:

• Groups - they are skipped when parsing binary input data

Wire supports custom options onmessages and fields. Other custom options are ignored. Use the --no_options compiler flag to omit optionsfrom the generated code.

FurtherDocumentation

See Google's excellentdocumentation on the structure and syntax of proto schemas.