Spark 4.0的VariantType 类型以及内部存储

背景

本文基于Spark 4.0

总结

Spark中的 VariantType 类型，用尽量少的字节来存储Json的格式化数据

分析

这里主要介绍 Variant 的存储，我们从VariantBuilder.buildJson方法(把对应的json数据存储为VariantType类型)开始：

public static Variant parseJson(JsonParser parser, boolean allowDuplicateKeys)
      throws IOException {
    VariantBuilder builder = new VariantBuilder(allowDuplicateKeys);
    builder.buildJson(parser);
    return builder.result();
  }

这个方法会调用buildJson这个方法：

JsonToken token = parser.currentToken();
    if (token == null) {
      throw new JsonParseException(parser, "Unexpected null token");
    }
    switch (token) {
      case START_OBJECT: {
        ArrayList<FieldEntry> fields = new ArrayList<>();
        int start = writePos;
        while (parser.nextToken() != JsonToken.END_OBJECT) {
          String key = parser.currentName();
          parser.nextToken();
          int id = addKey(key);
          fields.add(new FieldEntry(key, id, writePos - start));
          buildJson(parser);
        }
        finishWritingObject(start, fields);
        break;
      }
      case START_ARRAY: {
        ArrayList<Integer> offsets = new ArrayList<>();
        int start = writePos;
        while (parser.nextToken() != JsonToken.END_ARRAY) {
          offsets.add(writePos - start);
          buildJson(parser);
        }
        finishWritingArray(start, offsets);
        break;
      }
      case VALUE_STRING:
        appendString(parser.getText());
        break;
      case VALUE_NUMBER_INT:
        try {
          appendLong(parser.getLongValue());