关于HBase MVCC的设计原理以及MVCC所引起的一个scan问题

最近在使用HBase0.94版本的时，偶尔会出现，HRegionInfo was null or empty in Meta 的警告

java.io.IOException: HRegionInfo was null or empty in Meta for writetest, row=lot_let,9399239430349923234234,99999999999999
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:170)

在客户端的MetaScanner.metaScan实现中

metaTable = new HTable(configuration, HConstants.META_TABLE_NAME);

Result startRowResult = metaTable.getRowOrBefore(searchRow,HConstants.CATALOG_FAMILY);

if (startRowResult == null) { throw new TableNotFoundException("Cannot find row in .META. for table: " + Bytes.toString(tableName) + ", row=" + Bytes.toStringBinary(searchRow)); }
byte[] value = startRowResult.getValue(HConstants.CATALOG_FAMILY,
HConstants.REGIONINFO_QUALIFIER);
if (value == null || value.length == 0) { throw new IOException("HRegionInfo was null or empty in Meta for " + Bytes.toString(tableName) + ", row=" + Bytes.toStringBinary(searchRow)); }

可以发现在扫描MetaScanner,rowkey所在的范围在Meta 表中不存在；通过RPC定位到服务端的实现

HRegion中：

 public Result getClosestRowBefore(final byte [] row, final byte [] family)

  throws IOException {

    if (coprocessorHost != null) {

      Result result = new Result();

      if (coprocessorHost.preGetClosestRowBefore(row, family, result)) {

        return result;

      }

    }

    // look across all the HStores for this region and determine what the

    // closest key is across all column families, since the data may be sparse

    checkRow(row, "getClosestRowBefore");

    startRegionOperation();

    this.readRequestsCount.increment();

    try {

      Store store = getStore(family);

      // get the closest key. (HStore.getRowKeyAtOrBefore can return null)

      KeyValue key = store.getRowKeyAtOrBefore(row);

      Result result = null;

      if (key != null) {

        Get get = new Get(key.getRow());

        get.addFamily(family);

        result = get(get, null);

      }

      if (coprocessorHost != null) {

        coprocessorHost.postGetClosestRowBefore(row, family, result);

      }

      return result;

    } finally {

      closeRegionOperation();

    }

  }

在 KeyValue key = store.getRowKeyAtOrBefore(row);中获得了Meta表的rowkey，但是在后续的实现中

     if (key != null) {

        Get get = new Get(key.getRow());

        get.addFamily(family);

        result = get(get, null);

      }

获得空的result导致了这个问题;

为什么会存在这个现象。

先讲一下HBase 的MVCC的原理，

MVCC是保证数据一致性的手段，HBase在写数据的过程中，需要经过好几个阶段，写HLog，写memstore，更新MVCC;

只有更新了MVCC，才算真正memstore写成功，其中事务的隔离需要有mvcc的来控制，比如读数据不可以获取别的线程还未提交的数据。

1、put、delete数据都会调用applyFamilyMapToMemstore

HRegion中

private long applyFamilyMapToMemstore(Map<byte[], List<KeyValue>> familyMap,

    MultiVersionConsistencyControl.WriteEntry localizedWriteEntry) {

    long size = 0;

    boolean freemvcc = false;

    try {

      if (localizedWriteEntry == null) {

//开始一个写memstore，mvcc中的memstoreWrite++，并add待write pending队列中

        localizedWriteEntry = mvcc.beginMemstoreInsert();

        freemvcc = true;

      }

      for (Map.Entry<byte[], List<KeyValue>> e : familyMap.entrySet()) {

        byte[] family = e.getKey();

        List<KeyValue> edits = e.getValue();

        Store store = getStore(family);

        for (KeyValue kv: edits) {

          kv.setMemstoreTS(localizedWriteEntry.getWriteNumber());

          size += store.add(kv);

        }

      }

    } finally {

      if (freemvcc) {

        mvcc.completeMemstoreInsert(localizedWriteEntry);

      }

    }

     return size;

   }

  mvcc.completeMemstoreInsert，更新mvcc 的memstoreRead，也就是可以读的位置， 并通知readWaiters.notifyAll()，释放因flushcache调用waitForRead引起的阻塞;

waitForRead参见以下代码：

       public void waitForRead(WriteEntry e) {

    boolean interrupted = false;

    synchronized (readWaiters) {

//小于，表示还有写未提交

      while (memstoreRead < e.getWriteNumber()) {

        try {

          readWaiters.wait(0);

        } catch (InterruptedException ie) {

          // We were interrupted... finish the loop -- i.e. cleanup --and then

          // on our way out, reset the interrupt flag.

          interrupted = true;

        }

      }

    }

    if (interrupted) Thread.currentThread().interrupt();

  }

2、  在flushcache的过程中，获取到memstore中的keyvalues后,会调用mvcc.waitForRead(w)(因memstore所有的keyvalue,包括还未真正提交的，所以要等待其他事务提交后，才可以进行后续的flush操作，保证事务的一致性。

      w = mvcc.beginMemstoreInsert();

      mvcc.advanceMemstore(w);

    mvcc.waitForRead(w);

3、scan数据

在RegionScannerImpl.next方法实现中：

    public synchronized boolean next(List<KeyValue> outResults, int limit)

        throws IOException {

      if (this.filterClosed) {

        throw new UnknownScannerException("Scanner was closed (timed out?) " +

            "after we renewed it. Could be caused by a very slow scanner " +

            "or a lengthy garbage collection");

      }

      startRegionOperation();

      readRequestsCount.increment();

      try {

        // This could be a new thread from the last time we called next().

//this.readPoint在构造的时，初始化（readpoint为当前hregion的mvcc中的memstoreRead，为当前可读的点）和当前线程绑定

        MultiVersionConsistencyControl.setThreadReadPoint(this.readPt);

在MemStore中过滤掉还未提交的事务（新的keyvalue中有最新的point）

    protected KeyValue getNext(Iterator<KeyValue> it) {

      long readPoint = MultiVersionConsistencyControl.getThreadReadPoint();

      while (it.hasNext()) {

        KeyValue v = it.next();

//过滤掉大于当前线程readPoint的keyvalue

        if (v.getMemstoreTS() <= readPoint) {

          return v;

        }

      }

      return null;

    }

 

纵观MVCC的整个过程，再分析HRegion中的getClosestRowBefore方法实现，

      KeyValue key = store.getRowKeyAtOrBefore(row);

这个调用不会进行MVCC的控制，可以读到memstore中所有的数据

而get方法是会进行MVCC进行控制的，所以一种可能情况是在get调用的时， store.getRowKeyAtOrBefore(row)读到的key值还未提交，

所有都过滤掉了，查询范围为null。