Hbase扫描时抛出ScannerTimeoutException处理

最新推荐文章于 2022-04-29 01:23:46 发布

最新推荐文章于 2022-04-29 01:23:46 发布 · 391 阅读

文章标签：

#大数据 #java

本文探讨了在使用HBase进行Scan操作时遇到的ScannerTimeoutException问题，详细分析了异常产生的原因，并提供了通过调整HBase配置参数来解决该问题的具体方法。

在对Hbase进行Scan的时候有时候会抛出ScannerTimeoutException，场景如下：

2012 15:28:52 (14mins, 41sec)
org.apache.hadoop.hbase.client.ScannerTimeoutException: 60622ms passed since the last invocation, timeout is currently set to 60000
	at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1196)
	at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:133)
	at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:142)
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
	at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)

原因查看步骤：

查找源码如下：

 long timeout = lastNext + scannerTimeout;
              // If we are over the timeout, throw this exception to the client
              // Else, it's because the region moved and we used the old id
              // against the new region server; reset the scanner.
              if (timeout < System.currentTimeMillis()) {
                long elapsed = System.currentTimeMillis() - lastNext;
                ScannerTimeoutException ex = new ScannerTimeoutException(
                    elapsed + "ms passed since the last invocation, " +
                        "timeout is currently set to " + scannerTimeout);
                ex.initCause(e);
                throw ex;
              }

核实下如下代码

 long timeout = lastNext + scannerTimeout;

得出：ClientScanner.lastNext和HTable.scannerTimeout

跟踪：HTable.scannerTimeout

 this.scannerTimeout =
      (int) conf.getLong(HConstants.HBASE_REGIONSERVER_LEASE_PERIOD_KEY, HConstants.DEFAULT_HBASE_REGIONSERVER_LEASE_PERIOD);

查看HConstants.HBASE_REGIONSERVER_LEASE_PERIOD_KEY和HConstants.DEFAULT_HBASE_REGIONSERVER_LEASE_PERIOD

我们可以得知如果没有设置HConstants.HBASE_REGIONSERVER_LEASE_PERIOD_KEY那么他们采用默认值即

 public static long DEFAULT_HBASE_REGIONSERVER_LEASE_PERIOD = 60000;

跟踪：ClientScanner.lastNext发现此为上次访问时间

解决方案：

可以通过设置HConstants.HBASE_REGIONSERVER_LEASE_PERIOD_KEY解决

代码如下：

config.setLong(HConstants.HBASE_REGIONSERVER_LEASE_PERIOD_KEY, 120000);