Hbase 踩坑

HBase Major Compaction 日志解析
最新推荐文章于 2025-10-24 17:28:08 发布
原创最新推荐文章于 2025-10-24 17:28:08 发布 · 3.4k 阅读
0 ·
CC 4.0 BY-SA版权
Hbase 专栏收录该内容
10 篇文章
订阅专栏
本文记录了HBase在特定时间进行Major Compaction的过程，包括多个region的压缩活动详情，如文件数量、大小及压缩后的效果。此外，还展示了压缩过程中使用的临时目录、缓存配置以及压缩算法等关键信息。
2018-02-11 15:50:27,843 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] regionserver.RSRpcServices: Compacting archiveLogData,D,1517906411842.c5edfc6575a591b2b5eb06b9e069bd48.
2018-02-11 15:50:27,853 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] regionserver.RSRpcServices: Compacting archiveLogData,E,1517906411842.266f3403ecdf3f9e7e50bb81f3b4e15c.
2018-02-11 15:50:27,853 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] regionserver.HRegion: Starting compaction on result in region archiveLogData,E,1517906411842.266f3403ecdf3f9e7e50bb81f3b4e15c.
2018-02-11 15:50:27,854 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] regionserver.HStore: Starting compaction of 2 file(s) in result of archiveLogData,E,1517906411842.266f3403ecdf3f9e7e50bb81f3b4e15c. into tmpdir=hdfs://master:8020/apps/hbase/data/data/default/archiveLogData/266f3403ecdf3f9e7e50bb81f3b4e15c/.tmp, totalSize=2.2 M
2018-02-11 15:50:27,858 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] regionserver.RSRpcServices: Compacting archiveLogData,I,1517906411842.dae8cbe26db0f7c9eee4ef538b84114e.
2018-02-11 15:50:27,863 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] regionserver.RSRpcServices: Compacting archiveLogData,M,1517906411842.816d2f840c296bb2670179297dd8930b.
2018-02-11 15:50:27,865 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] regionserver.RSRpcServices: Compacting archiveLogData,O,1517906411842.44373bc9f52c764a167a4366d13fc008.
2018-02-11 15:50:27,870 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] regionserver.RSRpcServices: Compacting archiveLogData,R,1517906411842.1d044faf08659ff7a84077b094d75168.
2018-02-11 15:50:27,871 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] regionserver.RSRpcServices: Compacting archiveLogData,U,1517906411842.d6c4ff2720459b962ffc97888c28cb67.
2018-02-11 15:50:27,871 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=896080, freeSize=841005616, maxSize=841901696, heapSize=896080, minSize=799806592, minFactor=0.95, multiSize=399903296, multiFactor=0.5, singleSize=199951648, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2018-02-11 15:50:27,878 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] compress.CodecPool: Got brand-new compressor [.snappy]
2018-02-11 15:50:27,974 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] compress.CodecPool: Got brand-new decompressor [.snappy]
2018-02-11 15:50:27,998 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] regionserver.HStore: Completed major compaction of 2 (all) file(s) in result of archiveLogData,E,1517906411842.266f3403ecdf3f9e7e50bb81f3b4e15c. into b03f605b75e44cda98fe8aa58fe05d4f(size=418.6 K), total size for store is 418.6 K. This selection was in queue for 0sec, and took 0sec to execute.
2018-02-11 15:50:27,998 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] regionserver.CompactSplitThread: Completed compaction: Request = regionName=archiveLogData,E,1517906411842.266f3403ecdf3f9e7e50bb81f3b4e15c., storeName=result, fileCount=2, fileSize=2.2 M, priority=1, time=11832333055519425; duration=0sec
2018-02-11 15:50:27,998 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] regionserver.HRegion: Starting compaction on result in region archiveLogData,U,1517906411842.d6c4ff2720459b962ffc97888c28cb67.
2018-02-11 15:50:27,999 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] regionserver.HStore: Starting compaction of 2 file(s) in result of archiveLogData,U,1517906411842.d6c4ff2720459b962ffc97888c28cb67. into tmpdir=hdfs://master:8020/apps/hbase/data/data/default/archiveLogData/d6c4ff2720459b962ffc97888c28cb67/.tmp, totalSize=4.2 M
2018-02-11 15:50:28,005 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=896080, freeSize=841005616, maxSize=841901696, heapSize=896080, minSize=799806592, minFactor=0.95, multiSize=399903296, multiFactor=0.5, singleSize=199951648, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2018-02-11 15:50:28,135 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] regionserver.HStore: Completed major compaction of 2 (all) file(s) in result of archiveLogData,U,1517906411842.d6c4ff2720459b962ffc97888c28cb67. into 018c53a55ee14144a483ef1c34142442(size=943.0 K), total size for store is 943.0 K. This selection was in queue for 0sec, and took 0sec to execute.
2018-02-11 15:50:28,135 INFO  [regionserver/master/10.1.69.11:16020-shortCompactions-1517386971937] regionserver.CompactSplitThread: Completed compaction: Request = regionName=archiveLogData,U,1517906411842.d6c4ff2720459b962ffc97888c28cb67., storeName=result, fileCount=2, fileSize=4.2 M, priority=1, time=11832333073118003; duration=0sec
2018-02-11 15:51:39,264 INFO  [regionserver/master/10.1.69.11:16020.leaseChecker] regionserver.RSRpcServices: Scanner 4157 lease expired on region archiveLogData,E,1517906411842.266f3403ecdf3f9e7e50bb81f3b4e15c.
2018-02-11 15:52:15,352 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.17 MB, freeSize=801.73 MB, max=802.90 MB, blockCount=9, accesses=14039, hits=13590, hitRatio=96.80%, , cachingAccesses=13598, cachingHits=13560, cachingHitsRatio=99.72%, evictions=94889, evicted=29, evictedPerRun=3.0562025494873524E-4
2018-02-11 15:52:46,915 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: Stopping HBase metrics system...
2018-02-11 15:52:46,916 INFO  [timeline] impl.MetricsSinkAdapter: timeline thread interrupted.
2018-02-11 15:52:46,918 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system stopped.
2018-02-11 15:52:47,419 INFO  [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties
2018-02-11 15:52:47,539 INFO  [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink.
2018-02-11 15:52:47,539 INFO  [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = master, serviceName = hbase
2018-02-11 15:52:47,579 INFO  [HBase-Metrics2-1] availability.MetricSinkWriteShardHostnameHashingStrategy: Calculated collector shard slave2 based on hostname: master
2018-02-11 15:52:47,579 INFO  [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://slave2:6188/ws/v1/timeline/metrics
2018-02-11 15:52:47,579 INFO  [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Container Metrics Uri: http://slave2:6188/ws/v1/timeline/containermetrics
2018-02-11 15:52:47,583 INFO  [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started
2018-02-11 15:52:47,619 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2018-02-11 15:52:47,619 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started
主动对table进行Major_Compact