Hbase bulkLoad 批量入库遇到的问题及解决方法

最新推荐文章于 2023-02-01 11:45:00 发布

转载最新推荐文章于 2023-02-01 11:45:00 发布 · 4.3k 阅读

文章标签：

# bulkload批量入库遇到问题、解决方法

hbase 专栏收录该内容

2 篇文章

订阅专栏

针对HBase BulkLoad过程中遇到的reduce阶段卡顿、效率低下及数据量过大导致入库失败等问题，通过合理设置reduce数量和预分区，调整hbase.hregion.max.filesize与hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily参数，显著提升了入库效率。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.2、BulkLoad 入库遇到问题及解决方法

1.2.1、首先就是reduce相关的问题:

在实际的应用中你就会发现,对于稍大一点的数据量,map过程的执行效率还是比较让人满意的,但是到了reduce阶段就会出现比较严重的卡顿,我的困惑就是?我的代码里明明没有reduce过程,为什么还会有reduce过程来影响我入库的效率呢?

于是,我尝试着在job里,设置reduce的数量,把它设置为0,可是重新执行的时候就会发现还是会有那个烦人的reduce过程, 既然设置它为0没有效果,那我把它的数量设置的多一点,提高它的并行度总能加快效率了吧于是我又修改了reduce的数量,执行的时候发现还是只有一个..... 后来我才知道, 在这种情况下,我们不用自己写reduce过程,但是会使用Hbase给我们提供的reduce,也就是说,无论你怎么设置reduce数量,都是无效的. 这样我也就释然了

1.2.2、效率严重低下的问题!!!

首先我用100M的数据量做测试,居然需要30s才能入库完毕!用几个G的数据量测试,效率也没有明显的提升! 也就是说平均每秒的插入速度还不到15000条.,这甚至比mysql的入库还要慢很多,这种效率在实际生产中是完全不能接受的说好的这是入库最快的方式呢?我不仅产生了怀疑.. 说到底,这种问题还是因为reduce数量只有一个这个蛋疼的问题所导致的,也就是说,不管你的集群有多牛,都值相当于单机版,这显然是不合适的...那么该如何解决这个问题呢????

就是在建表的时候进行合理的预分区!!!预分区的数目会决定你的reduce过程的数目!简单来说,在一定的范围内,进行合适预分区的话,reduce的数量增加多少,效率就提高多少倍!!!

有关于hbase的预分区,进行合适的预分区,实际上是一个很复杂的问题,也不是本篇文章讨论的重点. 感兴趣的话可以去看看这位大神写的东西,给了我很大的启发

大神的博客链接

我只简单介绍一下hbase建表时预分区的shell语句和执行的结果:

create 'XUE_BULKLOAD','info',{SPLITS => [ '1','2','3', '4','5','6','7','8','9']}

这样就成功的将表名为 'XUE_BULKLOAD',列簇名为'info'的表在建表时预分了10个分区

预分区结束之后进行测试:发现reduce的数量为预分区的数量+1,而且执行效率大大提高! 插入效率大致在10W/s~20W/s之间,已经勉强能达到实际工作的要求!

1.2.3、数据量超过某个范围就会导致插入数据库失败的问题!

经过各种各样的调试,效率已经可以接受! 然后开始调大数据量测试,发现哪怕几十个G的数据量,在执行完MapReduce过程之后都会报错,去表里面查看数据,一条记录都没有!!报错信息如下

Trying toload more than 32 hfiles to one family of one region

18/01/1823:20:36 ERROR mapreduce.LoadIncrementalHFiles: Trying to load

more than 32 hfiles to family info of region with start key

Exception inthread "main" java.io.IOException: Trying to load morethan

32 hfiles to one family of one region

at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad

(LoadIncrementalHFiles.java:377)

at hbase_Insert.Hbase_Insert.main(Hbase_Insert.java:241)

at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)

at sun.reflect.NativeMethodAccessorImpl.invoke(

NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(

DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

报错的大致意思就是试图将超过32个Hfile文件导入到hbase里面的一个region导致失败那这个问题该如何解决呢?实际上就是两个重要的参数限制的一个是:

hbase.hregion.max.filesize

单个ColumnFamily的region大小，若按照ConstantSizeRegionSplitPolicy策略，超过设置的该值则自动split 默认的大小是1G hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily

允许的hfile的最大个数,默认配置是32 也就是说:这两个参数的默认值决定了,每次批量入库的数据量不能超过1*32也就是32个G,超过这个数量就会导致入库失败

可以在代码里,或者在hbase安装路径下conf目录下的hbase-site.xml里面针对这两个参数进行设置为了一劳永逸,我选择在hbase-site.xml里面进行设置,设置结果如下:

<name>hbase.hregion.max.filesize</name>

</property>

<name>hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily</name>

</property>

这样,每次能够批量入库的数据就达到了32个T,符合公司的数据量需要! 配置完毕后重启集群进行测试,不在报这个错误,执行结果如下:

18/01/2012:17:31 INFO mapreduce.Job: map 1% reduce 0%

18/01/2012:17:35 INFO mapreduce.Job: map 2% reduce 0%

18/01/2012:17:42 INFO mapreduce.Job: map 3% reduce 0%

18/01/2012:17:45 INFO mapreduce.Job: map 4% reduce 0%

18/01/2012:17:51 INFO mapreduce.Job: map 5% reduce 0%

18/01/2012:17:55 INFO mapreduce.Job: map 6% reduce 0%

18/01/2012:17:59 INFO mapreduce.Job: map 7% reduce 0%

18/01/2012:18:03 INFO mapreduce.Job: map 8% reduce 0%

18/01/2012:18:06 INFO mapreduce.Job: map 9% reduce 0%

18/01/2012:18:11 INFO mapreduce.Job: map 10% reduce 0%

18/01/2012:18:16 INFO mapreduce.Job: map 11% reduce 0%

18/01/2012:18:20 INFO mapreduce.Job: map 12% reduce 0%

18/01/2012:18:27 INFO mapreduce.Job: map 13% reduce 0%

18/01/2012:18:32 INFO mapreduce.Job: map 14% reduce 0%

18/01/2012:18:37 INFO mapreduce.Job: map 15% reduce 0%

18/01/2012:18:42 INFO mapreduce.Job: map 16% reduce 0%

18/01/2012:18:47 INFO mapreduce.Job: map 17% reduce 0%

18/01/2012:18:53 INFO mapreduce.Job: map 18% reduce 0%

18/01/2012:18:58 INFO mapreduce.Job: map 19% reduce 0%

18/01/2012:19:03 INFO mapreduce.Job: map 20% reduce 0%

18/01/2012:19:08 INFO mapreduce.Job: map 21% reduce 0%

18/01/2012:19:14 INFO mapreduce.Job: map 22% reduce 0%

18/01/2012:19:18 INFO mapreduce.Job: map 23% reduce 0%

18/01/2012:19:23 INFO mapreduce.Job: map 24% reduce 0%

18/01/2012:19:29 INFO mapreduce.Job: map 25% reduce 0%

18/01/2012:19:33 INFO mapreduce.Job: map 26% reduce 0%

18/01/2012:19:38 INFO mapreduce.Job: map 27% reduce 0%

18/01/2012:19:43 INFO mapreduce.Job: map 28% reduce 0%

18/01/2012:19:48 INFO mapreduce.Job: map 29% reduce 0%

18/01/2012:19:53 INFO mapreduce.Job: map 30% reduce 0%

18/01/2012:19:58 INFO mapreduce.Job: map 31% reduce 0%

18/01/2012:20:04 INFO mapreduce.Job: map 32% reduce 0%

18/01/2012:20:08 INFO mapreduce.Job: map 33% reduce 0%

18/01/2012:20:13 INFO mapreduce.Job: map 34% reduce 0%

18/01/2012:20:17 INFO mapreduce.Job: map 35% reduce 0%

18/01/2012:20:21 INFO mapreduce.Job: map 36% reduce 0%

18/01/2012:20:25 INFO mapreduce.Job: map 37% reduce 0%

18/01/2012:20:29 INFO mapreduce.Job: map 38% reduce 0%

18/01/2012:20:33 INFO mapreduce.Job: map 39% reduce 0%

18/01/2012:20:37 INFO mapreduce.Job: map 40% reduce 0%

18/01/2012:20:41 INFO mapreduce.Job: map 41% reduce 0%

18/01/2012:20:45 INFO mapreduce.Job: map 42% reduce 0%

18/01/2012:20:50 INFO mapreduce.Job: map 43% reduce 0%

18/01/2012:20:54 INFO mapreduce.Job: map 44% reduce 0%

18/01/2012:20:58 INFO mapreduce.Job: map 45% reduce 0%

18/01/2012:21:02 INFO mapreduce.Job: map 46% reduce 0%

18/01/2012:21:06 INFO mapreduce.Job: map 47% reduce 0%

18/01/2012:21:10 INFO mapreduce.Job: map 48% reduce 0%

18/01/2012:21:14 INFO mapreduce.Job: map 49% reduce 0%

18/01/2012:21:18 INFO mapreduce.Job: map 50% reduce 0%

18/01/2012:21:22 INFO mapreduce.Job: map 51% reduce 0%

........

18/01/2012:29:12 INFO mapreduce.Job: map 100% reduce 81%

18/01/2012:29:24 INFO mapreduce.Job: map 100% reduce 82%

18/01/2012:29:36 INFO mapreduce.Job: map 100% reduce 83%

18/01/2012:29:48 INFO mapreduce.Job: map 100% reduce 84%

18/01/2012:30:00 INFO mapreduce.Job: map 100% reduce 85%

18/01/2012:30:12 INFO mapreduce.Job: map 100% reduce 86%

18/01/2012:30:23 INFO mapreduce.Job: map 100% reduce 87%

18/01/2012:30:33 INFO mapreduce.Job: map 100% reduce 88%

18/01/2012:30:45 INFO mapreduce.Job: map 100% reduce 89%

18/01/2012:30:59 INFO mapreduce.Job: map 100% reduce 90%

18/01/2012:31:11 INFO mapreduce.Job: map 100% reduce 91%

18/01/2012:31:21 INFO mapreduce.Job: map 100% reduce 92%

18/01/2012:31:33 INFO mapreduce.Job: map 100% reduce 93%

18/01/2012:31:45 INFO mapreduce.Job: map 100% reduce 94%

18/01/2012:31:57 INFO mapreduce.Job: map 100% reduce 95%

18/01/2012:32:10 INFO mapreduce.Job: map 100% reduce 96%

18/01/2012:32:28 INFO mapreduce.Job: map 100% reduce 97%

18/01/2012:32:57 INFO mapreduce.Job: map 100% reduce 98%

18/01/2012:33:28 INFO mapreduce.Job: map 100% reduce 99%

18/01/2012:34:43 INFO mapreduce.Job: map 100% reduce 100%

18/01/2012:38:02 INFO mapreduce.Job: Job job_1516347580021_0001 completedsuccessfully

18/01/2012:38:02 INFO mapreduce.Job: Counters: 52

File System Counters

FILE: Number of bytes read=87576726096

FILE: Number of bytes written=142193600747

FILE: Number of readoperations=0

FILE: Number of large readoperations=0

FILE: Number of writeoperations=0

HDFS: Number of bytes read=83582905128

HDFS: Number of bytes written=166475667426

HDFS: Number of readoperations=5468

HDFS: Number of large readoperations=0

HDFS: Number of writeoperations=39

Job Counters

Failed map tasks=6

Launched map tasks=1086

Launched reduce tasks=10

Other local map tasks=6

Data-local map tasks=465

Rack-local map tasks=615

Total time spent by all maps inoccupied slots(ms)=82454392

Total time spent by all reducesin occupied slots(ms)=47463944

Total time spent by all map tasks(ms)=10306799

Total time spent by all reduce tasks(ms)=5932993

Total vcore-seconds taken byall map tasks=10306799

Total vcore-seconds taken byall reduce tasks=5932993

Total megabyte-seconds taken byall map tasks=84433297408

Total megabyte-seconds taken byall reduce tasks=48603078656

Map-Reduce Framework

Map input records=568152966

Map output records=568152966

Map output bytes=228099087448

Map output materialized bytes=54476960272

Input split bytes=186120

Combine input records=0

Combine output records=0

Reduce input groups=292435364

Reduce shuffle bytes=54476960272

Reduce input records=568152966

Reduce output records=2339482912

Spilled Records=1513624168

Shuffled Maps =10800

Failed Shuffles=0

Merged Map outputs=10800

GC time elapsed(ms)=794607

CPU time spent(ms)=21363440

Physical memory(bytes) snapshot=3038556569600

Virtual memory(bytes) snapshot=9401710268416

Total committed heap usage(bytes)=3512994889728

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=83582349648

File Output Format Counters

Bytes Written=166475667426

18/01/2012:38:02 INFO zookeeper.RecoverableZooKeeper: Processidentifier=hconnection-0x71f30c76 connecting to ZooKeeper ensemble=node003:4180,node002:4180,node001:4180,master:4180,node009:4180,node008:4180,node007:4180,node010:4180,node006:4180,node005:4180,node004:4180

18/01/2012:38:02 INFO zookeeper.ZooKeeper: Initiating client connection,connectString=node003:4180,node002:4180,node001:4180,master:4180,node009:4180,node008:4180,node007:4180,node010:4180,node006:4180,node005:4180,node004:4180 sessionTimeout=90000 watcher=hconnection-0x71f30c760x0, quorum=node003:4180,node002:4180,node001:4180,master:4180,node009:4180,node008:4180,node007:4180,node010:4180,node006:4180,node005:4180,node004:4180, baseZNode=/hbase

18/01/2012:38:02 INFO zookeeper.ClientCnxn: Opening socket connection toserver node004/192.168.1.38:4180. Will not attempt to authenticate using SASL(unknown error)

18/01/2012:38:02 INFO zookeeper.ClientCnxn: Socket connection established tonode004/192.168.1.38:4180, initiating session

18/01/2012:38:02 INFO zookeeper.ClientCnxn: Session establishment complete on servernode004/192.168.1.38:4180, sessionid = 0x26001af8d8190002, negotiated timeout = 40000

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Skippingnon-directory hdfs://192.168.1.31:9000/test_demo/result/test/_SUCCESS

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/28e47c53edaf4616a3dfc349d0f0e02awith size: 10931823633 bytes can be problematic as it may lead tooversplitting.

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/5e6501ccb7554b82a2e93024d61dbe0ewith size: 10931820982 bytes can be problematic as it may lead tooversplitting.

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/6245730468534f85a428ef7fb7acd499with size: 10931829083 bytes can be problematic as it may lead tooversplitting.

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/a3b7225320b24e838559d5a5772bdd87with size: 10931823391 bytes can be problematic as it may lead tooversplitting.

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/a8306bc4ef3941f5bd131d47f0b1c2c3with size: 10931822321 bytes can be problematic as it may lead tooversplitting.

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/c163c568b1c24d88ac0ed7599b81ecbawith size: 10931824861 bytes can be problematic as it may lead tooversplitting.

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/d069683ce064411793640f2a0ec6ca98with size: 10931822990 bytes can be problematic as it may lead tooversplitting.

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/e5df664e18c54da7b84370b72506923bwith size: 10931821709 bytes can be problematic as it may lead tooversplitting.

18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/fba38b4d0bd34f6782b844b288780e7bwith size: 10931826385 bytes can be problematic as it may lead tooversplitting.

18/01/2012:38:02 INFO hfile.CacheConfig: CacheConfig:disabled