datax-hbase2stream

最新推荐文章于 2024-12-23 23:28:05 发布

yandao

最新推荐文章于 2024-12-23 23:28:05 发布

阅读量338

点赞数

分类专栏：大数据文章标签：大数据

本文链接：https://blog.youkuaiyun.com/yandao/article/details/109389117

版权

大数据专栏收录该内容

25 篇文章

订阅专栏

该博客介绍了如何准备Hadoop和HBase的环境，并提供了详细参考资料，内容聚焦于将HBase数据迁移到Stream的过程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

####准备hadoop,hbase的环境，参考
https://blog.youkuaiyun.com/yandao/article/details/108690126

hbase(main):001:0> list
TABLE                                                                                                                                                                                                                                                                                         
0 row(s) in 0.5890 seconds

hbase(main):002:0> create 'datax','t'
0 row(s) in 2.4830 seconds

=> Hbase::Table - datax
hbase(main):003:0> list
TABLE                                                                                                                                            
datax                                                                                                                                                                                                                                                                                           
1 row(s) in 0.0300 seconds

=> ["datax", "t1", "t2"]
hbase(main):004:0> put 'datax','row1','t:name','name1'
0 row(s) in 0.3360 seconds

hbase(main):005:0> put 'datax','row1','t:age','21'
0 row(s) in 0.0200 seconds

hbase(main):006:0> put 'datax','row2','t:age','21'
0 row(s) in 0.0360 seconds

hbase(main):007:0> put 'datax','row2','t:name','name2'
0 row(s) in 0.0630 seconds

hbase(main):008:0> put 'datax','row3','t:name','name3'
0 row(s) in 0.0190 seconds

hbase(main):009:0> put 'datax','row3','t:age','23'
0 row(s) in 0.0330 seconds

hbase(main):010:0> put 'datax','row4','t:age','24'
0 row(s) in 0.0240 seconds

hbase(main):011:0> put 'datax','row4','t:name','name4'
0 row(s) in 0.0230 seconds

hbase(main):012:0>

####hbase2stream.json

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "hbase11xreader", 
                    "parameter": {
                       "hbaseConfig": {
                            "hbase.zookeeper.quorum": "192.168.0.200:2182"
                        },
                        "table": "datax",
                        "encoding": "utf-8",
                        "mode": "normal",
                        "column": [
                        {
                          "name":"rowkey",
                          "type":"string"
                        },
                        {
                          "name":"t:name",
                          "type":"string"
                        },
                        {
                          "name":"t:age",
                          "type":"string"
                        }
                        ], 
                      
                        "range": {
                            "endRowkey": "", 
                            "isBinaryRowkey": false, 
                            "startRowkey": ""
                        }
                  
                    }
                }, 
                "writer": {
                    "name": "streamwriter",
                    "parameter": {
                        "print": true
                    }
                }
            }
        ], 
        "setting": {
            "speed": {
                "channel": 5
            }
        }
    }
}

结论

2020-10-30 17:04:04.146 [0-0-0-reader] INFO  HbaseAbstractTask - The task set startRowkey=[], endRowkey=[].
row1    name1   21
row2    name2   21
row3    name3   23
row4    name4   24
2020-10-30 17:04:04.276 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[304]ms
2020-10-30 17:04:04.276 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2020-10-30 17:04:13.909 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4 records, 44 bytes | Speed 4B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.217s | Percentage 100.00%
2020-10-30 17:04:13.911 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2020-10-30 17:04:13.921 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do post work.
2020-10-30 17:04:13.923 [job-0] INFO  JobContainer - DataX Reader.Job [hbase11xreader] do post work.
2020-10-30 17:04:13.924 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2020-10-30 17:04:13.927 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: d:\java\datax\hook
2020-10-30 17:04:13.936 [job-0] INFO  JobContainer -
         [total cpu info] =>
                averageCpu                     | maxDeltaCpu                    | minDeltaCpu
                -1.00%                         | -1.00%                         | -1.00%


         [total gc info] =>
                 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime
                 PS MarkSweep         | 1                  | 1                  | 1                  | 0.038s             | 0.038s             | 0.038s
                 PS Scavenge          | 1                  | 1                  | 1                  | 0.018s             | 0.018s             | 0.018s

2020-10-30 17:04:13.936 [job-0] INFO  JobContainer - PerfTrace not enable!
2020-10-30 17:04:13.939 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4 records, 44 bytes | Speed 4B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.217s | Percentage 100.00%
2020-10-30 17:04:13.949 [job-0] INFO  JobContainer -
任务启动时刻                    : 2020-10-30 17:03:56
任务结束时刻                    : 2020-10-30 17:04:13
任务总计耗时                    :                 17s
任务平均流量                    :                4B/s
记录写入速度                    :              0rec/s
读出记录总数                    :                   4
读写失败总数                    :                   0


d:\java\datax\bin>