Centos7.9通过datax-web2.0_用Datax3.0分别同步postgresql到mysql和hbase_同步mysql同时到mysql及hbase---大数据之DataX工作笔记005

  首先来看如何把数据利用datax-web,通过datax3.0,把数据从postgresql同步到mysql

这里也是给出对应的datax需要的json文件

首先去data-web构建一下需要的json,这里先添加数据源,数据源这里我添加了mysql的两个,postgresql的一个,hbase的加不上,其他博文说了,hbase我们就自己写到json中就可以了,这个有提供的模板,到时候直接复制就可以.

记得Json的底部需要加上:

,
  "core": {
    "transport": {
      "channel": {
        "speed": {
          "byte": 1231412
        }
      }
    }
  }

 来看一下完整的JSON

{
  "job": {
    "setting": {
      "speed": {
        "channel": 3,
        "byte": 1048576
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "postgresqlreader",
          "parameter": {
            "username": "XVko54UY9nOe/3JQGQUikw==",
            "password": "XCYVpFosvZBBWobFzmLWvA==",
            "column": [
              "\"id\"",
              "\"name\"",
              "\"mobile\"",
              "\"email\"",
              "\"son_json\""
            ],
            "splitPk": "",
            "connection": [
              {
                "table": [
                  "public.userinfo"
                ],
                "jdbcUrl": [
                  "jdbc:postgresql://172.19.126.122:5432/test_db"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "mysqlwriter",
          "parameter": {
            "username": "yRjwDFuoPKlqya9h9H2Amg==",
            "password": "XCYVpFosvZBBWobFzmLWvA==",
            "column": [
              "`id`",
              "`name`",
              "`email`",
              "`mobile`",
              "`son_json`"
            ],
            "connection": [
              {
                "table": [
                  "userinfo"
                ],
                "jdbcUrl": "jdbc:mysql://172.19.126.123:3306/nifibase?rewriteBatchedStatements=true&useUnicode=true&characterEncoding=UTF8&serverTimezone=Asia/Shanghai&useSSL=false&verifyServerCertificate=false&autoReconnect=true&failOverReadOnly=false"
              }
            ]
          }
        }
      }
    ]
  },
  "core": {
    "transport": {
      "channel": {
        "speed": {
          "byte": 1231412
        }
      }
    }
  }
}

 这里执行就可以把数据传递过来了

 ---------------------

然后我们再来看将Postgresql中的数据通过datax-web导入到hbase中,这里利用Phoenix实现

这里也是给出对应的JSON文件就可以了

{
  "job": {
    "setting": {
      "speed": {
        "channel": 3,
        "byte": 1048576
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "postgresqlreader",
          "parameter": {
            "username": "XVko54UY9nOe/3JQGQUikw==",
            "password": "XCYVpFosvZBBWobFzmLWvA==",
            "column": [
              "\"id\"",
              "\"name\"",
              "\"mobile\"",
              "\"email\"",
              "\"son_json\""
            ],
            "splitPk": "",
            "connection": [
              {
                "table": [
                  "public.userinfo"
                ],
                "jdbcUrl": [
                  "jdbc:postgresql://172.19.126.122:5432/test_db"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "hbase20xsqlwriter",
          "parameter": {
            "batchSize": "100",
            "column": [
              "id",
              "name",
              "mobile",
              "email",
              "son_json"
            ],
            "queryServerAddress": "http://127.0.0.1:8765",
            "nullMode": "skip",
            "table": "userinfo"
          }
        }
      }
    ]
  },
  "core": {
    "transport": {
      "channel": {
        "speed": {
          "byte": 1231412
        }
      }
    }
  }
}

构建任务的话,这里就构建一个postgresql到mysql的,然后得到json以后再修改就可以了,这里直接给出对应的可以执行json

首先去执行对应的任务 

 先来清空一下hbase中的userinfo这个表,这里我用的squirrel来通过Phoenix操作的hbase

然后执行同步

 执行以后可以看到已经同步过来了.

------------------------------

然后我们再来看一下如何通过datax-web来实现,从postgresql中读取数据然后同时存入到hbase表中和mysql表中去

 首先我们清空一下mysql表中的数据

然后再清空一下hbase表中的数据

然后看一下json