DataX3.0离线同步工具

基本信息 

DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、DRDS 等各种异构数据源之间高效的数据同步功能。

Github地址：https://github.com/alibaba/DataX

DataX3.0下载地址：http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz

运行环境：   jdk (1.6以上)    java -version        python (2.6以上)   3.0以上应该需要修改脚本(官方要求2.7)

python 3的脚本地址 

安装完成，如果环境是Python3，则需要修改datax/bin下面的三个python文件。如果是Python2则不需修改。（修改需要的文件链接：https://github.com/WeiYe-Jing/datax-web/tree/master/doc/datax-web/datax-python3）

测试是否安装正常

进入到DataX3.0的bin目录下执行命令：python datax.py ../job/job.json

查看配置模板

python datax.py -r mysqlreader -w mysqlwriter

执行同步

python datax.py mysqls2mysqls.json

配置模板例子

mysql到mysql(单表)

mysql2mysql.json

  {
    "job": {
        "content": [
            {
                "reader": {
                    "name": "mysqlreader", 
                    "parameter": {
                        "column": ["id","role"], 
                        "connection": [
                            {
                                "jdbcUrl": ["jdbc:mysql://192.168.1.83:3306/data_1"], 
                                "table": ["tb_role"]
                            }
                        ], 
			"username": "root",
                        "password": "1" 
                        
                    }
                }, 
                "writer": {
                    "name": "mysqlwriter", 
                    "parameter": {
                        "column": ["id","role"],  
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:mysql://192.168.1.83:3306/data_2?useUnicode=true&characterEncoding=gbk", 
                                "table": ["tb_role"]
                            }
                        ], 
			"username": "root",
                        "password": "1" 
                        
                    }
                }
            }
        ], 
        "setting": {
            "speed": {
                "channel": "1"
            }
        }
    }
}

mysql到oracle(单表)

mysql2oracle.json

  {
    "job": {
        "content": [
            {
                "reader": {
                    "name": "mysqlreader", 
                    "parameter": {
                        "column": ["id","role"], 
                        "connection": [
                            {
                                "jdbcUrl": ["jdbc:mysql://192.168.1.83:3306/data_1"], 
                                "table": ["tb_role"]
                            }
                        ], 
			"username": "root",
                        "password": "1" 
                        
                    }
                }, 
                "writer": {
                    "name": "oraclewriter", 
                    "parameter": {
                        "column": ["id","role"],  
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:oracle:thin:@192.168.1.177:1521:fjtw", 
                                "table": ["tb_role"]
                            }
                        ], 
			"username": "XXXX",
                        "password": "XXXX" 
                        
                    }
                }
            }
        ], 
        "setting": {
            "speed": {
                "channel": "1"
            }
        }
    }
}

mysql到mysql(多表)

mysqls2mysqls.json

  {
    "job": {
        "content": [
            {
                "reader": {
                    "name": "mysqlreader", 
                    "parameter": {
                        "column": ["*"], 
                        "connection": [
                            {
                                "jdbcUrl": ["jdbc:mysql://192.168.1.83:3306/data_1"], 
                                "table": ["tb_role","tb_role2"]
                            }
                        ], 
			"username": "root",
                        "password": "1" 
                        
                    }
                }, 
                "writer": {
                    "name": "mysqlwriter", 
                    "parameter": {
                        "column": ["*"],  
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:mysql://192.168.1.83:3306/data_2?useUnicode=true&characterEncoding=gbk", 
                                "table": ["tb_role","tb_role2"]
                            }
                        ], 
			"username": "root",
                        "password": "1" 
                        
                    }
                }
            }
        ], 
        "setting": {
            "speed": {
                "channel": "1"
            }
        }
    }
}

参考文档:

https://blog.youkuaiyun.com/tototuzuoquan/article/details/102601515?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-3.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-3.nonecase

https://blog.youkuaiyun.com/sinat_33416409/article/details/83176725

https://www.cnblogs.com/hellogt/p/6839985.html

https://zhuanlan.zhihu.com/p/56600910