将本地mysql数据库表格导入到odps

最新推荐文章于 2025-06-07 23:42:30 发布

转载最新推荐文章于 2025-06-07 23:42:30 发布 · 2.4k 阅读

基础学习专栏收录该内容

31 篇文章

订阅专栏

本文介绍如何使用DataX工具将本地MySQL数据库表的数据导入到阿里云ODPS大数据计算服务中，包括DataX的安装配置及JSON任务文件的编写。

将本地mysql数据库表格导入到odps大数据计算服务项目中

（还有一种方式是通过odps tunnel的方式：https://help.aliyun.com/document_detail/27833.html）

一、工具下载以及部署

方法一、直接下载DataX工具包(如果仅是使用，推荐直接下载)：

1.1. 下载DataX :

  wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz

1.2. 解压 DataX：

  tar zxvf datax.tar.gz

1.3. 修改权限为755：

  chmod -R 755 datax

二、配置示例：从MySQL读取数据写入ODPS

cd datax/job

vim job.json

第一步、创建作业的配置文件（json格式）

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "column": [],
                        "connection": [
                            {
                                "jdbcUrl": [],
                                "table": []
                            }
                        ],
                        "password": "",
                        "username": "",
                        "where": ""
                    }
                },
                "writer": {
                    "name": "odpswriter",
                    "parameter": {
                        "accessId": "",
                        "accessKey": "",
                        "column": [],
                        "odpsServer": "",
                        "partition": "",
                        "project": "",
                        "table": "",
                        "truncate": true
                    }
                }
            }
        ],
        "setting": {
            "speed": {
                "channel": ""
            }
        }
    }
}

示例

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "****",
                        "password": "****",
                        "column": ["id","age","name"],
                        "connection": [
                            {
                                "table": [
                                    "test_table"
                                ],
                                "jdbcUrl": [
                                    "jdbc:mysql://127.0.0.1:3306/test"
                                ]
                            }
                        ]
                    }
                },
                "writer": {
                    "name": "odpswriter",
                    "parameter": {
                        "accessId": "****",
                        "accessKey": "****",
                        "column": ["id","age","name"],
                        "odpsServer": "http://service.odps.aliyun.com/api",
                        "partition": "pt='datax_test'",
                        "project": "datax_opensource",
                        "table": "datax_opensource_test",
                        "truncate": true
                    }
                }
            }
        ],
        "setting": {
            "speed": {
                "channel": 1
            }
        }
    }
}