datax hdfs2mysql.json模板

关注

原创最新推荐文章于 2026-07-08 14:08:31 发布 560 阅读 AI 写同款· GEO 优化 ›

DataX - 全量数据同步工具_datax工具通常情况下，离线数据同步任务需要每日定时重复执行（就像我们之前 flume 上传到 HDFS 也指定过），故HDFS上的目标路径通常会包含一层日期，以对每日同步的数据加以区分，也就是说每日同步数据的目标路径不是固定不变的，因此DataX配置文件中 HDFS Writer 的path参数的值应该是动态的。当出现上面的结果说明安装成功，这里我们用的是 DataX 自带的一个测试作业，它是一个 json 格式的文件，之后我们的 DataX 作业也是通过自己编写 json 文件来实现。阅读详情

运行如下命令可以查看官方文档
datax.py  -r hdfsreader -w mysqlwriter

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
Please refer to the hdfsreader document:
     https://github.com/alibaba/DataX/blob/master/hdfsreader/doc/hdfsreader.md 
Please refer to the mysqlwriter document:
     https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md

hdfs2mysql.json
{
 "job": {
  "content": [
   {
    "reader": {
     "name": "hdfsreader",
     "parameter": {
      "column": [
       "*"
      ],
      "defaultFS": "hdfs://xxxx:8020",
      "encoding": "UTF-8",
      "fieldDelimiter": "|",
      "fileType": "text",
      "path": "/xxxx",
      "nullFormat":"\\N" #处理hdfs中的Null值
     }
    },
    "writer": {
     "name": "mysqlwriter",
     "parameter": {
      "column": [
       "xm",
       "total_days",
       "total_count",
       "avg_czcs",
       "preday_counts"
      ],
      "connection": [
       {
        "jdbcUrl": "jdbc:mysql://ip:port/db?useUnicode=true&characterEncoding=utf-8",
        "table": [
         "table_name"
        ]
       }
      ],
      "password": "密码",
      "preSql": [],
      "session": [],
      "username": "用户名",
      "writeMode": "insert"
     }
    }
   }
  ],
  "setting": {
   "speed": {
    "channel": "1"
   }
  }
 }
}

模板2: 如果不是所有列导出,需要指定列的对应关系,下标从0开始,
hdfsreader数字类型用long,用int会报错,type是必填字段

{
 "job": {
  "content": [
   {
    "reader": {
     "name": "hdfsreader",
     "parameter": {
      "column": [
       {
        "index": 0,
        "type": "long"
       },
       {
        "index": 6,
        "type": "long"
       },
       {
        "index": 7,
        "type": "long"
       },
       {
        "index": 4,
        "type": "double"
       }
      ],
      "defaultFS": "hdfs://",
      "encoding": "UTF-8",
      "fieldDelimiter": "|",
      "fileType": "text",
      "path": "/*",
      "nullFormat": "\\N"
     }
    },
    "writer": {
     "name": "mysqlwriter",
     "parameter": {
      "column": [
       "czrID",
       "total_count",
       "total_days",
       "avg_count"
      ],
      "connection": [
       {
        "jdbcUrl": "",
        "table": [
         "ttable"
        ]
       }
      ],
      "password": "",
      "preSql": [""],
      "session": [],
      "username": "",
      "writeMode": "insert"
     }
    }
   }
  ],
  "setting": {
   "speed": {
    "channel": "1"
   }
  }
 }
}