datax hdfs2mysql.json模板

最新推荐文章于 2024-07-11 10:40:40 发布

原创最新推荐文章于 2024-07-11 10:40:40 发布 · 526 阅读

0 ·

CC 4.0 BY-SA版权

开发工具专栏收录该内容

13 篇文章

订阅专栏

本文介绍使用DataX工具将HDFS中的数据迁移到MySQL数据库的方法，包括配置文件的详细设置，如指定列映射、数据类型转换及连接参数等。

运行如下命令可以查看官方文档
datax.py  -r hdfsreader -w mysqlwriter

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
Please refer to the hdfsreader document:
     https://github.com/alibaba/DataX/blob/master/hdfsreader/doc/hdfsreader.md 
Please refer to the mysqlwriter document:
     https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md

hdfs2mysql.json
{
 "job": {
  "content": [
   {
    "reader": {
     "name": "hdfsreader",
     "parameter": {
      "column": [
       "*"
      ],
      "defaultFS": "hdfs://xxxx:8020",
      "encoding": "UTF-8",
      "fieldDelimiter": "|",
      "fileType": "text",
      "path": "/xxxx",
      "nullFormat":"\\N" #处理hdfs中的Null值
     }
    },
    "writer": {
     "name": "mysqlwriter",
     "parameter": {
      "column": [
       "xm",
       "total_days",
       "total_count",
       "avg_czcs",
       "preday_counts"
      ],
      "connection": [
       {
        "jdbcUrl": "jdbc:mysql://ip:port/db?useUnicode=true&characterEncoding=utf-8",
        "table": [
         "table_name"
        ]
       }
      ],
      "password": "密码",
      "preSql": [],
      "session": [],
      "username": "用户名",
      "writeMode": "insert"
     }
    }
   }
  ],
  "setting": {
   "speed": {
    "channel": "1"
   }
  }
 }
}

模板2: 如果不是所有列导出,需要指定列的对应关系,下标从0开始,
hdfsreader数字类型用long,用int会报错,type是必填字段

{
 "job": {
  "content": [
   {
    "reader": {
     "name": "hdfsreader",
     "parameter": {
      "column": [
       {
        "index": 0,
        "type": "long"
       },
       {
        "index": 6,
        "type": "long"
       },
       {
        "index": 7,
        "type": "long"
       },
       {
        "index": 4,
        "type": "double"
       }
      ],
      "defaultFS": "hdfs://",
      "encoding": "UTF-8",
      "fieldDelimiter": "|",
      "fileType": "text",
      "path": "/*",
      "nullFormat": "\\N"
     }
    },
    "writer": {
     "name": "mysqlwriter",
     "parameter": {
      "column": [
       "czrID",
       "total_count",
       "total_days",
       "avg_count"
      ],
      "connection": [
       {
        "jdbcUrl": "",
        "table": [
         "ttable"
        ]
       }
      ],
      "password": "",
      "preSql": [""],
      "session": [],
      "username": "",
      "writeMode": "insert"
     }
    }
   }
  ],
  "setting": {
   "speed": {
    "channel": "1"
   }
  }
 }
}