【采集项目-（6)全量数据采集】

最新推荐文章于 2025-09-23 16:37:33 发布

原创

最新推荐文章于 2025-09-23 16:37:33 发布 · 712 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#数据库 #mysql

本文介绍如何使用DataX进行MySQL到HDFS的全量数据同步，包括配置文件生成脚本、批量生成配置文件脚本及全量表数据同步脚本等关键步骤。

DataX全量数据采集

前置操作

DataX安装：https://blog.youkuaiyun.com/Tonystark_lz/article/details/126393252?spm=1001.2014.3001.5501

DataX配置文件生成脚本

1）在~/bin目录下创建gen_import_config.py脚本(生成datax json文件）

 vim ~/bin/gen_import_config.py

脚本内容如下（已配置HA）：

#!/usr/bin/python
# ecoding=utf-8
import json
import getopt
import os
import sys
import MySQLdb

#MySQL相关配置，需根据实际情况作出修改
mysql_host = "hadoop102"
mysql_port = "3306"
mysql_user = "root"
mysql_passwd = "000000"

#HDFS NameNode相关配置，需根据实际情况作出修改
#hdfs_nn_host = "hadoop102"
#hdfs_nn_port = "8020"

#HDFS HA 相关配置，需根据实际情况作出修改
my_nameservices = "mycluster"
my_namenodes_1 = "nn1"
my_namenodes_2 = "nn2"
my_namenodes_3 = "nn3"
my_rpc_address_1 = "hadoop102:8020"
my_rpc_address_2 = "hadoop103:8020"
my_rpc_address_3 = "hadoop104:8020"

#生成配置文件的目标路径，可根据实际情况作出修改
output_path = "/opt/module/datax/job/import"


def get_connection():
    return MySQLdb.connect(host=mysql_host, port=int(mysql_port), user=mysql_user, passwd=mysql_passwd)


def get_mysql_meta(database, table):
    connection = get_connection()
    cursor = connection.cursor()
    sql = "SELECT COLUMN_NAME,DATA_TYPE from information_schema.COLUMNS WHERE TABLE_SCHEMA=%s AND TABLE_NAME=%s ORDER BY ORDINAL_POSITION"
    cursor.execute(sql, [database, table])
    fetchall = cursor.fetchall()
    cursor.close()
    connection.close()
    return fetchall


def get_mysql_columns(database, table):
    return map(lambda x: x[0], get_mysql_meta(database, table))


def get_hive_columns(database, table):
    def type_mapping(mysql_type):
        mappings = {
   
   
            "bigint": "bigint",
            "int": "bigint",
            "smallint": "bigint",
            "tinyint": "bigint",
            "decimal": "string",
            "double": "double",
            "float": "float",
            "binary": "string",
            "char": "string",
            "varchar": "string",
            "datetime": "string",
            "time":