debezium+kafka connector 实时采集mysql

最新推荐文章于 2024-12-16 23:54:24 发布

原创最新推荐文章于 2024-12-16 23:54:24 发布 · 1k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#mysql #kafka #数据库

大数据同时被 2 个专栏收录

12 篇文章

订阅专栏

数据湖

4 篇文章

订阅专栏

本文介绍了如何配置MySQL以启用二进制日志，并创建用户和授权，然后详细阐述了安装和启动Kafka及Kafka-Connect的过程。接着，展示了如何配置和启动Debezium MySQL Connector，用于将数据库变更实时同步到Kafka。最后，提到了可能遇到的问题以及相关API的使用，并提供了相关文档链接。

配置mysql

创建用户并授权

创建用户

CREATE USER 'debezium'@'%' IDENTIFIED BY 'password';

授予权限

GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium' IDENTIFIED BY 'password';

FLUSH PRIVILEGES;

启用二进制日志

1.检查是否支持二进制日志

SELECT variable_value as "BINARY LOGGING STATUS (log-bin) ::" FROM information_schema.global_variables WHERE variable_name='log_bin';

2.如果是OFF，请使用以下属性配置您的 MySQL 服务器配置文件并重启服务器，如下表所述：

log_bin           = mysql-bin
binlog_format     = ROW
binlog_row_image  = FULL
expire_logs_days  = 10

3.检查二进制文件

SELECT variable_value as "BINARY LOGGING STATUS (log-bin) ::" FROM information_schema.global_variables WHERE variable_name='log_bin';

启用gdit

因为现在使用的单节点暂时未开启

安装kafka

启动kafka-connect

修改config/connect-distributed.properties的plugin.path和bootstrap.servers的值

修改日志文件 config/connect-log4j.properties

log4j.rootLogger=INFO, kafkaConnectAppender

log4j.appender.kafkaConnectAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.kafkaConnectAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.kafkaConnectAppender.File=/hadoop/kafka_2.11-0.11.0.2/logs/kafka-connect.log
log4j.appender.kafkaConnectAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaConnectAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR

启动kafka-connnector bin/connect-distributed.sh -daemon config/connect-distributed.properties

api查看kafka信息http://ip:8083

启动mysql-connector

1.配置mysql-connector文件

{
    "name": "inventory-connector", 
    "config": {
        "connector.class": "io.debezium.connector.mysql.MySqlConnector", 
        "database.hostname": "mysqlip", 
        "database.port": "3306", 
        "database.user": "debezium", 
        "database.password": "password", 
        "database.server.id": "184054", 
        "database.server.name": "fullfillment", 
        "database.include.list": "数据库1，数据库2", 
        "database.history.kafka.bootstrap.servers": "kafka:9092", 
        "database.history.kafka.topic": "dbhistory.fullfillment", 
        "include.schema.changes": "true" 
    }
}

其他详细配置参数官方文档https://debezium.io/documentation/reference/1.6/connectors/mysql.html#mysql-connector-properties

2.通过api启动mysql-connect

post方式提交http://ip:8083/connectors，参数为json文件

3.查看connect信息：get http://ip:8083/connectors 查看所有的connnectors

get http://ip:8083/connectors/inventory-connector/status 获取当前的状态

详细api请看：https://kafka.apache.org/documentation.html#connect_rest