canal-adapter实现mysql同步es

最新推荐文章于 2025-03-09 10:19:54 发布

liustcau

最新推荐文章于 2025-03-09 10:19:54 发布

阅读量870

点赞数

分类专栏：数据同步文章标签： mysql elasticsearch

本文链接：https://blog.youkuaiyun.com/liustcau/article/details/116017822

版权

本文介绍了如何通过canal-admin配置canal-instance从MySQL的yuncang_order01.yc_order表中同步数据到MQ，并通过canal-adapter将数据进一步同步到Elasticsearch。配置包括过滤表和字段，使用topic根据分片键分区。在消费类处理中，针对UPDATE操作，根据old字段判断更新情况，并进行异常队列的处理。此外，还提供了工程代码的链接。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

原始业务表数据同步

canal-admin

配置canal-admin 同步mysqlbinlog到mq中

服务器：10.****

canal.admin-1.1.4.tar.gz
tar -xvf canal.admin-1.1.4.tar.gz -C ./canal-admin/
vim application.yml


server:
  port: 8089
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8

spring.datasource:
  address: test:3306
  database: canal_manager_yuncang
  username: root
  password: 
  driver-class-name: com.mysql.jdbc.Driver
  url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false
  hikari:
    maximum-pool-size: 30
    minimum-idle: 1

canal:
  adminUser: admin
  adminPasswd: admin

sh startup.sh

访问地址：http://10****:8089/ admin 123456

canal-deployer

采用 canal-admin配置instance

 mkdir canal-deployer
tar -xvf canal.deployer-1.1.4.tar.gz -C ./canal-deployer


vim canal_local.properties
# register ip
canal.register.ip = 10.***

# canal admin config
canal.admin.manager = 10.****:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster =


sh startup.sh local

cannal-instace 配置

###集群主配置

#################################################
######### 		common argument		#############
#################################################
# tcp bind ip
canal.ip = 
# register ip to zookeeper
canal.register.ip = 
canal.port = 11111
canal.metrics.pull.port = 11112
# canal instance user/passwd
#canal.user = canal
#canal.passwd = E3619321C1A937C46A0D8BD1DAC39F93B27D4458

# canal admin config
canal.admin.manager = 10.***:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441

canal.zkServers = 10.****:2181
# flush data to zk
canal.zookeeper.flush.period = 1000
canal.withoutNetty = false
# tcp, kafka, RocketMQ
canal.serverMode = RocketMQ
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024 
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE
canal.instance.memory.rawEntry = true

## detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false

# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.s