Canal学习文档

牛马java转生嵌入式

于 2023-09-11 11:32:56 发布

阅读量1k

点赞数

文章标签：学习 java kafka

本文链接：https://blog.youkuaiyun.com/weixin_51359864/article/details/132802434

版权

本文是Canal学习文档，介绍了Canal日志及配置位置、部署问题及解决方法，如异步读取batchId重复消费、类型转换错误等。还提及客户端集群部署方式，以及Canal集成Kafka的服务端和客户端配置，包括配置文件和获取消息的方式等。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Canal学习文档


测试环境demo搭建

堡垒机地址：

测试部署地址：10.201.21.206

git工程地址：



```

重启

3.登录mysql查看是否开启bin-log日志

```
show variables like 'log_bin';
show variables like '%binlog_format%';
```



4、数据库用户需要配置的权限

用户需要拥有 SUPER  或 REPLICATION CLIENT 权限

查看权限命令

```sql
-- 查看当前用户权限
show GRANTS;
-- 查看其他用户权限 
SHOW GRANTS FOR 'username'@'%';
-- 查看数据库binlog日志状态
show variables like 'log_bin';

-- 查看binlog日志类型
show variables like '%binlog_format%';
```

二、linux搭建canal服务

1、登录linux服务器

2、创建canal启动脚本

```shell
vim /docker/canal/start-canal-server.sh
 
# 内容 容器名字 端口映射
docker run -d --name canal-server-prod -p 11111:11111 canal/canal-server:v1.1.4
```

3、执行脚本安装、启动canal

```shell
chmod 755 /docker/canal/start-canal-server-prod.sh
sh /docker/canal/start-canal-server-prod.sh
```

4、进入容器修改配置

```shell
docker exec -it canal-server-prod /bin/bash
```

5、编辑配置

```shell
vi canal-server/conf/example/instance.properties

# 几个重要的配置
# 数据库ip：端口
canal.instance.master.address=

# 数据库用户名密码
canal.instance.dbUsername=
canal.instance.dbPassword=

# 数据库监控过滤器
# 1. 所有表：.* or .*\\..*
# 2. canal schema下所有表： canal\\..*
# 3. canal下的以canal打头的表：canal\\.canal.*
# 4. canal schema下的一张表：canal\\.test1
# 5. 多个规则组合使用：canal\\..*,mysql.test1,mysql.test2 (逗号分隔)


![外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传](https://img-home.csdnimg.cn/images/20230724024159.png?origin_url=C%3A%5CUsers%5Cwufc01%5CAppData%5CRoaming%5CTypora%5Ctypora-user-images%5Cimage-20230607113632663.png&pos_id=img-HOORYbQN-1694400685327)



6、重启容器

\#按ctrl+D退出容器，并重启容器

```shell
docker restart canal-server-pre
```

7、进入容器查看日志

```
docker exec -it canal-server-pre /bin/bash
tail -100f canal-server/logs/example/example.log
```

8、再次查看日志，显示连接成功，正在同步

Canal日志以及配置位置：

1.canal部署位置：

在这里插入图片描述

2. 各个环境容器

请添加图片描述

3.canal容器常用命令

请添加图片描述

3.canal客户端源码（客户端配置流程）

进入canal客户端配置注入类

请添加图片描述

注入完配置后对client进行构建

请添加图片描述

构建完后运行

请添加图片描述

自动装配过滤器示例：

请添加图片描述

接受到消息后进行处理

请添加图片描述

过滤出要监听的表后将数据转交至数据处理器

请添加图片描述

构建数据对象（对应表的对象）

这里出现过因为用错类复制方法导致出现“”报错的问题，后面修改为了toBean

请添加图片描述

具体的细节忘了，把注释解开因该能遇到相同问题

进入到对应表监控处理器，类似xxljob的handler

Canal部署问题：

1.异步读取batchId重复消费 not firstly

串行读取

使用消息队列分发

2.读取数据库数据转化成实体的时候产生类型抓换错误“”不能转string

使用toBean，Inteager.parseInt()会导致字符串为""时报错

3.batch not exist

将scan = false

设置客户端重试机制

4.handler中如果有DS会使得handler不被注册进canal

5.表结构变更可能会影响同步阻塞

解决方法：删除conf/example/目录下的h2跟meta相关文件，重启canal，期间数据会丢失需要重新插入期间数据）

Canal客户端集群部署

通过mq进行多客户端订阅

Canal集成kafka

canal服务端kafka相关配置

canal.properties

# tcp, kafka, RocketMQ
canal.serverMode = kafka
##################################################
######### 		     MQ 		     #############
##################################################
canal.mq.servers = 10.201.21.206:9092,10.201.21.207:9092,10.201.21.205:9092
canal.mq.retries = 0
# flagMessage模式下可以调大该值, 但不要超过MQ消息体大小上限
canal.mq.batchSize = 16384
canal.mq.maxRequestSize = 1048576
# flatMessage模式下请将该值改大, 建议50-200
canal.mq.lingerMs = 100
canal.mq.bufferMemory = 33554432
# Canal的batch size, 默认50K, 由于kafka最大消息体限制请勿超过1M(900K以下)
canal.mq.canalBatchSize = 100
# Canal get数据的超时时间, 单位: 毫秒, 空为不限超时
canal.mq.canalGetTimeout = 100
# 是否为flat json格式对象
canal.mq.flatMessage = false
# kafka消息投递是否使用事务
canal.mq.transaction = false
canal.mq.compressionType = none
canal.mq.acks = all
#canal.mq.properties. =
canal.mq.producerGroup = test
# Set this value to "cloud", if you want open message trace feature in aliyun.
canal.mq.accessChannel = local
# aliyun mq namespace
#canal.mq.namespace =

##################################################
#########     Kafka Kerberos Info    #############
##################################################
canal.mq.kafka.kerberos.enable = false
canal.mq.kafka.kerberos.krb5FilePath = "../conf/kerberos/krb5.conf"
canal.mq.kafka.kerberos.jaasFilePath = "../conf/kerberos/jaas.conf"

instance.properties

# table regex 过滤表 监听amp_bis_db所有表。。。
canal.instance.filter.regex=amp_bis_db\\..*,amp_lease_process\\..*
# table black regex
canal.instance.filter.black.regex=
canal.instance.filter.query.dml = true
canal.instance.parser.parallelThreadSize=16
# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch
# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch

# mq config
# kafka topic配置
canal.mq.topic=canal-prod
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*  可以配置动态topic
canal.mq.partition=0
# hash partition config
canal.mq.partitionsNum=3
#canal.mq.partitionHash=test.table:id^name,.*\\..*

canal+kafka客户端

pom

ps：客户端需要单独引入kafka配置不然会报错

<dependency>
			<groupId>org.apache.kafka</groupId>
			<artifactId>kafka-clients</artifactId>
			<version>1.1.1</version>
			<scope>provided</scope>
		</dependency>

获取消息的方式：

1.使用List新建下列连接对象时需要设置flatMessage为true

2.使用List 新建下列连接对象时需要设置flatMessage为false

public KafkaCanalClient build() {
            KafkaCanalConnector connector = new KafkaCanalConnector(servers, topic, partition, groupId, batchSize, false); // 最后一项为flatMessage（boolean）
            KafkaCanalClient kafkaCanalClient = new KafkaCanalClient();
            kafkaCanalClient.setMessageHandler(messageHandler);
            kafkaCanalClient.setConnector(connector);
            kafkaCanalClient.filter = this.filter;
            kafkaCanalClient.unit = this.unit;
            kafkaCanalClient.batchSize = this.batchSize;
            kafkaCanalClient.timeout = this.timeout;
            return kafkaCanalClient;
        } // 以上值需要在yml文件里进行配置

report工程

yml配置：

canal:
  server: 10.201.21.206:9092,10.201.21.207:9092,10.201.21.205:9092
  destination: canal-prod // kafka对应topic
  user-name: canal
  password: canal
  async: true
  group-id: canal-report
  mode: kafka // canal模式

Canal生产环境配置：

配置文件

恢复默认配置（表结构变更可能会影响同步阻塞–解决方法：删除conf/example/目录下的h2跟meta相关文件，重启canal，期间数据会丢失需要重新插入期间数据）

instance.properties:

ql serverId , v1.0.26+ will autoGen
canal.instance.mysql.slaveId=22

# enable gtid use true/false
canal.instance.gtidon=false

# position info
canal.instance.master.address=localhost:3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=

# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=

# table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=ha%n4sTdC4zNspxg

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

# username/password
canal.instance.dbUsername=canal
canal.instance.dbPassword=SS\#c4$gdf*s
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

# table regex
canal.instance.filter.regex=amp_bis_db\\..*,amp_lease_process\\..*
# table black regex
canal.instance.filter.black.regex=
canal.instance.filter.query.dml = true
canal.instance.parser.parallelThreadSize=16
# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch
# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch

# mq config
canal.mq.topic=canal-prod
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*
canal.mq.partition=0
# hash partition config
canal.mq.partitionsNum=3
#canal.mq.partitionHash=test.table:id^name,.*\\..*
#################################################

canal.properties:

#################################################
######### 		common argument		#############
#################################################
# tcp bind ip
canal.ip =
# register ip to zookeeper
canal.register.ip =
canal.port = 11111
canal.metrics.pull.port = 11112
# canal instance user/passwd
# canal.user = canal
# canal.passwd = E3619321C1A937C46A0D8BD1DAC39F93B27D4458

# canal admin config
#canal.admin.manager = 127.0.0.1:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441

canal.zkServers =
# flush data to zk
canal.zookeeper.flush.period = 1000
canal.withoutNetty = false
# tcp, kafka, RocketMQ
canal.serverMode = kafka
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024 
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE
canal.instance.memory.rawEntry = true

## detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false

# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size =  1024
# mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60

# network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30

# binlog filter config
canal.instance.filter.druid.ddl = true
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false
canal.instance.filter.transaction.entry = false

# binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED 
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB

# binlog ddl isolation
canal.instance.get.ddl.isolation = false

# parallel parser config
canal.instance.parser.parallel = true
## concurrent thread number, default 60% available processors, suggest not to exceed Runtime.getRuntime().availableProcessors()
#canal.instance.parser.parallelThreadSize = 16
## disruptor ringbuffer size, must be power of 2
canal.instance.parser.parallelBufferSize = 256

# table meta tsdb info
canal.instance.tsdb.enable = true
canal.instance.tsdb.dir = ${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url = jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
canal.instance.tsdb.dbUsername = canal
canal.instance.tsdb.dbPassword = canal
# dump snapshot interval, default 24 hour
canal.instance.tsdb.snapshot.interval = 24
# purge snapshot expire , default 360 hour(15 days)
canal.instance.tsdb.snapshot.expire = 360

# aliyun ak/sk , support rds/mq
canal.aliyun.accessKey =
canal.aliyun.secretKey =

#################################################
######### 		destinations		#############
#################################################
canal.destinations = canal-prod
# conf root dir
canal.conf.dir = ../conf
# auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5

canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml
#canal.instance.tsdb.spring.xml = classpath:spring/tsdb/mysql-tsdb.xml

canal.instance.global.mode = spring
canal.instance.global.lazy = false
canal.instance.global.manager.address = ${canal.admin.manager}
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
canal.instance.global.spring.xml = classpath:spring/file-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml

##################################################
######### 		     MQ 		     #############
##################################################
canal.mq.servers = 10.201.17.76:9092,10.201.17.75:9092,10.201.17.74:9092  
canal.mq.retries = 0
canal.mq.batchSize = 16384
canal.mq.maxRequestSize = 1048576
canal.mq.lingerMs = 100
canal.mq.bufferMemory = 33554432
canal.mq.canalBatchSize = 100
canal.mq.canalGetTimeout = 100
canal.mq.flatMessage = false
canal.mq.compressionType = none
canal.mq.acks = all
#canal.mq.properties. =
#canal.mq.producerGroup = test
# Set this value to "cloud", if you want open message trace feature in aliyun.
canal.mq.accessChannel = local
# aliyun mq namespace
#canal.mq.namespace =

##################################################
#########     Kafka Kerberos Info    #############
##################################################
canal.mq.kafka.kerberos.enable = false
canal.mq.kafka.kerberos.krb5FilePath = "../conf/kerberos/krb5.conf"
canal.mq.kafka.kerberos.jaasFilePath = "../conf/kerberos/jaas.conf"
#canal.mq.producerGroup = test
# Set this value to "cloud", if you want open message trace feature in aliyun.
canal.mq.accessChannel = local
# aliyun mq namespace
#canal.mq.namespace =

##################################################
#########     Kafka Kerberos Info    #############
##################################################
canal.mq.kafka.kerberos.enable = false
canal.mq.kafka.kerberos.krb5FilePath = "../conf/kerberos/krb5.conf"
canal.mq.kafka.kerberos.jaasFilePath = "../conf/kerberos/jaas.conf"