mysql 数据实时到 hdfs和kafka

原创

已于 2022-03-16 09:33:59 修改 · 468 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#hdfs #mysql #kafka

于 2019-11-28 18:07:01 首次发布

本文介绍了如何将MySQL中的数据实时同步到HDFS和Kafka，包括创建MySQL表，设置Flume配置文件，配置HDFS和Kafka的sink，以及增量数据的处理。测试显示数据已成功采集并消费。

1.在mysql创建表导入数据

use test;

create table wlslog
(id int not null,
time_stamp varchar(40),
category varchar(40),
type varchar(40),
servername varchar(40),
code varchar(40),
msg varchar(40),
primary key ( id )
);

insert into wlslog(id,time_stamp,category,type,servername,code,msg) values(1,‘apr-8-2014-7:06:16-pm-pdt’,‘notice’,‘weblogicserver’,‘adminserver’,‘bea-000365’,‘server state changed to standby’);
insert into wlslog(id,time_stamp,category,type,servername,code,msg) values(2,‘apr-8-2014-7:06:17-pm-pdt’,‘notice’,‘weblogicserver’,‘adminserver’,‘bea-000365’,‘server state changed to starting’);
insert into wlslog(id,time_stamp,category,type,servername,code,msg) values(3,‘apr-8-2014-7:06:18-pm-pdt’,‘notice’,‘weblogicserver’,‘adminserver’,‘bea-000365’,‘server state changed to admin’);
insert into wlslog(id,time_stamp,category,type,servername,code,msg) values(4,‘apr-8-2014-7:06:19-pm-pdt’,‘notice’,‘weblogicserver’,‘adm