mysql applier with hadoop

最新推荐文章于 2022-09-26 08:52:19 发布

ylzhjlinux

最新推荐文章于 2022-09-26 08:52:19 发布

阅读量174

点赞数

CC 4.0 BY-SA版权

分类专栏： Hadoop 文章标签：数据库大数据

本文链接：https://blog.youkuaiyun.com/ylzhjlinux/article/details/84690017

Hadoop 专栏收录该内容

34 篇文章

订阅专栏

本文介绍了一种通过MySQL Applier实现Hadoop数据复制的方法，该方法通过连接MySQL主服务器并读取已提交的二进制日志事件，然后将这些事件写入HDFS文件。数据库更改事件被描述为表创建操作或表数据更改。数据以逗号分隔或其他可配置格式写入Hive/HDFS。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

MySQL Applier for Hadoop

Replication via the Hadoop Applier is implemented by connecting to the MySQL master and reading binary log events as soon as they are committed, and writing them into a file in HDFS. "Events" describe database changes such as table creation operations or changes to table data.

MySQL to HDFS Integration

The Hadoop Applier uses an API provided by libhdfs, a C library to manipulate files in HDFS. The library comes precompiled with Hadoop distributions.

It connects to the MySQL master to read the binary log and then:

Fetches the row insert events occurring on the master
Decodes these events, extracts data inserted into each field of the row, and uses content handlers to get it in the format required
Appends it to a text file in HDFS.

Databases are mapped as separate directories, with their tables mapped as sub-directories with a Hive data warehouse directory. Data inserted into each table is written into text files (named as datafile1.txt) in Hive / HDFS. Data can be in comma separated format; or any other, that is configurable by command line arguments.