mac部署logstash,并增量同步mysql数据到elasticsearch

最新推荐文章于 2024-07-14 14:26:26 发布

何超杰

最新推荐文章于 2024-07-14 14:26:26 发布

阅读量999

点赞数

分类专栏： logstash

本文链接：https://blog.youkuaiyun.com/hechaojie_com/article/details/88528021

版权

logstash 专栏收录该内容

1 篇文章

订阅专栏

1.下载 logstash-6.2.3.tar.gz

2.解压

3.安装插件
bin/logstash-plugin install logstash-input-jdbc
bin/logstash-plugin install logstash-output-elasticsearch

4.logstash-6.2.3下新建目录
postion目录和jar目录

5.将mysql驱动mysql-connector-java-5.1.46.jar复制到jar目录

6.修改config/pipelines.yml文件

- pipeline.id: table1
  path.config: "pipelines"

注：path.config为执行运行sql的配置文件目录

7.logstash-6.2.3目录下新建pipelines目录
新建配置文件school.conf

input {
 stdin { }
    jdbc {
        jdbc_connection_string => "jdbc:mysql://xxx/common?characterEncoding=UTF-8"
        jdbc_user => "root"
        jdbc_password => "root"
        jdbc_driver_library => "./jar/mysql-connector-java-5.1.46.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_paging_enabled => "true"
        jdbc_page_size => "500"
        schedule => "* * * * *"
        use_column_value => true
        tracking_column => update_at
        tracking_column_type => "numeric"
        record_last_run => true
        last_run_metadata_path => "./postion/school_postion.txt"
        statement => "select t.*,t.university_id as id from sys_university t where update_at > :sql_last_value"
    }
 }
  
 output {
     stdout {
        codec => json_lines
    }
    elasticsearch {
        hosts => "10.1.0.176:9200"
        index => "common_index"
        document_type => "sys_university"
        document_id => "%{id}"
    }
}

注：
#如果为ture,需要提供增量查询字段,如果update_at
use_column_value => true
#指定增量的字段名
tracking_column => update_at
#增量字段的类型
tracking_column_type => “numeric”
#记录最后一条数据结果
record_last_run => true
#保存位置
last_run_metadata_path => “./postion/school_postion.txt”

8.运行
./bin/logstash