logstash抽取数据库发送到kafka，以及使用logstash的something

最新推荐文章于 2023-08-11 21:31:46 发布

原创最新推荐文章于 2023-08-11 21:31:46 发布 · 397 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#kafka #大数据

Big data 同时被 3 个专栏收录

10 篇文章

订阅专栏

kafka

5 篇文章

订阅专栏

ETL

4 篇文章

订阅专栏

本文介绍了如何使用Logstash从数据库中抽取数据，并将其发送到Kafka，同时提到了Logstash配置中关于时间戳的存储格式以及定时任务的设置，如九点执行一次或每小时九点执行。

input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql://ip:PORT/database?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=utf8&useSSL=false&allowMultiQueries
=true&zeroDateTimeBehavior=convertToNull"
        jdbc_user => ""
        jdbc_password => ""
        jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        statement => "SELECT jl.*,GROUP_CONCAT(jll.COMPANY_NAME) COMPANY_NAMES FROM (SELECT j.* FROM JUDRISK_LAWSUIT j WHERE j.ID > :sql_last_value LIMIT 15000) jl LEFT JOIN
 JUDRISK_LAWSUIT_LIST jll ON jl.ID = jll.JUDRISK_LAWSUIT_ID GROUP BY jl.ID ORDER BY jl.ID"
        schedule => "* 3-5 * * *" # 3点到5点跑
        record_last_run => true
        use_column_value => true
        jdbc_page_size => 1500
        jdbc_paging_enabled => true
        tracking_column => "ID"
       # tracking_column_type => "timestamp"
        last_run_metadata_path => "/data/logstash-6.5.2/last_id_record/judrisk_lawsuit_ID"
        clean_run => false
        lowercase_column_names => false

   }
}

filter {}

output {
   kafka{
    bootstrap_servers => "ip:port,"
    topic_id => "topic"
    max_request_size => 90000000
    codec => "json"
  }

}