logstash + kafka

logstash和kafka整合首先要注意软件版本之间的兼容性；

logstash 2.3.1

kafka 2.11-0.8.2.2

第一步：启动zookeeper

cd /home/wql/app/zookeeper-3.4.10

./bin/zkServer.sh start

第二步：启动kafka

cd /home/wql/app/kafka/bin

nohup ./kafka-server-start.sh /home/wql/app/kafka/config/server.properties &

启动消费者：

./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic accesslog

第三步：启动logstash

定义：flow-kafka.conf

input {

  file {

    path => "/home/wql/app/nginx_logs/*"

    discover_interval => 5

    start_position => "beginning"

  }

}

output {

        kafka {

          topic_id => "accesslog"

          codec => plain {

                format => "%{message}"

                charset => "UTF-8"

          }

          bootstrap_servers => "localhost:9092"

        }

}

 

启动程序：

cd /home/wql/app/elk2/logstash-2.3.1

./bin/logstash agent -f flow-kafka.conf