zk+kafka+elk安装

最新推荐文章于 2024-04-23 10:32:43 发布

weixin_34129145

最新推荐文章于 2024-04-23 10:32:43 发布

阅读量159

点赞数

CC 4.0 BY-SA版权

文章标签：大数据 scala python

原文链接：https://my.oschina.net/anxiaole/blog/1560587

本文介绍了一个基于Kafka和Elasticsearch的日志集成方案，包括安装配置过程及常见问题解决方案。通过Logstash将Kafka中的日志数据导入Elasticsearch进行全文检索。

2019独角兽企业重金招聘Python工程师标准>>>

启动顺序

0、zk

/usr/local/kafka_2.12-0.11.0.1/bin/zookeeper-server-start.sh config/zookeeper.properties &

1、kafka

/usr/local/kafka_2.12-0.11.0.1/bin/kafka-server-start.sh config/server.properties &

2、es

安装ik分词器

电商项目商品搜索模块 - ES&ik分词器安装

启动脚本

/usr/local/elasticsearch-1.7.1-tmp/bin/elasticsearch -d;
/usr/local/elasticsearch-1.7.1-yys/bin/elasticsearch -d;
/usr/local/elasticsearch-1.7.1-yys2/bin/elasticsearch -d;

es相关配置

/usr/local/elasticsearch-1.7.1-tmp/config/elasticsearch.yml

node.name: "lxf-1"
network.host: 0.0.0.0
index:
  analysis:                   
    analyzer:      
      ik:
          
          type: org.elasticsearch.index.analysis.IkAnalyzerProvider
      ik_max_word:
          type: ik
          use_smart: false
      ik_smart:
          type: ik
          use_smart: true
index.analysis.analyzer.default.type : "ik"
script.groovy.sandbox.enabled: true
http.cors.allow-origin: "/.*/"
http.cors.enabled: true

/usr/local/elasticsearch-1.7.1-yys/config/elasticsearch.yml

node.name: "lxf-2"
script.groovy.sandbox.enabled: true
index:
  analysis:                   
    analyzer:      
      ik:
          
          type: org.elasticsearch.index.analysis.IkAnalyzerProvider
      ik_max_word:
          type: ik
          use_smart: false
      ik_smart:
          type: ik
          use_smart: true
http.cors.allow-origin: "/.*/"
http.cors.enabled: true

/usr/local/elasticsearch-1.7.1-yys2/config/elasticsearch.yml

node.name: "lxf-3"
script.groovy.sandbox.enabled: true
index:
  analysis:                   
    analyzer:      
      ik:
          
          type: org.elasticsearch.index.analysis.IkAnalyzerProvider
      ik_max_word:
          type: ik
          use_smart: false
      ik_smart:
          type: ik
          use_smart: true
http.cors.allow-origin: "/.*/"
http.cors.enabled: true

3、logstash

/usr/local/logstash-1.5.5/bin/logstash -f /usr/local/logstash-1.5.5/config/kafka2es.conf &

4、Kinaba

/usr/local/kibana-4.1.2-linux-x64/bin/kibana;

配置：/usr/local/logstash-1.5.5/config/kafka2es.conf

input{
	stdin{}
    kafka {
        codec => "plain"
        group_id => "logstash1"
        auto_offset_reset => "smallest"
        reset_beginning => true
        topic_id => "jsc-admin"
        #white_list => ["jsc-admin"]
        #black_list => nil
        zk_connect => "10.10.18.85:2181" # zookeeper的地址
   }

}
output {
	file {
			path => "/tmp/jsc-admin-log-%{+YYYY-MM-dd}.messages.gz"
			gzip => true
			# 输出日志到文件中
	}

	elasticsearch {
			host => ["10.10.18.85"]
			protocol => "http"
			index => "jsc-admin-%{+YYYY.MM.dd}"
	}
}

安装、运行kafka

http://orchome.com/6

启动后需要修改一下配置，才能让其他机器当做producer

[2016-10-09 15:36:41,877] ERROR Error when sending message to topic test with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

解决方法：

vi KAFKA_HOME/config/server.properties

修改listeners=PLAINTEXT://10.33.97.111:9092

为：

listeners=PLAINTEXT://10.10.18.85:9092

#10.10.18.85为kafka所在服务器ip

问题即可解决！

但是又带来一个问题，tomcat 日志文件里面狂报错！

2016-10-09 15:42:01-[WARN] [org.apache.kafka.common.network.Selector] Error in I/O with /0.0.0.0
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.kafka.common.network.Selector.poll(Selector.java:238)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
at java.lang.Thread.run(Thread.java:745)

不得已又要找解决方法：
其实方法很简单，只要不用localhost，换成ip地址即可！

[root@localhost bin]# /usr/local/kafka1/bin/kafka-console-producer.sh --broker-list 10.33.97.111:9092 --topic  test

另外一个session：

[root@localhost bin]# /usr/local/kafka1/bin/kafka-console-consumer.sh --zookeeper 10.33.97.111:2181 --from-beginning --topic test

为什么localhost不行，而ip地址可以，因为配置文件里面指定的是ip地址，而没使用localhost

2:kafka 启动失败：
提示：

Caused by: java.net.UnknownHostException: kafka1: kakfa1: unknown error
解决方法：/etc/hosts中加入本地解析
10.10.11.89 kafka01

logstash

logstash读取kafka中的数据，把数据保存到es中。

kafka2es.conf

input{
        stdin{}
    kafka {
        codec => "plain"
        group_id => "logstash1"
        auto_offset_reset => "smallest"
        reset_beginning => true
        topic_id => "hello"
        #white_list => ["hello"]
        #black_list => nil
        zk_connect => "10.10.18.85:2181" # zookeeper的地址
   }

}
output {
        file {
                        path => "/tmp/jsc/log-%{+YYYY-MM-dd}.messages.gz"
                        gzip => true
        }

        elasticsearch {
                        host => ["10.10.18.85"]
                        protocol => "http"
                        index => "jsc-%{+YYYY.MM.dd}"
        }
}

执行命令启动logstash

./logstash-1.5.5/bin/logstash -f logstash-1.5.5/config/kafka2es.conf

遇到的问题总结

1、刚开始的时候，我直接用java程序链接另一台服务器中的kafka，结果一直失败。

但是却可以正常写到本地的kafka中。于是，我猜测应该是防火墙或者端口被禁用了。

参考：教你如何开启/关闭ubuntu防火墙

2、端口或防火墙搞定之后，还是无法访问到其他服务器上的kafka，所以就要修改

vi KAFKA_HOME/config/server.properties

修改listeners=PLAINTEXT://10.33.97.111:9092

为：

listeners=PLAINTEXT://10.10.18.85:9092

#10.10.18.85为kafka所在服务器ip

3、es 5.x版本之后，es 官方不在维护kopf插件，需要自己从github下载，并自己使用额外应用服务器运行。（搞得我还在es 5.x的版本上费好大安装kopf>_<）

问题总结

kafka

log4j, [2017-11-07T18:09:11.288]  WARN: kafka.client.ClientUtils$: Fetching topic metadata with correlation id 1 for topics [Set(jsc-admin)] from broker [id:0,host:10.10.18.85,port:9092] failed
java.nio.channels.ClosedChannelException
        at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
        at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
        at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)




anxiaole@anxiaole-virtual-machine:/usr/local/logstash-1.5.5$ fg 5
./bin/logstash -f config/kafka2es.conf
[2017-11-07 17:35:36,676] INFO Accepted socket connection from /10.10.18.85:43249 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-11-07 17:35:36,681] INFO Client attempting to renew session 0x15f957099db0019 at /10.10.18.85:43249 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-11-07 17:35:36,682] INFO Invalid session 0x15f957099db0019 for client /10.10.18.85:43249, probably expired (org.apache.zookeeper.server.ZooKeeperServer)
[2017-11-07 17:35:36,683] INFO Closed socket connection for client /10.10.18.85:43249 which had sessionid 0x15f957099db0019 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-11-07 17:35:36,687] INFO Accepted socket connection from /10.10.18.85:43250 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-11-07 17:35:36,689] INFO Client attempting to establish new session at /10.10.18.85:43250 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-11-07 17:35:36,698] INFO Established session 0x15f957099db001a with negotiated timeout 6000 for client /10.10.18.85:43250 (org.apache.zookeeper.server.ZooKeeperServer)
log4j, [2017-11-07T17:35:36.890]  WARN: kafka.client.ClientUtils$: Fetching topic metadata with correlation id 2 for topics [Set(jsc-admin)] from broker [id:0,host:10.10.18.85,port:9092] failed
java.nio.channels.ClosedChannelException
        at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
        at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
        at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

~~原因：猜测是因为kafka所在服务器防火墙没有关闭或者kafka端口号没有开放导致的。解决办法：教你如何开启/关闭ubuntu防火墙~~

参考：WARN Fetching topic metadata with correlation id 955 for topics [Set(test3)] from broker [BrokerEndP

JAVA笔记：kafka集群配置遇到的问题

关于Eclipse开发环境下 Spark+Kafka 获取topic的时候连接出错

/usr/local/logstash-1.5.5/bin/logstash -f /usr/local/logstash-1.5.5/config/kafka2es.conf
[2017-11-07 18:18:25,487] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x15f95e7d19b000e, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
        at java.lang.Thread.run(Thread.java:748)

参考：

logstash-output elasticsearch插件使用

转载于:https://my.oschina.net/anxiaole/blog/1560587