ELK+Kafka搭建日志系统（单机版）、ESAPM追踪链系统

ELK+Kafka搭建分布式日志系统、ESAPM追踪链系统

系统配置

查看linux服务器信息
[root@linux-node2 ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@linux-node2 ~]# uname -r
2.6.32-573.el6.x86_64

涉及组件
Elasticsearch-7.5.1
Logstash-7.5.1
Kibana-7.5.1
Filebeat-7.5.1
zookeeper-3.6.0
Kafka_2.13-2.4.0
Tomcat8.x
Jdk 8.x

软件安装

Elasticsearch-7.6.1：
https://blog.youkuaiyun.com/linpxing1/article/details/105871992

非root用户启动

[es@linux-node2 ~] es9200/bin/./elasticsearch -d

Logstash-7.6.1：
https://blog.youkuaiyun.com/linpxing1/article/details/105872104
https://blog.youkuaiyun.com/linpxing1/article/details/105872748

非root用户启动

[es@linux-node2 ~]#  su es
[es@linux-node2 ~]# nohup ./bin/logstash -f in_or_output_cfg/bee.conf > logs/logstash.log 2>&1 &

Filebeat-7.6.1：
https://blog.youkuaiyun.com/linpxing1/article/details/105872791

非root用户启动

[es@linux-node2 ~]# nohup ./filebeat -c filebeat.yml -e > logs/filebeat.log 2>&1 &

zookeeper-3.6.0 假集群模式

1,分别copy 解压好的目录 成三份

drwxr-xr-x  8 es   es         155 3月  11 09:52 zookeeper2181
drwxr-xr-x  8 es   es         147 3月  11 09:50 zookeeper2182
drwxr-xr-x  8 es   es         147 3月  11 09:50 zookeeper2183

2,修改配置文件
vim zookeeper2181/conf/zoo.cfg

tickTime=2000
dataDir=/app/soft/zookeeper2181/data
dataLogDir=/app/soft/zookeeper2181/logs
clientPort=2181

initLimit=5
syncLimit=2
server.1=192.168.9.93:2888:3888
server.2=192.168.9.93:4888:5888
server.3=192.168.9.93:6888:7888

vim zookeeper2181/data/myid
1
:wq

vim zookeeper2182/conf/zoo.cfg

tickTime=2000
dataDir=/app/soft/zookeeper2182/data
dataLogDir=/app/soft/zookeeper2182/logs
clientPort=2182

initLimit=5
syncLimit=2
server.1=192.168.9.93:2888:3888
server.2=192.168.9.93:4888:5888
server.3=192.168.9.93:6888:7888

配置myid zookeeper选举时使用
vim zookeeper2182/data/myid
2
:wq

vim zookeeper2183/conf/zoo.cfg

tickTime=2000
dataDir=/app/soft/zookeeper2183/data
dataLogDir=/app/soft/zookeeper2183/logs
clientPort=2183

initLimit=5
syncLimit=2
server.1=192.168.9.93:2888:3888
server.2=192.168.9.93:4888:5888
server.3=192.168.9.93:6888:7888    

vim zookeeper2183/data/myid
3
:wq

切换成非root用户

依次启动
zookeeper2181/bin/zkServer.sh start
zookeeper2182/bin/zkServer.sh start
zookeeper2183/bin/zkServer.sh start

查看集群状态：
zookeeper2181/bin/zkServer.sh status
zookeeper2182/bin/zkServer.sh status
zookeeper2181/bin/zkServer.sh status

jps查看进程

[es@w102 soft]$ jps
22982 QuorumPeerMain
23047 QuorumPeerMain
23131 QuorumPeerMain

zookeeper 进程有，flower leader flower 则说明正常启动！
ps: 注意文件夹名称不能有 - 比如 /app/soft-ss/bb 会快速失败，文件夹权限授权失败；

Kafka 安装(单机版)

kafka可以通过官网下载：https://kafka.apache.org/downloads
kafka根据Scala版本不同，又分为多个版本，我不需要使用Scala，所以就下载官方推荐版本kafka_2.13-2.4.0.tgz

解压文件
为文件创建 软链接 ln -s kafka_2.13-2.4.0 kafka

配置kafka的配置文件
vim config/server.properties

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://192.168.9.93:9092

# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://kafka93:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/app/soft/kafka/logs

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=3

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=192.168.9.93:2181,192.168.9.93:2182,192.168.9.93:2183
#zookeeper.connect=192.168.9.93:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000


############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

# The largest record batch size allowed by Kafka.
message.max.bytes=100000000

重要的配置

broker.id是kafka broker的编号，集群里每个broker的id需不同。从0开始
listeners是监听地址，需要提供外网服务的话，要设置本地的IP地址
广播监听地址
advertised.listeners
Zookeeper连接配置
num.partitions 为新建Topic的默认Partition数量，partition数量提升，一定程度上可以提升并发性

启动脚本：

./bin/kafka-server-start.sh -daemon config/server.properties 后台启动

查看topic list:

bin/kafka-topics.sh --list --zookeeper kafka93:2181,kafka93:2182,kafka93:2183

创建生产组：

bin/kafka-console-producer.sh --broker-list kafka93:9092 --topic beelogs 回车后为要发送的内容

创建消费组：

bin/kafka-console-consumer.sh --bootstrap-server kafka93:9092  --consumer-property group.id=test_kafka_bee-g  --topic beelogs

查看队列阻塞情况（前提是有消费组）：

bin/kafka-topics.sh --zookeeper kafka93:2181,kafka93:2182,kafka93:2183 --describe  --topic beelogs

删除主题

bin/kafka-topics.sh --zookeeper  kafka93:2181,kafka93:2182,kafka93:2183 --delete  --topic beelogs

注：到此所有要安装的软件已经安装好，现在要做的是将之串联起来，形成正确的数据流

将软件的数据进行串联

日志系统的数据流

logfile > monitor > filebeat > kafka > logstash > elasticsearch > kibana

filebeat.yml

#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
  # Change to true to enable this input configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /app/fileRoot_base/logs/*/*.txt
    - /app/fileRoot_base/logs/*/framework/*.txt
    #- c:\programdata\elasticsearch\logs\* 
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  fields:
     from: beelogs
  #  level: debug
  #  review: 1

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${
   
   path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings: