关于linux使用docker部署ELK带安全认证并配置logstash同步mysql到ES

1.拉取ELK镜像

一、拉取es:7.6.1、kibana:7.6.1、logstash7.6.1的镜像

二、创建docker中elk的network

docker network create elk_network

查看网络

docker network ls

2.安装elasticserach

一、创建容器

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name es --net elk_network -d elasticsearch:7.6.1

创建成功后查看

二、进入容器内修改配置文件，添加安全认证

[root@VM-12-5-centos ~]# docker exec -it es bash
[root@7efcdd48229b elasticsearch]# ls
LICENSE.txt  README.asciidoc  config  jdk  logs     plugins
NOTICE.txt   bin              data    lib  modules
[root@7efcdd48229b elasticsearch]# cd config/
[root@7efcdd48229b config]# ls
elasticsearch.keystore  jvm.options        role_mapping.yml  users
elasticsearch.yml       log4j2.properties  roles.yml         users_roles
[root@7efcdd48229b config]# vi elasticsearch.yml 

修改elasticsearch.yml文件，开启安全认证

cluster.name: "docker-cluster"
network.host: 0.0.0.0

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

重启es并设置密码

docker restart es
cd bin
elasticsearch-setup-passwords interactive
// 输出
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]Y
 
 
Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

退出重启es

exit
docker restart es

访问界面 地址 http://ip:9200

3.安装kibana

一、创建容器

docker run -d --name kibana --privileged=true --net elk_network -p 5601:5601    kibana:7.6.1

二、进入容器内修改配置文件

[root@VM-12-5-centos ~]# docker exec -it kibana bash
bash-4.2$ ls
LICENSE.txt  bin	   data		 optimize      src
NOTICE.txt   built_assets  node		 package.json  webpackShims
README.txt   config	   node_modules  plugins       x-pack
bash-4.2$ cd config/
bash-4.2$ ls
apm.js	kibana.yml
bash-4.2$ vi kibana.yml 

修改kibana.yml文件(注：elasticsearch.hosts 修改为自己的IP地址以及密码)

server.name: kibana
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: [ "http://x.x.x.x:9200" ] 
xpack.monitoring.ui.container.elasticsearch.enabled: true

elasticsearch.username: "elastic"
elasticsearch.password: "123456a" 
i18n.locale: "zh-CN"

退出重启

exit
docker restart kibana

访问界面 地址 http://ip:5601

4.安装logstash并配置从mysql同步数据到es

一、创建挂载目录

logstash.yml配置文件

http.host: "0.0.0.0"

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: 123456a
xpack.monitoring.elasticsearch.hosts: ["http://x.x.x.x:9200"]

logstash.conf配置文件

input {
    stdin {
    }
    jdbc {
      jdbc_connection_string => "jdbc:mysql://x.x.x.x:3307/MSDI_PLUS?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC"
      jdbc_user => "*"
      jdbc_password => "*"
      jdbc_driver_library => "/usr/share/logstash/config/mysql-connector-java-8.0.18.jar"
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_paging_enabled => "true"
      jdbc_page_size => "300000"
      statement => "SELECT id, ID as code,NAME as name FROM T_PARTNER"
      schedule => "* * * * *"
    }
}
 
output {
    elasticsearch {
        hosts => ["x.x.x.x:9200"]
        index => "t_partner1"
        document_id => "%{id}"
        user => "elastic"
        password => "123456a"
    }
    stdout {
        codec => json_lines
    }
}

注：logstash.conf文件中不要出现中文，会解析失败，
这里我将配置文件目录挂载到外部方便随时更改，因为配置文件容易出错，另外mysql-connector-java-8.0.18.jar 需要手动到网页下载，

二、创建容器

docker run -it -d --privileged=true -p 5044:5044 --name logstash --net elk_network    -v /data/elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml -v /data/elk/logstash/config/mysql-connector-java-8.0.18.jar:/usr/share/logstash/config/mysql-connector-java-8.0.18.jar -v /data/elk/logstash/pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf logstash:7.6.1

设置完成后查看效果
数据库新增数据

查看logstash容器日志

docker logs -f -t --tail 1000 logstash 

注：logstash同步数据的原理是通过设置cron表达式 定时通过sql脚本查询对增量数据进行操作

查看kibana界面es数据是否写入