Centos下daemontools监控logstash进程

本文介绍如何使用Supervisor管理Logstash服务,包括安装配置Supervisor、设置Logstash监控及通过HTTP界面进行管理。

Logstash安装参考http://udn.yyuap.com/doc/logstash-best-practice-cn/get_start/daemon.html

首先用root用户,

$ yum install python-setuptools
$ easy_install supervisor
然后可以查看具体的安装supervisord的版本:

$ supervisord --version
3.3.0

利用easy install安装完后并不会创建一个Supervisor的配置文件,但幸运的是Supervisor包括一个通用的配置文件,然后可以

$ echo_supervisord_conf > /etc/supervisord.conf

在supervisord.conf添加要监测的文件:

[program:logstash_hy]
command=/home/hadoop1/bms/logstash-1.5.4/bin/logstash -f /home/hadoop1/bms/logstash-1.5.4/conf/haiyun.conf
logfile=/home/hadoop1/bms/logstash-1.5.4/logs/test.log

然后就可以启动supervisor,

/etc/init.d/supervisord -c /etc/supervisord.conf

使配置更改生效
每次修改主配置文件或增改子配置文件都需要执行supervisorctl update使新配置生效。或者用supervisorctl reload重新重新启动supervisor程序。


开启基于http的UI控制台

修改主配置文件 
vim /etc/supervisor/supervisord.conf,添加如下内容:
[inet_http_server]
port = 192.168.1.60:9001 #IP和绑定端口
username = admin #管理员名称
password = 123456 #管理员密码

这样,在浏览器中输入配置中写的地址和端口号,输入配置好的用户名和密码,便可以看到基于网页的控制台界面。从中,可以察看supervisor监控的>信息,可以察看进程的日志等。

这样就可以检测logstash的实时运行了


同时可以通过:supervisorctl查看管理监控的进程情况:

supervisor> help
 
Documented commands (type help <topic>):
========================================
EOF    exit  maintail  quit    restart   start   stop
clear  help  open      reload  shutdown  status  tail
 
supervisor> help stop
stop <processname>            Stop a process.
stop <processname> <processname>    Stop multiple processes
stop all                Stop all processes
  When all processes are stopped, they are stopped in
  reverse priority order (see config file)
supervisor> help status
status          Get all process status info.
status <name>     Get status on a single process by name.
status <name> <name>    Get status on multiple named processes.
 
#停止某个进程
supervisor> stop publisher_for_summary
publisher_for_summary: stopped
 
#查看此时此刻的状态
supervisor> status
publisher_for_detail RUNNING    pid 27557, uptime 0:05:41
publisher_for_detail_nt RUNNING    pid 27567, uptime 0:05:41
publisher_for_summary STOPPED    Feb 27 02:48 PM
publisher_for_summary_nt RUNNING    pid 27568, uptime 0:05:41
worker_for_detail_all RUNNING    pid 27581, uptime 0:05:41
worker_for_detail_recent RUNNING    pid 27582, uptime 0:05:41
worker_for_summary RUNNING    pid 27559, uptime 0:05:41
#发现被supervisorctl停掉的进程不会被自动重启
 
#开启刚才停掉的进程
supervisor> start publisher_for_summary
publisher_for_summary: started
supervisor> status
publisher_for_detail RUNNING    pid 27557, uptime 0:08:02
publisher_for_detail_nt RUNNING    pid 27567, uptime 0:08:02
publisher_for_summary RUNNING    pid 3035, uptime 0:00:04
publisher_for_summary_nt RUNNING    pid 27568, uptime 0:08:02
worker_for_detail_all RUNNING    pid 27581, uptime 0:08:02
worker_for_detail_recent RUNNING    pid 27582, uptime 0:08:02
worker_for_summary RUNNING    pid 27559, uptime 0:08:02
 
#停掉所有进程
supervisor> stop all
worker_for_detail_recent: stopped
worker_for_detail_all: stopped
publisher_for_summary_nt: stopped
publisher_for_detail_nt: stopped
publisher_for_summary: stopped
worker_for_summary: stopped
publisher_for_detail: stopped
supervisor> status
publisher_for_detail STOPPED    Feb 27 02:51 PM
publisher_for_detail_nt STOPPED    Feb 27 02:51 PM
publisher_for_summary STOPPED    Feb 27 02:51 PM
publisher_for_summary_nt STOPPED    Feb 27 02:51 PM
worker_for_detail_all STOPPED    Feb 27 02:51 PM
worker_for_detail_recent STOPPED    Feb 27 02:51 PM
worker_for_summary STOPPED    Feb 27 02:51 PM
 
#开启所有进程
supervisor> start all
publisher_for_detail: started
worker_for_summary: started
publisher_for_summary: started
publisher_for_detail_nt: started
publisher_for_summary_nt: started
worker_for_detail_all: started
worker_for_detail_recent: started
supervisor> status
publisher_for_detail RUNNING    pid 5111, uptime 0:00:15
publisher_for_detail_nt RUNNING    pid 5141, uptime 0:00:15
publisher_for_summary RUNNING    pid 5135, uptime 0:00:15
publisher_for_summary_nt RUNNING    pid 5147, uptime 0:00:15
worker_for_detail_all RUNNING    pid 5153, uptime 0:00:15
worker_for_detail_recent RUNNING    pid 5159, uptime 0:00:14
worker_for_summary RUNNING    pid 5112, uptime 0:00:15


参考资料:

http://liyangliang.me/posts/2015/06/using-supervisor/

http://www.alphadevx.com/a/455-Installing-Supervisor-and-Superlance-on-CentOS

http://blog.youkuaiyun.com/xia7139/article/details/9033483

<think>我们正在安装Elasticsearch集群(3节点),部署head插件,使用Logstash导入CSV数据,并获取mediamatch_usermsg数据。 由于版本兼容性非常重要,我们将选择当前较新且兼容的版本。这里选择Elasticsearch 7.16.3(发布于2021年12月,稳定且兼容CentOS 7)。 注意:Elasticsearch 8.x 有较大变化(如默认启用安全特性),为简化部署我们先使用7.x版本。 步骤概览: 1. 准备CentOS 7系统(3台,假设IP为192.168.1.10, 192.168.1.11, 192.168.1.12) 2. 安装Java(Elasticsearch 7需要Java 11) 3. 安装Elasticsearch(3节点) 4. 配置集群(节点兼具管理和数据功能) 5. 部署head插件(可以使用docker方式,或者使用elasticsearch-head插件) 6. 安装Logstash并配置导入CSV 7. 使用curl或Kibana获取mediamatch_usermsg数据 具体步骤: 1. 准备系统 确保三台机器的主机名和hosts解析(可选但建议): 192.168.1.10 node1 192.168.1.11 node2 192.168.1.12 node3 关闭防火墙(或开放端口): ```bash systemctl stop firewalld systemctl disable firewalld ``` SELinux设置为permissive: ```bash setenforce 0 sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config ``` 2. 安装Java 11 ```bash yum install java-11-openjdk-devel -y ``` 3. 安装Elasticsearch 添加Elasticsearch仓库并安装(所有节点): ```bash # 导入GPG key rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch # 添加仓库 cat > /etc/yum.repos.d/elasticsearch.repo << EOF [elasticsearch] name=Elasticsearch repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF # 安装 yum install elasticsearch-7.16.3 -y ``` 4. 配置Elasticsearch集群(每个节点修改配置文件) 主节点(node1)配置示例:/etc/elasticsearch/elasticsearch.yml ```yaml cluster.name: media-cluster # 引用[2]提到集群名称相同才能自动发现 node.name: node1 network.host: 0.0.0.0 http.port: 9200 discovery.seed_hosts: ["node1", "node2", "node3"] # 使用主机名,确保/etc/hosts解析 cluster.initial_master_nodes: ["node1", "node2", "node3"] # 初始主节点列表 node.master: true # 作为主节点 node.data: true # 作为数据节点 ``` 其他节点类似,修改node.name分别为node2和node3,其他配置相同。 启动服务并设置开机启动: ```bash systemctl daemon-reload systemctl enable elasticsearch systemctl start elasticsearch ``` 检查集群状态(任一节点): ```bash curl -X GET "localhost:9200/_cluster/health?pretty" ``` 应该看到"status"为"green"或"yellow",且"number_of_nodes"为3。 5. 部署head插件(这里使用docker方式,方便且不影响集群版本) 选择其中一台节点(如node1)安装docker: ```bash yum install -y yum-utils device-mapper-persistent-data lvm2 yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum install docker-ce -y systemctl start docker systemctl enable docker ``` 运行elasticsearch-head容器: ```bash docker run -d --name es-head -p 9100:9100 mobz/elasticsearch-head:5 ``` 访问:http://node1:9100 (输入集群中任意节点地址,如http://node1:9200) 注意:Elasticsearch 7.x需要配置跨域(CORS)才能让head插件访问,修改elasticsearch.yml(所有节点): ```yaml http.cors.enabled: true http.cors.allow-origin: "*" ``` 重启Elasticsearch服务。 6. 安装Logstash并导入CSV 在其中一个节点(如node1)上安装Logstash: ```bash yum install logstash-7.16.3 -y ``` 创建Logstash配置文件(例如:/etc/logstash/conf.d/csv_import.conf),配置三个pipeline分别导入三个CSV文件,或者一个pipeline处理多个文件(这里使用三个独立配置): 注意:需要根据CSV文件的实际列名和数据类型调整。 以mediamatch_usermsg.csv为例(假设列:id, user_id, message, created_at): ```conf input { file { path => "/path/to/mediamatch_usermsg.csv" start_position => "beginning" sincedb_path => "/dev/null" } } filter { csv { separator => "," columns => ["id", "user_id", "message", "created_at"] } # 转换数据类型(如果需要) mutate { convert => { "id" => "integer" "user_id" => "integer" } } date { match => ["created_at", "yyyy-MM-dd HH:mm:ss"] target => "@timestamp" } } output { elasticsearch { hosts => ["http://node1:9200", "http://node2:9200", "http://node3:9200"] index => "mediamatch_usermsg" document_id => "%{id}" } stdout {} } ``` 同样,为另外两个CSV文件创建配置文件。 启动Logstash(指定配置文件): ```bash /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/csv_import.conf ``` 注意:确保CSV文件的路径正确,且Logstash有权限读取。 7. 获取mediamatch_usermsg数据 导入数据后,可以使用curl查询: ```bash curl -X GET "http://node1:9200/mediamatch_usermsg/_search?pretty" ``` 或者使用Kibana(如果安装了)在Dev Tools中查询。 8. 注意事项 - 确保所有节点的系统时间同步(使用ntp) - 根据数据量调整Elasticsearch的堆内存(/etc/elasticsearch/jvm.options) - Logstash导入时注意字段类型映射,可以在配置中指定模板或让Elasticsearch自动生成 以上步骤完成了从安装到数据导入和查询的全过程。 §§相关问题§§ 1. 如何调整Elasticsearch的堆内存大小? 2. 如何为Logstash的CSV导入指定字段映射模板? 3. 如何安装Kibana并连接到这个集群? 4. Elasticsearch集群如何添加安全认证(用户名密码)? </think>### CentOS 7下Elasticsearch集群部署及数据导入指南 #### 一、环境准备 1. **系统要求** - CentOS 7 x64(3台节点,假设IP:`192.168.1.10/11/12`) - 推荐版本兼容组合: - Elasticsearch/Kibana/Logstash:`7.16.3`(长期支持版) - JDK:`OpenJDK 11` 2. **基础配置(所有节点)** ```bash # 关闭防火墙 systemctl stop firewalld systemctl disable firewalld # 禁用SELinux setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config # 修改文件句柄限制 echo "* soft nofile 65536" >> /etc/security/limits.conf echo "* hard nofile 65536" >> /etc/security/limits.conf ``` #### 二、部署3节点Elasticsearch集群 1. **安装JDK 11** ```bash yum install -y java-11-openjdk ``` 2. **安装Elasticsearch** ```bash # 添加ES仓库 cat > /etc/yum.repos.d/elasticsearch.repo <<EOF [elasticsearch] name=Elasticsearch repository baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 EOF # 安装ES yum install -y elasticsearch-7.16.3 ``` 3. **配置集群(所有节点)** 编辑 `/etc/elasticsearch/elasticsearch.yml`: ```yaml # 节点1配置 (192.168.1.10) cluster.name: media-cluster # 引用[2]:相同集群名称自动发现 node.name: node1 network.host: 0.0.0.0 http.port: 9200 discovery.seed_hosts: ["192.168.1.10", "192.168.1.11", "192.168.1.12"] cluster.initial_master_nodes: ["node1", "node2", "node3"] node.master: true # 启用管理节点 node.data: true # 启用数据节点 # 其他节点修改node.name即可 ``` 4. **启动服务并验证** ```bash systemctl daemon-reload systemctl enable elasticsearch systemctl start elasticsearch # 检查集群状态 curl -X GET "192.168.1.10:9200/_cluster/health?pretty" ``` 输出应包含 `"number_of_nodes": 3` 和 `"status": "green"` [^1] #### 三、部署Head插件 1. **安装Node.js** ```bash curl -sL https://rpm.nodesource.com/setup_14.x | bash - yum install -y nodejs ``` 2. **安装Head插件(在任意节点)** ```bash cd /opt git clone git://github.com/mobz/elasticsearch-head.git cd elasticsearch-head npm install ``` 3. **修改ES跨域配置(所有节点)** 在 `elasticsearch.yml` 追加: ```yaml http.cors.enabled: true http.cors.allow-origin: "*" ``` 重启ES:`systemctl restart elasticsearch` 4. **启动Head服务** ```bash cd /opt/elasticsearch-head npm run start & ``` 访问 `http://192.168.1.10:9100` 输入ES节点地址查看集群状态 #### 四、使用Logstash导入CSV数据 1. **安装Logstash** ```bash yum install -y logstash-7.16.3 ``` 2. **创建CSV导入配置** `/etc/logstash/conf.d/csv_import.conf`: ```ruby input { file { path => "/data/*.csv" # CSV存放路径 start_position => "beginning" } } filter { csv { separator => "," columns => ["field1", "field2", ...] # 替换为实际列名 } } output { elasticsearch { hosts => ["192.168.1.10:9200", "192.168.1.11:9200", "192.168.1.12:9200"] index => "%{[@metadata][filename]}" # 自动创建同名索引 } } ``` 3. **启动数据导入** ```bash # 将CSV文件放入/data目录 mv mediamatch_*.csv order_index_test.csv /data/ # 启动Logstash /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/csv_import.conf ``` #### 五、查询mediamatch_usermsg数据 1. **通过API查询** ```bash curl -X GET "192.168.1.10:9200/mediamatch_usermsg/_search?pretty" ``` 2. **通过Head插件查询** 访问 `http://192.168.1.10:9100` → 选择索引 `mediamatch_usermsg` → 执行查询 #### 六、验证结果 ```bash # 查看所有索引 curl -X GET "192.168.1.10:9200/_cat/indices?v" # 输出示例 health status index uuid pri rep docs.count green open mediamatch_usermsg abcdefg123456 1 1 15000 green open mediamatch_userevent hijklmn789012 1 1 20000 green open order_index_test opqrstu345678 1 1 5000 ``` > **关键说明**: > 1. 节点兼具管理/数据功能通过 `node.master: true` + `node.data: true` 实现 > 2. 集群自动发现依赖 `discovery.seed_hosts` 配置[^1][^2] > 3. CSV导入时需根据实际文件调整列名(修改filter.csv.columns) --- ###
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值