Zabbix-故障预处理和报警升级

实验效果：实现httpd停止服务，zabbix执行远程命令进行恢复，1分钟之后如果恢复失败，发邮件给运维，1分钟之后，问题没解决发邮件给领导

1.环境规划

ip地址	主机
192.168.100.11	zabbix-server，anasible
192.168.100.12	zabbix-agent，httpd

2.zabbix-server安装

略

3.zabbix-agent安装

略

4.zabbix-agent上httpd安装

yum install httpd -y

5.自定义监控项

vim /etc/zabbix/zabbix_agentd.conf
UserParameter=httpd,ps -ef|grep -v grep|grep -c '/usr/sbin/httpd'
systemctl  restart zabbix-agent

 6.创建自定义触发器

6.创建自定义触发器 

7.创建邮件报警 

a：创建发件人

b：创建两个收件人

 c.boss用户需要超级管理权限

8.在zabbix-server上使用ansible远程重启httpd

yum install ansible -y
vim /etc/ansible/hosts
[web]
192.168.100.12  ansible_ssh_user='root' ansible_ssh_pass='123456'
​
ansible web -m shell -a 'systemctl restart httpd.service'

客户端效果

[root@oldboy ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: activating (start) since Tue 2019-08-06 22:20:08 CST; 4s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
 Main PID: 2894 (httpd)
   CGroup: /system.slice/httpd.service
           └─2894 /usr/sbin/httpd -DFOREGROUND
​
Aug 06 22:20:08 oldboy systemd[1]: Starting The Apache HTTP Server...
Aug 06 22:20:08 oldboy httpd[2894]: AH00558: httpd: Could not reliably determine the server's fully qua...ssage
Hint: Some lines were ellipsized, use -l to show in full.
​
[root@oldboy ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-08-06 22:20:13 CST; 295ms ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
 Main PID: 2894 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─2894 /usr/sbin/httpd -DFOREGROUND
           ├─2898 /usr/sbin/httpd -DFOREGROUND
           ├─2899 /usr/sbin/httpd -DFOREGROUND
           ├─2900 /usr/sbin/httpd -DFOREGROUND
           ├─2901 /usr/sbin/httpd -DFOREGROUND
           └─2902 /usr/sbin/httpd -DFOREGROUND

9.zabbix-server上的zabbix-agent启用远程命令功能

vim /etc/zabbix/zabbix_agentd.conf 
EnableRemoteCommands=1
AllowRoot=1
​
systemctl restart zabbix-agent.service

10.在zabbix-web上配置触发器动作

整体效果如下：

远程命令细节：

命令：ansible web -m shell -a 'systemctl restart httpd.service'

 11.测试zabbix-agent报警升级

测试1：

停止httpd服务

[root@oldboy ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: activating (start) since Tue 2019-08-06 22:20:08 CST; 4s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
 Main PID: 2894 (httpd)
   CGroup: /system.slice/httpd.service
           └─2894 /usr/sbin/httpd -DFOREGROUND

查看httpd服务状态

[root@oldboy ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-08-06 22:20:13 CST; 295ms ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
 Main PID: 2894 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─2894 /usr/sbin/httpd -DFOREGROUND
           ├─2898 /usr/sbin/httpd -DFOREGROUND
           ├─2899 /usr/sbin/httpd -DFOREGROUND
           ├─2900 /usr/sbin/httpd -DFOREGROUND
           ├─2901 /usr/sbin/httpd -DFOREGROUND
           └─2902 /usr/sbin/httpd -DFOREGROUND

测试2：

停止httpd服务

[root@oldboy ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: activating (start) since Tue 2019-08-06 22:20:08 CST; 4s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 2839 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
 Main PID: 2894 (httpd)
   CGroup: /system.slice/httpd.service
           └─2894 /usr/sbin/httpd -DFOREGROUND

#使用nc监听80端口，模拟httpd重启失败

yum install nc  -y
nc -l 80

至此,已经实现httpd停止服务，zabbix执行远程命令进行恢复，1分钟之后如果恢复失败，发邮件给运行，1分钟之后，问题没解决发邮件给领导