nagios小白故障的示例分析-创新互联
这篇文章主要为大家展示了“nagios小白故障的示例分析”，内容简而易懂，条理清晰，希望能够帮助大家解决疑惑，下面让小编带领大家一起研究并学习一下“nagios小白故障的示例分析”这篇文章吧。
运行了一年多的nagios突然出问题，网站可以打开，但是所有host、service都看不到状态。提示nagios maybe not running
第一反应是nagios挂了
重启nagios服务
[root@nagios-server ~]#/etc/init.d/nagiosstop Stopping nagios: done. [root@nagios-server ~]#/etc/init.d/nagiosstart Starting nagios: done.
刷新网页还是一样，清理浏览器缓存，重开浏览器还是看不到。
难道配置文件有错误？近期没改什么啊。
检查配置文件
[root@nagios-server ~]#/etc/init.d/nagioscheckconfig
没有error
服务器重启
[root@nagios-server ~]#reboot
还是不行
看看nagios错误日志吧
[root@nagios-server ~]#cat/usr/local/nagios/nagios.log
日志空空如也！
奇怪了，那再看看系统日志
[root@nagios-server ~]#cat/var/log/messages
满屏都是这种记录
The check of service 'MEM' on host 'xxx'looks like it was orphaned (results never came back). I'm scheduling an immediate check of theservice...
看的头疼，于是做了一个错误的决定，这个决定导致我丧失了第一时间发现问题根源的机会!
对，我把系统日志清空了
[root@nagios-server ~]#>/var/log/messages
整个世界清净了，再重启nagios服务，看日志会记录什么错误
[root@nagios-server ~]#/etc/init.d/nagiosstop [root@nagios-server ~]#/etc/init.d/nagiosstart [root@nagios-server etc]# tail -30/var/log/messages Dec 30 17:37:38 nagios-server nagios: Nagios3.5.1 starting... (PID=50752) Dec 30 17:37:38 nagios-server nagios:Local time is Fri Dec 30 17:37:38 CST 2016 Dec 30 17:37:38 nagios-server nagios:LOG VERSION: 2.0 Dec 30 17:37:38 nagios-server nagios:Warning: Host 'xxx' has no services associated with it! Dec 30 17:37:38 nagios-server nagios:Finished daemonizing... (New PID=50753) Dec 30 17:39:21 nagios-server nagios:Nagios 3.5.1 starting... (PID=51146) Dec 30 17:39:21 nagios-server nagios:Local time is Fri Dec 30 17:39:21 CST 2016 Dec 30 17:39:21 nagios-server nagios:LOG VERSION: 2.0 Dec 30 17:39:21 nagios-server nagios:Warning: Host 'xxx' has no services associated with it! Dec 30 17:39:21 nagios-server nagios:Lockfile '/usr/local/nagios/var/nagios.lock' looks like its already held byanother instance of Nagios (PID 50753). Bailing out...
嗯，发现异常了，nagios.lock被其他进程占用了。
4.根据这个异常提示打开度娘搜索，找了半天发现有个帖子说的情况和我很像。说是删除nagios.log、objects.cache、retention.dat，再重启nagios服务
[root@nagios-server ~]# cd/usr/local/nagios/var [root@nagios-server var]# ll total 36 drwxrwxr-x. 2 nagios nagios 24576 Dec 3000:00 archives -rw-r--r-- 1 nagios nagios 0 Dec 30 17:27 nagios.lock -rw-rw-r-- 1 nagios nagios 0 Dec 30 17:27 nagios.log -rw-r--r--. 1 nagios nagios 0 Dec 30 17:27 objects.cache -rw------- 1 nagios nagios 0 Dec 30 17:27 retention.dat drwxrwsr-x. 2 nagios nagcmd 4096 Dec 30 17:27 rw drwxr-xr-x. 4 root root 4096 Oct 22 2015 spool -rw-rw-r-- 1 nagios nagios 0 Dec 30 17:27 status.dat
怎么nagios.log、objects.cache、retention.dat文件大小全部为0呢？不管它，按照帖子操作试试。
到了这里，老鸟应该发现问题在哪了。但我忽略了这个重要细节，再一次丧失了发现问题根源的机会！
[root@nagios-server var]# rm -fobjects.cache retention.dat status.dat [root@nagios-server var]/etc/init.d/nagiosstop [root@nagios-server var]/etc/init.d/nagiosstart
然并卵，故障依旧。这个时候有点方了，不知道如何下手去排查。查看nagios.cfg、nrpe.cfg都正常。
5.反复重启nagios，发现另外一个异常点，nagios服务“貌似没启动”，注意这里的引号。
[root@nagios-server etc]#/etc/init.d/nagios start Starting nagios: done. [root@nagios-server etc]#/etc/init.d/nagios status nagios is not running
就是说虽然启动命令执行成功，但是查询nagios状态去提示没运行，这就奇怪了，难道nagios程序有问题了？
继续看日志。从日志中能看出重复启动了nagios，可能有多实例(instance)运行
[root@nagios-server etc]# tail -30/var/log/messages Dec 30 17:37:38 nagios-server nagios:Nagios 3.5.1 starting... (PID=50752) Dec 30 17:37:38 nagios-server nagios:Local time is Fri Dec 30 17:37:38 CST 2016 Dec 30 17:37:38 nagios-server nagios:LOG VERSION: 2.0 Dec 30 17:37:38 nagios-server nagios:Warning: Host 'xxx' has no services associated with it! Dec 30 17:37:38 nagios-server nagios:Finished daemonizing... (New PID=50753) Dec 30 17:39:21 nagios-server nagios:Nagios 3.5.1 starting... (PID=51146) Dec 30 17:39:21 nagios-server nagios:Local time is Fri Dec 30 17:39:21 CST 2016 Dec 30 17:39:21 nagios-server nagios:LOG VERSION: 2.0 Dec 30 17:39:21 nagios-server nagios:Warning: Host 'xxx' has no services associated with it! Dec 30 17:39:21 nagios-server nagios:Lockfile '/usr/local/nagios/var/nagios.lock' looks like its already held byanother instance of Nagios (PID 50753). Bailing out...
看一下nagios.lock的内容吧
[root@nagios-server etc]#echo /usr/local/nagios/var/nagios.lock
文件是空的。
把nagios进程干掉
[root@nagios-server ~]# kill -9 50753 [root@nagios-server ~]# kill -9 50753 -bash: kill: (50753) - No such process
找不到之前的日志了，反正N次重启，发现一个问题：其实nagios进程一直是启动的，只是它启动的时候，没有把PID写入/usr/local/nagios/var/nagios.lock里面！
手动把PID写进去
[root@nagios-server ~]# echo 44336 >/usr/local/nagios/var/nagios.lock [root@nagios-server ~]# /etc/init.d/nagiosstatus nagios (pid 44336) is running...
原来nagios启动脚本会去检查nagios.lock里面的内容，如果为空，就返回nagios is not running，但实际nagios进程是正常启动了的。
所以反复重启之后，系统日志里面会有这句话“nagios-server nagios: Lockfile '/usr/local/nagios/var/nagios.lock'looks like its already held by another instance of Nagios (PID 50753). Bailing out...”
发现这个问题之后，把所有PID杀掉，重启nagios。果然里面没有这个nagios.lock错误。取代的是满屏的“The check of service 'MEM' on host 'xxx' looks like it was orphaned(results never came back). I'mscheduling an immediate check of the service...”
改用谷歌，以这个关键字进行搜索nagios The check of service looks like it was orphaned (resultsnever came back)，在搜索的结果里面看了N多帖子，终于看到一句，去看看磁盘是不是没空间了。嗯，看看磁盘。
[root@nagios-server spool]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 8.7G 8.3G 1.1M 100% / tmpfs 935M 0 935M 0% /dev/shm /dev/sda1 194M 34M 151M 19% /boot [root@nagios-server spool]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda3 577088 104974 472114 19% / tmpfs 239320 1 239319 1% /dev/shm /dev/sda1 51200 39 51161 1% /boot
原因找到了！！！磁盘可用空间只有1.1M，之前的故障点都有合理的解释。为什么nagios.log日志为0;为什么nagios.lock是空的。全部都是磁盘没剩余空间的原因！！！
[root@nagios-server nagios]# du -a /var| sort -n -r | head -n 10
6044444 /var 5946296 /var/log 5882592 /var/log/httpd 3127308/var/log/httpd/error_log-20161225 2659808 /var/log/httpd/error_log 66488 /var/lib 54616 /var/lib/rpm 45952 /var/log/httpd/access_log-20161225 43136 /var/lib/rpm/Packages 31512 /var/log/httpd/access_log [root@nagios-server nagios]# du -sh /var 5.8G /var [root@nagios-server nagios]# cd/var/log/httpd/ [root@nagios-server httpd]# du -hsx * |sort -rh | head -10 3.0G error_log-20161225 2.6G error_log 45M access_log-20161225 31M access_log 7.2M access_log-20161211 7.1M access_log-20161218 2.0M access_log-20161204 532K ssl_request_log-20160630 460K ssl_access_log-20160630 140K ssl_request_log-20161024
教训：
回看之前的/var/log/messages，其实里面隐藏着非常重要的消息！
Dec 28 12:30:01 nagios-server auditd[1032]:Audit daemon is low on disk space for logging
Dec 28 13:30:01 nagios-server auditd[1032]:Audit daemon is suspending logging due to low disk space.
Dec 28 14:54:30 nagios-server nagios:Warning: The check of host 'xxx' looks like it was orphaned (results never cameback). I'm scheduling an immediate checkof the host...
Dec 28 14:54:30 nagios-server nagios:Warning: The check of host 'xxx' looks like it was orphaned (results never cameback). I'm scheduling an immediate checkof the host...
以上是“nagios小白故障的示例分析”这篇文章的所有内容，感谢各位的阅读！相信大家都有了一定的了解，希望分享的内容对大家有所帮助，如果还想学习更多知识，欢迎关注创新互联行业资讯频道！
另外有需要云服务器可以了解下创新互联scvps.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。
本文题目：nagios小白故障的示例分析-创新互联
标题链接：http://shouzuofang.com/article/dooigh.html