前因

在一次机房突然断电之后，登陆到zabbix系统中查看，发现有一台agent没有连接到，登陆到agent机器查看发现zabbix-agent服务无法启动，查看zabbix-agent日志看到如下报错信息。

....
zabbix_agentd [836]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [839]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [842]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [846]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [856]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [860]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [864]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [867]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [871]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [897]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [907]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [911]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_agentd [916]: cannot open log: cannot create semaphore set: [28] No space left on device
....

解决方法

经过大量的排查，怀疑是kernel.sem(共享内存信号量)参数设置过小，查看kernel.sem参数设置。可以通过ipcs -lm命令查看目前系统共享内存的参数限制。

[root@qiniu zabbix]#  cat /proc/sys/kernel/sem
250	32000	32	128
kernel.sem 4个参数的意义

250       SEMMSL    max semaphores per array     信号集容纳最大信号数量   
32000     SEMMNS    max semaphores system wide   所有信号的最大数量
32        SEMOPM    max ops per semop call       调用单个信号集中最大信号数量
128       SEMMNI    max number of arrays         信号集的最大值

然后尝试将kernel.sem参数设置的大一点。

[root@qiniu zabbix]# vim /etc/sysctl.conf 
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
fs.file-max=65535
kernel.sem = 500  64000  64  256  //kernel.sem这个可以根据需求在适当的调大一些。
[root@qiniu zabbix]# sysctl -p  //使之设置生效

重启zabbix-agent之后发现No space left on device报错已经没有了。

注：
1.SELINUX需要关闭，否则zabbix-agent是无法启动的。
2.参考文档:CSDN
3.参考文档:imaojia

文章目录