linux下mysql oom killer

1、现象描述

昨晚新上线一个服务，有人做了一个del操作，接着mysql主从全部挂机，如下：

主：

Apr 6 22:56:58 wmappdb1 kernel: Out of memory: Kill process 32627 (mysqld) score 984 or sacrifice child
Apr 6 22:56:58 wmappdb1 kernel: Killed process 32627, UID 27, (mysqld) total-vm:54979084kB, anon-rss:31860436kB, file-rss:2044kB
Apr 6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: TCP connection to [10.249.1.37]:3306 failed !!!
Apr 6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: Removing service [10.249.1.37]:3306 from VS [10.249.1.116]:3306
Apr 6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: Executing [/usr/bin/killkeepalived.sh] for service [10.249.1.37]:3306 in

从：

Apr 6 22:56:58 wmappdb1 kernel: Out of memory: Kill process 32627 (mysqld) score 984 or sacrifice child
Apr 6 22:56:58 wmappdb1 kernel: Killed process 32627, UID 27, (mysqld) total-vm:54979084kB, anon-rss:31860436kB, file-rss:2044kB
Apr 6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: TCP connection to [10.249.1.37]:3306 failed !!!
Apr 6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: Removing service [10.249.1.37]:3306 from VS [10.249.1.116]:3306
Apr 6 22:57:03 wmappdb1 Keepalived_healthcheckers[18693]: Executing [/usr/bin/killkeepalived.sh] for service [10.249.1.37]:3306 in

2、问题处理过程

查看完以上log，发现linux系统发生了oom killer

原因分析

a、zabbix监控发现，当时load较高。
b、发现log中，swap等被用完。
Apr 6 22:39:17 wmappdb2 kernel: 12152 total pagecache pages
Apr 6 22:39:17 wmappdb2 kernel: 7320 pages in swap cache
Apr 6 22:39:17 wmappdb2 kernel: Swap cache stats: add 13734845, delete 13727525, find 192747597/194527644
Apr 6 22:39:17 wmappdb2 kernel: Free swap = 0kB
Apr 6 22:39:17 wmappdb2 kernel: Total swap = 16416760kB
Apr 6 22:39:17 wmappdb2 kernel: 8388607 pages RAM
Apr 6 22:39:17 wmappdb2 kernel: 217229 pages reserved
Apr 6 22:39:17 wmappdb2 kernel: 7591 pages shared

c、分析mysql 中临时表的使用，发现有部分逻辑使用了临时表，且type=memory，需进一步优化。

4.解决办法

a、规划各程序的内存分配，控制各程序的总内存使用上限，避免内存耗尽。
b、消除程序中数据集较大的临时表(memory)，采用myisam表替代。
c、调整mysql的oom_score值（保留措施）