Zabbix_proxy重启无任何相关进程处理

作者：广州乐维工程师 许远

一次断电意外导致本地使用中的zabbix_proxy无法正常工作，随即进行排查。

一、 通过系统命令查看proxy状态原以为一切正常，但仔细查看zabbix有关服务时，却没有发现发现相应的proxy端口在使用。如图1、2.

图1

图2

那实际上zabbix_proxy是没在工作的。

二、 对其日志文件进行查看，如图3.查看命令：tail -n 1000 zabbix_proxy.log #筛选最新的1000行数据进行查看。

图3

从最新的几行报错可以出看是数据库连接失败引起的问题。

三、 查看数据库状态，图4.

图4

四、 查看指定目录下的是否有对应sock文件，图5

图5

没该文件。

五、 查看数据库报错日志error.log（查看命令使用了“tail -n 1000 error.log”），图6 #如果不清楚位置可使用命令 find / -name error.log 进行全局查找.

图6

可以尝试在my.cnf中配置相应的参数（>0），而打开数据库。

六、 配置数据库配置文件my.cnf，增加一行innodb_force_recovery = 1 ，如图7

图7

innodb_force_recovery

级别 含义

1 (SRV_FORCE_IGNORE_CORRUPT)忽略检查到的corrupt页

2 (SRV_FORCE_NO_BACKGROUND)阻止主线程的运行，如主线 程需要执行full purge操作，会导致crash

3 (SRV_FORCE_NO_TRX_UNDO)不执行事务回滚操作。

4 (SRV_FORCE_NO_IBUF_MERGE)不执行插入缓冲的合并操作。

5 (SRV_FORCE_NO_UNDO_LOG_SCAN)不查看重做日志，InnoDB存储引擎会将未提交的事务视为已提交

6 (SRV_FORCE_NO_LOG_REDO)不执行前滚的操作。

七、 重新启动数据库，再去查看数据库日志（error.log），图8.

图8

存在该部分错表。

八、 登录数据库，删除错表，图9

mysql>DROP TABLE mysql.innodb_index_stats;

mysql>DROP TABLE mysql.innodb_table_stats;

mysql>DROP TABLE mysql.slave_master_info;

mysql>DROP TABLE mysql.slave_relay_log_info;

mysql>DROP TABLE mysql.slave_worker_info;

图9

未找到该表。

九、 关闭数据库，删除旧文件，图10，图11

rm -rf /data/mysql/data/mysql/innodb_index_stats.*

rm -rf /data/mysql/data/mysql/innodb_table_stats.*

rm -rf /data/mysql/data/mysql/slave_master_info.*

rm -rf /data/mysql/data/mysql/slave_relay_log_info.*

rm -rf /data/mysql/data/mysql/slave_worker_info.*

图10

图11

十、 打开数据库，重新构建数据库表，

如下执行内容共5条，一条一条地复制执行即可： 图12为其中一条的执行结果。

USE mysql;

1.

CREATE TABLE innodb_index_stats (

database_name varchar(64) COLLATE utf8_bin NOT NULL,

table_name varchar(64) COLLATE utf8_bin NOT NULL,

index_name varchar(64) COLLATE utf8_bin NOT NULL,

last_update timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

stat_name varchar(64) COLLATE utf8_bin NOT NULL,

stat_value bigint(20) unsigned NOT NULL,

sample_size bigint(20) unsigned DEFAULT NULL,

stat_description varchar(1024) COLLATE utf8_bin NOT NULL,

PRIMARY KEY (database_name,table_name,index_name,stat_name)

) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0;

2.

CREATE TABLE innodb_table_stats (

database_name varchar(64) COLLATE utf8_bin NOT NULL,

table_name varchar(64) COLLATE utf8_bin NOT NULL,

last_update timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

n_rows bigint(20) unsigned NOT NULL,

clustered_index_size bigint(20) unsigned NOT NULL,

sum_of_other_index_sizes bigint(20) unsigned NOT NULL,

PRIMARY KEY (database_name,table_name)

) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0;

3.

CREATE TABLE slave_master_info (

Number_of_lines int(10) unsigned NOT NULL COMMENT 'Number of lines in the file.',

Master_log_name text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log currently being read from the master.',

Master_log_pos bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last read event.',

Host char(64) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT 'The host name of the master.',

User_name text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The user name used to connect to the master.',

User_password text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The password used to connect to the master.',

Port int(10) unsigned NOT NULL COMMENT 'The network port used to connect to the master.',

Connect_retry int(10) unsigned NOT NULL COMMENT 'The period (in seconds) that the slave will wait before trying to reconnect to the master.',

Enabled_ssl tinyint(1) NOT NULL COMMENT 'Indicates whether the server supports SSL connections.',

Ssl_ca text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Authority (CA) certificate.',

Ssl_capath text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path to the Certificate Authority (CA) certificates.',

Ssl_cert text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL certificate file.',

Ssl_cipher text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the cipher in use for the SSL connection.',

Ssl_key text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL key file.',

Ssl_verify_server_cert tinyint(1) NOT NULL COMMENT 'Whether to verify the server certificate.',

Heartbeat float NOT NULL,

Bind text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'Displays which interface is employed when connecting to the MySQL server',

Ignored_server_ids text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The number of server IDs to be ignored, followed by the actual server IDs',

Uuid text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The master server uuid.',

Retry_count bigint(20) unsigned NOT NULL COMMENT 'Number of reconnect attempts, to the master, before giving up.',

Ssl_crl text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Revocation List (CRL)',

Ssl_crlpath text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path used for Certificate Revocation List (CRL) files',

Enabled_auto_position tinyint(1) NOT NULL COMMENT 'Indicates whether GTIDs will be used to retrieve events from the master.',

PRIMARY KEY (Host,Port)

) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Master Information';

4.

CREATE TABLE slave_relay_log_info (

Number_of_lines int(10) unsigned NOT NULL COMMENT 'Number of lines in the file or rows in the table. Used to version table definitions.',

Relay_log_name text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the current relay log file.',

Relay_log_pos bigint(20) unsigned NOT NULL COMMENT 'The relay log position of the last executed event.',

Master_log_name text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log file from which the events in the relay log file were read.',

Master_log_pos bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last executed event.',

Sql_delay int(11) NOT NULL COMMENT 'The number of seconds that the slave must lag behind the master.',

Number_of_workers int(10) unsigned NOT NULL,

Id int(10) unsigned NOT NULL COMMENT 'Internal Id that uniquely identifies this record.',

PRIMARY KEY (Id)

) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Relay Log Information';

5.

CREATE TABLE slave_worker_info (

Id int(10) unsigned NOT NULL,

Relay_log_name text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,

Relay_log_pos bigint(20) unsigned NOT NULL,

Master_log_name text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,

Master_log_pos bigint(20) unsigned NOT NULL,

Checkpoint_relay_log_name text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,

Checkpoint_relay_log_pos bigint(20) unsigned NOT NULL,

Checkpoint_master_log_name text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,

Checkpoint_master_log_pos bigint(20) unsigned NOT NULL,

Checkpoint_seqno int(10) unsigned NOT NULL,

Checkpoint_group_size int(10) unsigned NOT NULL,

Checkpoint_group_bitmap blob NOT NULL,

PRIMARY KEY (Id)

) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Worker Information';

图12

执行成功。

十一、 注释掉原来配置文件的innodb_force_recovery = 1 内容，然后重启数据库，查看数据库日志是否还有报错。图13.

图13

十二、 重启zabbix_proxy然后，查看zabbix_proxy的使用端口情况。图14

图14

至此启动成功。