计算节点高可用

HotDB Server支持高可用架构部署，利用keepalived高可用服务原理搭建主备服务关系，可保证在主计算节点(即Active计算节点) 服务故障后，自动切换到备计算节点 (即Standby计算节点)，应用层面可借助Keepalived的VIP 访问数据库服务，保证服务不间断。

1.启动说明
在启动高可用架构下的主备计算节点服务时，需要注意启动的顺序问题，如下为标准启动顺序：

1. 先启动主计算节点所在服务器上的Keepalived，再启动主计算节点：
   查看计算节点日志：
   2018-06-13 09:40:04.408 [INFO] [INIT] [Labor-3] j(-1) – HotDB-Managerlistening on 3325
   2018-06-13 09:40:04.412 [INFO] [INIT] [Labor-3] j(-1) – HotDB-Serverlistening on 3323

查看端口监听状态：

root> ss -npl| grep 3323
LISTEN     0      1000       *:3323         *:*      users:(("java",12639,87))
root> ps -aux|grep hotdb
Warning: badsyntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root     12639 60.7  34.0 4194112  2032134 ? Sl   Jun04  7043:58 /usr/java/jdk1.7.0_80/bin/java-DHOTDB_HOME=/usr/local/hotdb-2.4/hotdb-server -classpath/usr/local/hotdb-2.4/hotdb-server/conf: …省略更多… -Xdebug -Xrunjdwp:transport=dt_socket,address=8065,server=y,suspend=n-Djava.net.preferIPv4Stack=true cn.hotpu.hotdb.HotdbStartup

使用命令“ip a”可查看当前主计算节点的Keepalived VIP是已绑定成功，如下例子中，192.168.200.190为主计算节点所在服务器地址；192.168.200.140为配置的VIP地址

root> ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth1:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWNqlen 1000
    link/ether 00:1d:0f:14:8b:fa brdff:ff:ff:ff:ff:ff
inet192.168.200.190/24 brd192.168.200.255 scope global eth1
inet192.168.200.140/24 scopeglobal secondary eth1:1
    inet6 fe80::21d:ff:fe14:8bfa/64 scope link
       valid_lft forever preferred_lft forever

• 再启动备计算节点所在服务器上的Keepalived，再启动备计算节点：
  查看计算节点日志：
  2018-06-04 18:14:32:321 [INFO] [INIT] [main] j(-1) – Using nio networkhandler
  2018-06-04 18:14:32:356 [INFO] [INIT] [main] j(-1) – HotDB-Managerlistening on 3325
  2018-06-04 18:14:32:356 [INFO] [AUTHORITY] [checker] Z(-1) – Thanks forchoosing HotDB

查看端口监听状态：

root> ss -npl| grep 3325
LISTEN     0      1000                      *:3325                     *:*      users:(("java",11603,83))
root> ps -aux|grep hotdb
Warning: badsyntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root     11603 12.0  13.6 3788976  1086196 ? Sl   Jun04  1389:44 /usr/java/jdk1.7.0_80/bin/java-DHOTDB_HOME=/usr/local/hotdb-2.4/hotdb-server -classpath/usr/local/hotdb-2.4/hotdb-server/conf: …省略更多… -Xdebug-Xrunjdwp:transport=dt_socket,address=8065,server=y,suspend=n-Djava.net.preferIPv4Stack=true cn.hotpu.hotdb.HotdbStartup

高可用切换说明

当主计算节点（以192.168.200.190为例）服务上故障时，检测脚本(vrrp_scripts)检测到主计算节点服务端口不可访问或hacheck连续失败超过3次，优先级会进行调整，变成 90(weight -10)。
备计算节点（以192.168.200.191为例）服务上的 keepalived 收到比自己优先级低的 vrrp 包(192.168.200.191上优先级为 95)后，将切换到 master 状态，抢占 vip(以192.168.200.140为例)。同时在进入 master 状态后，执行 notify_master 脚本，访问192.168.200.191上的计算节点管理端口执行 online 命令启动并初始化192.168.200.191上的计算节点服务端口。若该计算节点启动成功，则主备切换成功继续提供服务。192.168.200.191上的计算节点日志如下：

2018-06-12 21:54:45.128 [INFO] [INIT] [Labor-3] j(-1) – HotDB-Serverlistening on 3323
2018-06-12 21:54:45.128 [INFO] [INIT] [Labor-3] j(-1) –=============================================
2018-06-12 21:54:45.141 [INFO] [MANAGER] [Labor-4] q(-1) – Failed tooffline master Because mysql: [Warning] Using a password on the command lineinterface can be insecure.
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.200.190'(111)
2018-06-12 21:54:45.141 [INFO] [RESPONSE] [$NIOREACTOR-8-RW] af(-1) –connection killed for HotDB backup startup
…省略更多…

Keepalived的VIP已在192.168.200.191服务器上:

root> ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
    link/ether 18:a9:05:1b:0f:a8 brdff:ff:ff:ff:ff:ff
inet192.168.200.191/24 brd192.168.200.255 scope global eth0
inet192.168.200.140/24 scopeglobal secondary eth0:1
    inet6 fe80::1aa9:5ff:fe1b:fa8/64 scope link
       valid_lft forever preferred_lft forever

注：若通过管理平台手动切换，切换成功的会修改server.xml中的（haState、haNodeHost）的配置，将主备的信息互换，故障切换不会修改配置。