风影子
作者风影子2013-09-08 10:20
数据库管理员, 深圳

【原创】LVS+keepalived实现TOMCAT集群

字数 19741阅读 11676评论 0赞 2
    因一下级企业项目需要用到TOMCAT集群,提供两台linux服务器,问了一圈人都没人懂。领导就叫我研究下,实在不行就叫下级企业自己出钱到外边请人装。经过两天的研究,终于是弄成功了,顺便记录一下。
1       规划
1.1     服务器环境规划

我这里把负载服务器及WEB服务器放在了一起

负载服务器masterWEB服务器1真实IP        192.168.10.16

负载服务器backupWEB服务器2真实IP        192.168.10.17

负载服务器虚拟IP                                                192.168.10.19

1.2     软件环境规划

--整体环境

操作系统:REDHAT LINUX 5.4 X86-64,内核版本:2.6.18-164.el5

需要安装GCC编译器及openssl等包,操作系统安装时都选上,关闭防火墙或配相关策略

 

--192.168.10.16192.168.10.17

安装ipvsadm-1.24-6(目前主流的操作系统都不需要升级内核了)

http://www.linuxvirtualserver.org/software/ipvs.html#kernel-2.6

安装keepalived-1.2.7

http://www.keepalived.org/download.html

 

--192.168.10.16192.168.10.17

安装JDK1.6

http://www.oracle.com/technetwork/java/javase/downloads/jdk6u35-downloads-1836443.html

 

安装TOMCAT6.0

http://tomcat.apache.org/

解压路径:/opt/tomcat60

2       负载服务器配置

假设下载的安装软件都放在/soft

2.1     安装ipvsadm-1.24-6

[root@x3250-1 ~]# cd /soft

[root@x3250-1 soft]# rpm -ivh ipvsadm-1.24-6.src.rpm  

[root@x3250-1 soft]# cd /usr/src/redhat/SOURCES

[root@x3250-1 SOURCES]# tar -zxvf ipvsadm-1.24.tar.gz

[root@x3250-1 ipvsadm-1.24]# uname -r

2.6.18-164.el5

[root@x3250-1 ipvsadm-1.24]# ln -s  /usr/src/kernels/2.6.18-164.el5-x86_64/  /usr/src/linux

[root@x3250-1 ipvsadm-1.24]# make;make install

2.2     安装keepalived-1.2.7

[root@x3250-1 ipvsadm-1.24]# cd /soft

[root@x3250-1 soft]# tar -zxvf keepalived-1.2.7.tar.gz

[root@x3250-1 soft]# cd keepalived-1.2.7

[root@x3250-1 keepalived-1.2.7]# ./configure --prefix=/usr/local/keepalived

[root@x3250-1 keepalived-1.2.7]# make;make install

2.3     配置开机启动keepalived服务

方便管理keepalived启动服务,添加到系统开机启动服务中

[root@x3250-1]#cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

[root@x3250-1]#cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

[root@x3250-1]#mkdir /etc/keepalived

[root@x3250-1]#cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

[root@x3250-1]#cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

[root@x3250-1]#chkconfig --add keepalived

[root@x3250-1]#chkconfig keepalived on

[root@x3250-1]#chkconfig --list keepalived

keepalived      0:off   1:off   2:on    3:on    4:on    5:on    6:off

2.4     配置keepalived.conf文件

开始配置keepalived.conf文件,两台LVS_serverkeepalived.conf配置文件基本都是一样,只需要更改红色标注的地方。

[root@x3250-1]#mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

[root@x3250-1]#vi /etc/keepalived//keepalived.conf  

 

--配置完后要重启服务

[root@x3250-1 keepalived-1.2.7]# service keepalived restart

配置内容如下

! Configuration File for keepalived

 

global_defs {

   notification_email {             #设置报警接收邮件地址,可以有多个邮件

     acassen@firewall.loc

     failover@firewall.loc

     sysadmin@firewall.loc

   }

   notification_email_from Alexandre.Cassen@firewall.loc     #设置邮件的发送地址

   smtp_server 127.0.0.1                     #设置smtp_server服务器的地址

   smtp_connect_timeout 30                 #设置连接smtp服务器超时时间

   router_id LVS_DEVEL      #标识keepalived服务的ID号,两边lvs_server服务都一致

}

 

vrrp_instance VI_1 {

    state MASTER                  #备份LVS_server服务器上将MASTER改为BACKUP

    interface eth0                                        #网卡设备,提供虚拟IP服务的网卡

    virtual_router_id 51

    priority 100                          #备份LVS_server服务器上必须改为小于100才行,如99

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.10.19                                #虚拟IP地址

    }

}

 

virtual_server 192.168.10.19 8080 {

    delay_loop 6                       #(每隔6秒查询real_server状态)

    lb_algo wrr                        #(负载均衡调度 算法,常用wlcrr)

    lb_kind DR                         #(负载均衡转发规则,一般包括DR,NAT,TUN)

    protocol TCP                       #(TCP协议检查real_server状态)

 

    real_server 192.168.10.16 8080 {

        weight 1                      #(权重)

        TCP_CHECK {                   #(通过tcpcheck判断real_Server的健康状态)

            connect_timeout 3          #(real_Server服务10秒无响应超时)

            nb_get_retry 3              #(重连次数3)

            delay_before_retry 3         #(重练间隔时间3)

            connect_port 8080           #(健康检查端口)

        }

    }

    real_server 192.168.10.17 8080 {

        weight 1                      #(权重)

        TCP_CHECK {                   #(通过tcpcheck判断real_Server的健康状态)

            connect_timeout 3          #(real_Server服务10秒无响应超时)

            nb_get_retry 3              #(重连次数3)

            delay_before_retry 3        #(重练间隔时间3)

            connect_port 8080          #(健康检查端口)

        }

    }

}

3       WEB服务器配置 3.1     安装JDK.6

Jdk安装文件放在/soft

[root@x3250-1 ~]cd /soft

[root@x3250-1 ~]#rpm -qa|grep gcj                                                                                                                                     

java-1.4.2-gcj-compat-1.4.2.0-40jpp.115

libgcj-4.1.2-46.el5

libgcj-4.1.2-46.el5

 

--下面这里先卸载系统自带的jdk 1.4

[root@x3250-1 ~]# rpm -e java-1.4.2-gcj-compat-1.4.2.0-40jpp.115                                              

error: Failed dependencies:

        java-gcj-compat >= 1.0.64 is needed by (installed) gjdoc-0.7.7-12.el5.x86_64

        java-gcj-compat is needed by (installed) antlr-2.7.6-4jpp.2.x86_64

[root@x3250-1 ~]# rpm -e --nodeps java-1.4.2-gcj-compat-1.4.2.0-40jpp.115

[root@x3250-1 ~]# cd /soft

[root@x3250-1 soft]# chmod +x jdk-6u45-linux-x64-rpm.bin

[root@x3250-1 soft]# ./jdk-6u45-linux-x64-rpm.bin

[root@x3250-1 soft]# java -version

java version "1.6.0_45"

Java(TM) SE Runtime Environment (build 1.6.0_45-b06)

Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

 

--添加以下环境变量

[root@x3250-1 soft]# vi /etc/profile

 

export JAVA_HOME=/usr/java/jdk1.6.0_45

export JAVA_BIN=/usr/java/jdk1.6.0_45/bin

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export JAVA_HOME JAVA_BIN PATH CLASSPATH

 

3.2     安装配置TOMCAT 3.2.1     安装及配置

直接解压放到任意目录/opt/tomcat60

a)       TOMCAT端口配置

其中server.xml位于Tomcatconf 目录下;例图中为8080端口。

b)      中间层数据库配置

配置文件目录:TomcatwebappsROOTWEB-INFclasses

配置文件:applicationContext.xml

c)       中间层系统升级

请更换TomcatwebappsROOTWEB-INFclassescom目录

3.2.2     启动tomcat并设置开机自动运行

#cd /opt/tomcat60/bin

#./startup.sh

#cat “/opt/tomcat60/bin/startup.sh” >> /etc/rc.local

3.3     配置LVS脚本 3.3.1     配置脚本

两台web服务器安装http服务(tomcat)后,创建lvs.sh文件赋予权限,配置lvs脚本,脚本作用是抑制arp广播,将请求包都由负载均衡lvs服务分配。

 

[root@x3250-1 keepalived-1.2.7]# vi /sbin/realdr.sh

#!/bin/bash

VIP=192.168.10.19

/etc/rc.d/init.d/functions

case "$1" in

start)

        echo "start LVS of REALServer"

        /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up

#       /sbin/route add -host $VIP dev lo:0

        echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

        echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

        echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

        echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

;;

stop)

        echo "close LVS REALserver"

        /sbin/ifconfig lo:0 down

#       /sbin/route del -host $VIP dev lo:0

        echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

        echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

        echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

        echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

;;

*)

echo "Usage: $0 {start|stop}"

exit 1

esac

 

3.3.2     设置开机运行LVS脚本

[root@x3250-1 keepalived-1.2.7]# chmod 755 /sbin/realdr.sh

[root@x3250-1 keepalived-1.2.7]# /sbin/realdr.sh start

[root@x3250-1 keepalived-1.2.7]#cat “/sbin/realdr.sh start” > /etc/rc.local

4       测试

/opt/tomcat60/webapps/ROOT下新建ip.html文件,192.168.10.16中该文件内容为192.168.10.16192.168.10.17中该文件内容为192.168.10.17,用于通过虚拟IP访问的时候能从页面看出连接的是哪一台服务器

4.1     负载均衡

如果多次打开浏览器,通过虚拟IP访问网站,应当会将负载均衡到两台服务器上

第一次打开一个浏览器中输入http://192.168.10.19:8080/ip.html,显示192.168.10.16(或17)

第二次打开浏览器(新开浏览器窗口),输入http://192.168.10.19:8080/ip.html,显示192.168.10.17(或16)

4.2     故障转移

停止192.168.10.16TOMCAT服务,这时通过虚拟IP就能访问到网站,且是访问的192.168.17服务器

[root@x3250-1 bin]# /opt/tomcat60/bin/shutdown.sh

Using CATALINA_BASE:   /opt/tomcat60

Using CATALINA_HOME:   /opt/tomcat60

Using CATALINA_TMPDIR: /opt/tomcat60/temp

Using JRE_HOME:       /usr/java/jdk1.6.0_45

第一次打开一个浏览器中输入http://192.168.10.19:8080/ip.html,显示192.168.10.17

第二次打开浏览器(新开浏览器窗口),输入http://192.168.10.19:8080/ip.html,显示192.168.10.17

可以看到网站依然可以访问,且都是访问的192.168.10.17服务器,此时我们再将192.168.10.16服务器的tomcat服务启动,应又能进行负载均衡

[root@x3250-1 bin]# /opt/tomcat60/bin/startup.sh

Using CATALINA_BASE:   /opt/tomcat60

Using CATALINA_HOME:   /opt/tomcat60

Using CATALINA_TMPDIR: /opt/tomcat60/temp

Using JRE_HOME:       /usr/java/jdk1.6.0_45

5       其它操作 5.1     查看WEB服务器虚拟IP

查看方法:ip add show,因为我们这里是WEB服务器和LVS服务器是同一台机器,所以本处loeth0上都有虚拟IP地址,WEB服务器上是看lo这里

[root@x3250-1 bin]# ip add show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet 192.168.10.19/32 brd 192.168.10.19 scope global lo:0

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 36:40:b5:88:68:f2 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::3440:b5ff:fe88:68f2/64 scope link

       valid_lft forever preferred_lft forever

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 100

    link/ether 34:40:b5:88:68:ef brd ff:ff:ff:ff:ff:ff

    inet 192.168.10.16/24 brd 192.168.10.255 scope global eth0

    inet 192.168.10.19/32 scope global eth0

    inet6 fe80::3640:b5ff:fe88:68ef/64 scope link

       valid_lft forever preferred_lft forever

4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000

    link/ether 34:40:b5:88:68:f0 brd ff:ff:ff:ff:ff:ff

5: sit0: <NOARP> mtu 1480 qdisc noop

link/sit 0.0.0.0 brd 0.0.0.0

5.2     两台lvs_server服务器都启动keepalived服务

已经设置为开机以服务的形势自动启动,仅当虚拟IP无法访问时尝试重启

[root@x3250-1 bin]# service keepalived restart

5.3     查看主机lvs_server服务器上的虚拟ip

查看主机lvs_server服务器上的虚拟ip是否有绑定到eth0接口上,因为我们这里是WEB服务器和LVS服务器是同一台机器,所以本处loeth0上都有虚拟IP地址,LVS服务器上是看eth0这里

[root@x3250-1 bin]# ip add show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet 192.168.10.19/32 brd 192.168.10.19 scope global lo:0

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 36:40:b5:88:68:f2 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::3440:b5ff:fe88:68f2/64 scope link

       valid_lft forever preferred_lft forever

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 100

    link/ether 34:40:b5:88:68:ef brd ff:ff:ff:ff:ff:ff

    inet 192.168.10.16/24 brd 192.168.10.255 scope global eth0

    inet 192.168.10.19/32 scope global eth0

    inet6 fe80::3640:b5ff:fe88:68ef/64 scope link

       valid_lft forever preferred_lft forever

4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000

    link/ether 34:40:b5:88:68:f0 brd ff:ff:ff:ff:ff:ff

5: sit0: <NOARP> mtu 1480 qdisc noop

link/sit 0.0.0.0 brd 0.0.0.0

 

5.4     查看负载均衡状态

LVS服务器上查看

 [root@x3250-1 bin]# ipvsadm

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  192.168.10.19:webcache wrr

  -> 192.168.10.16:webcache       Local   1      0          0        

  -> 192.168.10.17:webcache       Route   1      0          0   

5.5     TOMCAT启动与停止

TOMCAT服务

[root@x3250-1 bin]#/opt/tomcat60/bin/shutdown.sh

         启动TOMCAT服务

[root@x3250-1 bin]#/opt/tomcat60/bin/startup.sh

5.6     查看lvs_server服务器启动keepalived的日志信息

[root@x3250-1 bin]#tail -f /var/log/messages

如果觉得我的文章对您有用,请点赞。您的支持将鼓励我继续创作!

2

添加新评论0 条评论

Ctrl+Enter 发表

关于TWT  使用指南  社区专家合作  厂商入驻社区  企业招聘  投诉建议  版权与免责声明  联系我们
© 2019  talkwithtrend — talk with trend,talk with technologist 京ICP备09031017号-30