18.11 LVS DR模式搭建
准备工作
三台机器, 三台机器均有公网IP。
- 调度器(director)
IP:192.168.230.135 - real server1(real1)
IP:192.168.230.130 - real server2(real2)
IP:192.168.230.145 - VIP:192.168.230.200
开始搭建
配置director
[root@cham002 ~]# vim /usr/local/sbin/lvs_dr.sh
#! /bin/bash
echo 1 > /proc/sys/net/ipv4/ip_forward
ipv=/usr/sbin/ipvsadm
vip=192.168.230.200
rs1=192.168.230.130
rs2=192.168.230.145
#注意这里的网卡名字
ifdown ens33
ifup ens33
ifconfig ens33:2 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip dev ens33:2
$ipv -C
$ipv -A -t $vip:80 -s rr
$ipv -a -t $vip:80 -r $rs1:80 -g -w 1
$ipv -a -t $vip:80 -r $rs2:80 -g -w 1
执行脚本:
[root@cham002 ~]# sh /usr/local/sbin/lvs_dr.sh
成功断开设备 'ens33'。
成功激活的连接(D-Bus 激活路径:/org/freedesktop/NetworkManager/ActiveConnection/6)
[root@cham002 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:b6:9f:e3 brd ff:ff:ff:ff:ff:ff
inet 192.168.230.135/24 brd 192.168.230.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.230.200/32 brd 192.168.230.200 scope global ens33:2
valid_lft forever preferred_lft forever
inet 192.168.230.150/24 brd 192.168.230.255 scope global secondary ens33:0
valid_lft forever preferred_lft forever
inet6 fe80::6f15:52d3:ebeb:e193/64 scope link
valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:b6:9f:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.118.147/24 brd 192.168.118.255 scope global ens37
valid_lft forever preferred_lft forever
inet 192.168.100.1/24 brd 192.168.100.255 scope global ens37
valid_lft forever preferred_lft forever
inet6 fe80::1801:cbbb:ebcc:89a3/64 scope link
valid_lft forever preferred_lft forever
注: VIP绑定到了ens33网卡上。
脚本解释
#! /bin/bash
echo 1 > /proc/sys/net/ipv4/ip_forward
#开启端口转发
ipv=/usr/sbin/ipvsadm
vip=192.168.230.200
rs1=192.168.230.130
rs2=192.168.230.145
#注意这里的网卡名字
ifdown ens33
ifup ens33
#在此重启网卡的目的是避免重复设置命令行提供的IP
ifconfig ens33:2 $vip broadcast $vip netmask 255.255.255.255 up
#绑定VIP到dir的虚拟网卡ens33:2
route add -host $vip dev ens33:2
#添加网关
$ipv -C
$ipv -A -t $vip:80 -s wrr
$ipv -a -t $vip:80 -r $rs1:80 -g -w 1
$ipv -a -t $vip:80 -r $rs2:80 -g -w 1
#设置ipvsadm规则,-g=gateway:使用默认网关(DR模式)
配置real server
分别在real1、real2配置下面的脚本:
##real1(130)
[root@cham1 ~]# vi /usr/local/sbin/lvs_rs.sh
#/bin/bash
vip=192.168.230.200
#把vip绑定在lo上,是为了实现rs直接把结果返回给客户端
ifdown lo
ifup lo
ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip lo:0
#以下操作为更改arp内核参数,目的是为了让rs顺利发送mac地址给客户端
#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
执行脚本:
[root@cham1 ~]# vi /usr/local/sbin/lvs_rs.sh
[root@cham1 ~]# sh !$
sh /usr/local/sbin/lvs_rs.sh
[root@cham1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.230.2 0.0.0.0 UG 100 0 0 ens33
192.168.230.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.230.200 0.0.0.0 255.255.255.255 UH 0 0 0 lo
[root@cham1 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.230.200/32 brd 192.168.230.200 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:90:35:d9 brd ff:ff:ff:ff:ff:ff
inet 192.168.230.130/24 brd 192.168.230.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::7fe3:4489:d9af:a1ed/64 scope link
valid_lft forever preferred_lft forever
inet6 fe80::6f15:52d3:ebeb:e193/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever
##rs2(145)
[root@test ~]# vim /usr/local/sbin/lvs_rs.sh
[root@test ~]# sh !$
sh /usr/local/sbin/lvs_rs.sh
[root@test ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.230.2 0.0.0.0 UG 100 0 0 ens33
192.168.230.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.230.200 0.0.0.0 255.255.255.255 UH 0 0 0 lo
[root@test ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.230.200/32 brd 192.168.230.200 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:50:f9:44 brd ff:ff:ff:ff:ff:ff
inet 192.168.230.145/24 brd 192.168.230.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::9b07:b28d:f5e9:d107/64 scope link
valid_lft forever preferred_lft forever
135
[root@cham002 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.230.200:80 rr
-> 192.168.230.130:80 Route 1 0 1
-> 192.168.230.145:80 Route 1 0 1
[root@cham002 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.230.200:80 rr
-> 192.168.230.130:80 Route 1 2 1
-> 192.168.230.145:80 Route 1 1 1
[root@cham002 ~]#
测试
在浏览器访问VIP:192.168.230.200,刷新网页,访问结果由real1、real2交替回复。
开另一台同网段虚拟机测试!
Type `help' to learn how to use Xshell prompt.
[d:\~]$ ssh 192.168.230.140
Connecting to 192.168.230.140:22...
Connection established.
To escape to local shell, press 'Ctrl+Alt+]'.
Last login: Fri Nov 17 15:14:29 2017 from 192.168.230.1
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 130
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 130
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 130
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 130
18.12 Keepalived LVS
完整的架构需要两台服务器(角色为dir),分别安装Keepalived工具,目的是实现高可用,但Keepalived本身也有负载均衡功能,所以本次使用可以只安装一台Keepalived。Keepalived内置了ipvsadm的功能,所以不需要安装ipvsadm包,也不用编写和执行lvs_dr脚本。
准备工作
三台机器:
- 调度器director:
IP:192.168.230.135;安装Keepalived - real server(real1):
IP:192.168.230.130 - real server(real2):
IP:192.168.230.145 - VIP:192.168.230.200
•两台rs上,依然要执行/usr/local/sbin/lvs_rs.sh脚本
• keepalived有一个比较好的功能,可以在一台rs宕机时,不再把请求转发过去
• 测试
配置director (因为我们之前做keepalived高可用的时候已经安装过了)
[root@cham002 ~]# vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
#备用服务器上为 BACKUP
state MASTER
#绑定vip的网卡为ens33,你的网卡和阿铭的可能不一样,这里需要你改一下
interface ens33
virtual_router_id 51
#备用服务器上为90
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass aminglinux
}
virtual_ipaddress {
192.168.230.200
}
}
virtual_server 192.168.230.200 80 {
#(每隔10秒查询realserver状态)
delay_loop 10
#(lvs 算法)
lb_algo wlc
#(DR模式)
lb_kind DR
#(同一IP的连接60秒内被分配到同一台realserver)
persistence_timeout 0
#(用TCP协议检查realserver状态)
protocol TCP
real_server 192.168.230.130 80 {
#(权重)
weight 100
TCP_CHECK {
#(10秒无响应超时)
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.230.145 80 {
weight 100
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
执行ipvsadm -C 把之前的ipvsadm规则清空掉
[root@cham002 ~]# ipvsadm -C
重启网络可以把之前的vip关掉
[root@cham002 ~]# systemctl restart network
启动Keepalived服务:
[root@cham002 ~]# systemctl start keepalived
[root@cham002 ~]# ps aux |grep keep
root 11296 0.0 0.1 120720 1404 ? Ss 1月24 0:00 /usr/sbin/keepalived -D
root 11297 0.0 0.3 127460 3272 ? S 1月24 0:00 /usr/sbin/keepalived -D
root 11298 0.0 0.3 131656 3032 ? S 1月24 0:05 /usr/sbin/keepalived -D
root 47689 0.0 0.0 112680 976 pts/1 S+ 00:31 0:00 grep --color=auto keep
查看网卡信息:
[root@cham002 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:b6:9f:e3 brd ff:ff:ff:ff:ff:ff
inet 192.168.230.135/24 brd 192.168.230.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.230.200/32 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.230.150/24 brd 192.168.230.255 scope global secondary ens33:0
valid_lft forever preferred_lft forever
inet6 fe80::6f15:52d3:ebeb:e193/64 scope link
valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:b6:9f:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.118.147/24 brd 192.168.118.255 scope global ens37
valid_lft forever preferred_lft forever
inet 192.168.100.1/24 brd 192.168.100.255 scope global ens37
valid_lft forever preferred_lft forever
inet6 fe80::1801:cbbb:ebcc:89a3/64 scope link
[root@cham002 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.230.200:80 wlc
-> 192.168.230.130:80 Route 100 0 0
-> 192.168.230.145:80 Route 100 0 0
添加Keepalived后会自动将宕机的real server清除出rs列表。
如有一方的ngixn 宕掉了,能自动检测,开启后自动加载回来
[root@cham002 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.230.200:80 wlc
-> 192.168.230.145:80 Route 100 0 0
[root@cham002 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.230.200:80 wlc
-> 192.168.230.130:80 Route 100 0 0
-> 192.168.230.145:80 Route 100 0 0
测试
To escape to local shell, press 'Ctrl+Alt+]'.
Last login: Tue Jan 30 23:15:25 2018 from 192.168.230.1
130宕机
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
[root@cham3 ~]# curl 192.168.230.200
cham 145 backup backup.
145宕机
[root@cham3 ~]# curl 192.168.230.200
cham 130
[root@cham3 ~]# curl 192.168.230.200
cham 130
[root@cham3 ~]# curl 192.168.230.200
cham 130
[root@cham3 ~]# curl 192.168.230.200
cham 130
Keepalived+LVS作用
- Keepalived搭建高可用保证LVS中director宕机后服务器不瘫痪
- 如果只使用LVS,那么当LVS架构中某个real server宕机后,director仍然会继续向其发送请求,