VRRP¶
Virtual Router Redundancy Protocol provides a way, for a set of routers, to control a virtual address and MAC address, including automatic fail-over mechanism. Such an address may be used by hosts for some service access, e.g. as static default gateway. The gain from using VRRP is a higher availability of the service without requiring automatic reconfiguration of end hosts.
The VRRP protocol is defined in RFC 3768 (VRRP v2) and RFC 5798 (VRRP v3). The major differences between VRRP v2 and VRRP v3 are the support of IPv6 and the support of low failover timer values which are only available in the latter.
Overview¶
There are two contexts involved in VRRP configuration:
the
vrrp
global context, from which options common to all VRRP interfaces are set, and from which VRRP fail-over groups are defined.the
interface
typevrrp
, from which a particular VRRP instance is configured.
Here is a simple example of VRRP, similar to Sample Configuration 1 described in section 4.1 of RFC 3768:
The configuration of rtr1 is done with the following commands:
vsr running vrf main#
vsr running vrf main# vrrp
vsr running vrrp# enabled true
vsr running vrrp# router-id vrrp_router1
vsr running vrrp# vrrp-startup-delay 30
vsr running vrrp# ..
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51#! link-interface eth1
vsr running vrrp vrrp51#! vrid 5
vsr running vrrp vrrp51# priority 200
vsr running vrrp vrrp51# virtual-address 10.22.0.1/24
vsr running vrrp vrrp51# commit
Note
The link interface eth1 must be up and have an IP address configured.
Let’s fetch the state after committing this configuration:
vsr running vrf main# vrrp
vsr running vrrp# show state
vrrp
enabled true
router-id vrrp_router1
traps-enabled false
..
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# show state
vrrp vrrp51
vmac-xmit-base false
preempt-delay 0
init-state backup
state master
garp-delay 5
link-interface eth1
enabled true
use-vmac true
advertisement-interval 1000
mtu 1500
vrid 5
oper-status UNKNOWN
priority 200
preempt true
version 2
counters
in-errors 0
out-discards 0
out-octets 0
in-octets 0
out-unicast-pkts 9
out-errors 0
in-unicast-pkts 0
in-discards 0
..
ethernet
mac-address 00:00:5e:00:01:05
..
virtual-address 10.22.0.1/24
..
The same configuration can be made using this NETCONF XML configuration:
<config xmlns="urn:6wind:vrouter">
<vrf>
<name>main</name>
<vrrp xmlns="urn:6wind:vrouter/vrrp">
<enabled>true</enabled>
<router-id>vrrp_router1</router-id>
<traps-enabled>false</traps-enabled>
</vrrp>
<interface xmlns="urn:6wind:vrouter/interface">
<vrrp xmlns="urn:6wind:vrouter/vrrp">
<name>vrrp51</name>
<enabled>true</enabled>
<init-state>backup</init-state>
<version>2</version>
<garp-delay>5</garp-delay>
<use-vmac>true</use-vmac>
<vmac-xmit-base>false</vmac-xmit-base>
<priority>200</priority>
<preempt>true</preempt>
<preempt-delay>0</preempt-delay>
<advertisement-interval>1000</advertisement-interval>
<authentication/>
<link-interface>eth1</link-interface>
<vrid>5</vrid>
<virtual-address>
<ip>10.22.0.1/24</ip>
</virtual-address>
</vrrp>
</interface>
</vrf>
</config>
This configuration is obtained by merging the output of the following commands:
vsr running config# show config xml absolute vrf main vrrp
vsr running config# show config xml absolute vrf main interface vrrp vrrp51
The configuration on the second router (backup) is similar, except the
priority which should be lower than 200, and the router-id
which is
set to vrrp_router2
.
See also
The command reference for details.
VRRP states¶
Here are the existing VRRP states and their meanings:
fault
- this state is caused by a fault on a tracked element (link interface, route…). When the state switches tofault
, the router sends a VRRP announcement with a priority of 0.backup
- the router is starting or it is ready to take the virtual address in case of failure of the master.master
- the only state where the router owns the virtual addresses.
VRRP interface settings¶
Version¶
version
defines whether version 2 or 3 is used. Using version 3 allows faster
failovers thanks to low timer values and is recommended. See
information about timers.
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# version 3
Caution
Setting IPv6 VRRP does not imply version 3. Virtual Service Router allows using IPv6 on VRRP version 2 although its support is deprecated.
Link interface¶
link-interface
is the interface the VRRP instance is bound to.
VRRP announcements are send from the IP address of the link interface.
On an IPv6 VRRP instances, setting an IPv6 global address on the link interface is not mandatory because the IPv6 link-local address is sufficient.
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# link-interface eth1
Priority¶
The router with the highest priority will be elected master. The other one will remain backup.
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# priority 150
We recommend to set a priority higher than the default value of 100 on one of the routers. Otherwise, on VRRP version 2, having the same priority on both routers makes the router with the highest IP to be the master. On VRRP version 3, the first router that appears on the network is elected as master.
Note
The priority value of 255 is a special value that can
only be set when the virtual-address IP and the link-interface IP
address are identical. It means that the router on which the priority
255 is set switches to the master
state when the router comes up.
The value of 255 is only supported with the Vmac VRRP mode and when no HA services are configured.
Init state¶
init-state
is a deprecated option that defines on which state the VRRP
instance is initialized before the election occurs, at startup and after a link
goes to up state.
We recommend to let it at the default value of backup
. If you want
the router election to converge quickly, use lower
timers values.
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# init-state backup
Virtual router identifier¶
The virtual router identifier vrid
identifies a set of VRRP routers on a LAN.
Several groups of VRRP routers can coexist on the same broadcast domain as long as
they all have a different vrid
. For a given VRRP instance, the vrid
value
must be identical on both routers.
We recommend to set the same vrid
on all instances within a HA group of routers.
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# vrid 30
Caution
A vMAC is built from the vrid
value. Setting the same vrid
value
on several groups of routers within the same broadcast domain would mean
that the same vMAC is re-used at different places. See
Virtual MAC address.
Preemption¶
“To preempt” in the context of VRRP means to take over as master when the other router is master. The condition for a router to preempt is basically to receive three times a lower priority announcement than its own.
The preemption feature is enabled by default and should remain enabled.
For some special cases, it can be configured using the preempt
option:
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# preempt (true|false)
See also
The timers section gives information about the preemption convergence time.
Advertisement interval and preempt delay¶
advertisement-interval
specifies the rate at which VRRP advertisements are
sent. The configured values in our CLI are in milliseconds.
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# advertisement-interval 1000
The available range of values depends on the VRRP version:
On version 2, it can be configured between 1000 and 50 000 milliseconds. The configured value is rounded to the nearest multiple of 1000. For example, if it is configured with value 1400 that will be rounded to 1000, while if it is 1700 the value will be rounded to 2000.
On version 3, it can be configured between 100 and 40 950 milliseconds. The configured value is rounded to the nearest multiple of 10. For example, if it is configured with value 2554 that will be rounded to 2550, while if it is 2555 the value will be rounded to 2560.
The advertisement interval defines the following timers:
Master_Down_Interval: Time interval for Backup to declare Master down. Calculated as:
(3 * advertisement-interval) + Skew_Time
.Skew_Time: Time to skew Master_Down_Interval in seconds. Calculated as:
on VRRP version 2:
(((256 - local priority) / 256)
.on VRRP version 3:
(((256 - local priority) * advertisement-interval) / 256)
(advertisement-interval
is in seconds).
local priority: Priority of the VRRP instance on the local router. A higher priority leads to a smaller
Skew_Time
.
Note
We strongly recommend to put the same value of advertisement-interval
on both routers.
The master router sends every advertisement-interval
a VRRP announcement
on all instances. If it receives a higher priority advertisement, it will switch to backup
state after Master_Down_Interval.
The backup router sends no advertisement and it listens for advertisements from the other router.
If it stops receiving advertisements, it considers that the master router might not be available anymore. It waits for Master_Down_Interval for confirmation before switching to
master
state.If it receives an advertisement from the master router with a priority of 0, it considers that the master is faulty. Similarly, it waits for Master_Down_Interval before switching to
master
state.If it receives an advertisement from the master router with a non-zero priority less than its own, it considers that it should become the master router unless the
preempt
option is disabled. It waits forMaster_Down_Interval + preempt-delay
before switching tomaster
state. Defaultpreempt-delay
is 0.
Note
preempt-delay
does not apply if no advertisements are received when
the link interface goes up.
Note
A preempt-delay
value of 0 will not allow to preempt immediately, as
Master_Down_Interval always applies.
For example, if you need a failover time of 1 second, you can set:
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# advertisement-interval 300
vsr running vrrp vrrp51# preempt-delay 0
vsr running vrrp vrrp51# priority 170
vsr running vrrp vrrp51# version 3
Note
We recommend to use the same version, priority and timers on all VRRP instances within a router.
Gratuitous ARP delay¶
A gratuitous ARP is a special type of ARP packets that broadcasts without being requested an IP address resolution to all hosts.
After transiting to master
state, the router sends immediately several
gratuitous ARP for each virtual address it owns. Then, it waits for
garp-delay
before sending a second set of gratuitous ARP. This mechanism is
actually useful when the use-vmac option is disabled.
When use-vmac
is enabled (which is the default mode), the virtual address
resolution is always the same.
Note
Since it is a delay, reducing garp-delay
will not speed up the failover.
Its default value of 5 seconds should not be changed.
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# garp-delay 5
Trackers¶
A tracker is used to set a VRRP instance into fault
state when a tracking
condition fails.
Caution
The up state and the presence of an IP address on the
link interface are tracked by the VRRP instance by default.
When configuring a new VRRP instance in a failover group, make sure the link interface matches these
conditions. Otherwise, the HA group would go to fault
state.
IP address tracker¶
A VRRP instance can track IP addresses. When a tracked address is
unreachable, the instance goes to fault
state.
To enable IP tracking:
vsr running config# / tracker
vsr running tracker# icmp my-tracker vrf main address 10.100.0.1
vsr running tracker# / vrf main
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# track my-tracker
See also
The Tracker guide.
Fast path status tracker¶
A VRRP instance can track the fast path status. If the fast path status does not
match the configuration, the instance goes to fault
state. This occurs
for instance when the fast path is starting or stopping, or if the fast path
configuration cannot be applied.
To enable fast path tracking:
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# track-fast-path true
ARP settings¶
Having arp-ignore
set to its default value of any
results in the VRRP
instance and the link interface both responding to ARP requests addressed to
either, meaning that each ARP request receives two conflicting responses. To
avoid this, we recommend explicitly setting arp-ignore
to check-interface
on both the VRRP instance and the link interface.
To configure arp-ignore
:
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# network-stack ipv4
vsr running ipv4# arp-ignore check-interface
vsr running ipv4# / vrf main
vsr running vrf main# interface physical eth1
vsr running physical eth1# network-stack ipv4
vsr running ipv4# arp-ignore check-interface
To prevent the link interface from responding to ARP requests addressed to
itself, set arp-filter
to false
.
To configure arp-filter
:
vsr running vrf main# interface physical eth1
vsr running physical eth1# network-stack ipv4
vsr running ipv4# arp-filter false
Synchronization of instance states¶
This section explains how to configure VRRP on Virtual Service Router to synchronize the instance states within a VRF and to work with the HA features such as HA conntrack and HA IKE .
Failover groups configuration¶
A VRRP group is used to group VRRP interfaces from a given VRF that should share the same VRRP state.
The VRRP instances of a group run VRRP independently but share their state with the group. This way, all group instances are in a consistent state and follow the following rules:
If an instance goes to
fault
state, all the instances of its group will go tofault
state as well.If any backup instance stops receiving announcements from the master router, the group will not go to the
master
state unless all instances stop receiving announcements as well.
Caution
All the instances of a group must have the same priority. If they don’t, it will lead to constant re-elections.
The following example shows how to group two VRRP instances:
vsr running vrf main#
vsr running vrf main# vrrp
vsr running vrrp# group my-group
vsr running group my-group# instance vrrp51
vsr running group my-group# instance vrrp52
vsr running group my-group# commit
Track a group in a different vrf¶
A VRRP group can track a group in another VRF, and apply a penalty to its
own priority when the tracked group goes to fault
state.
This feature is used to ensure that, when two VRRP groups are configured in two different vrfs, both groups will share the same fate.
The priority penalty should be calculated so that (master priority - penalty) <
backup priority
. Its value has to be between 1 and 253, and is set to 253 by
default. We recommend to keep the default value.
Caution
A configuration where one side has a priority of 254 and the other has a priority of 1 is not compatible with this feature.
Here is an example of configuration:
rtr1 running config# vrf main
rtr1 running vrf main# vrrp
rtr1 running vrrp# group my-group
rtr1 running group my-group# instance vrrp51
rtr1 running group my-group# track-group group my-group vrf wan
rtr1 running group my-group# ..
rtr1 running vrrp# ..
rtr1 running vrf main# ..
rtr1 running config# vrf wan
rtr1 running vrf wan# vrrp
rtr1 running vrrp# group my-group
rtr1 running group my-group# instance vrrp52
rtr1 running group my-group# track-group group my-group vrf main
rtr2 running config#
rtr2 running config# vrf main
rtr2 running vrf main# vrrp
rtr2 running vrrp# group my-group
rtr2 running group my-group# instance vrrp51
rtr2 running group my-group# track-group group my-group vrf wan
rtr2 running group my-group# ..
rtr2 running vrrp# ..
rtr2 running vrf main# ..
rtr2 running config# vrf wan
rtr2 running vrf wan# vrrp
rtr2 running vrrp# group my-group
rtr2 running group my-group# instance vrrp52
rtr2 running group my-group# track-group group my-group vrf main
rtr2 running config#
When the vrrp51
instance goes to fault
on rtr1
, the main
vrf VRRP
group my-group
will go fault
as well. The VRRP instance in the main
vrf
on rtr2
will take over.
Thanks to the specified track-group group my-group vrf main
command, the wan
vrf VRRP group my-group
will have its priority of 200 reduced by 253
, to
the minimum value of 1. This value is lower than the priority of the
corresponding group on rtr2
(100). The VRRP instance in the wan
vrf on
rtr2
will take over.
Note
The vrf option accepts a all
keyword. When vrf all
is set, all the groups
with the specified name in all the vrfs (except the current one) will be
tracked. The priority will be reduced by (number of faulty instance) *
(priority-penalty value)
. The minimum reduced priority is 1.
High-availability group notification¶
A VRRP group or VRRP instance can control the state of a high-availability group: when the VRRP state changes, all the subscribers of the high-availability group are notified and can act accordingly.
The following example configures a VRRP instance as a controller for the high-availability group my-ha-group.
See also
High-availability Groups for details.
vsr running config# ha group my-ha-group
vsr running group my-ha-group# / vrf main interface vrrp vrrp51
vsr running vrrp vrrp51#! link-interface eth1 vrid 5
vsr running vrrp vrrp51# notify-ha-group my-ha-group
vsr running vrrp vrrp51# commit
Other services support high-availability can declare themselves as subscribers for this group.
Note
If a HA group notification is configured, the following rules apply:
all the VRRP instances of a VRF must be in a unique VRRP group. Otherwise, the configuration would not be accepted at commit time.
only the master router can forward traffic. It synchronizes protocol information such as neighbor and packet sequence numbers to the backup. The backup router cannot synchronize information to the master. If it receives some data plane traffic by mistake, the traffic will be dropped.
Protection against undesired traffic on the backup node¶
When VRRP controls the state of an HA group (as described in High-availability group notification), it may set the HA group state to backup. The HA services then no longer expect to process traffic.
Undesired traffic may however temporarily be received, forwarded or sent by the device, which may cause adverse effect. For example, if the backup node encrypts traffic and sends IPsec packets to a remote host, the replay window of the remote host will slide forward and the remote peer will drop IPsec traffic sent by the master node due to the anti-replay mechanism, causing a potentially long-lasting traffic loss.
To protect the device from such issues, the VRRP service automatically installs filtering rules in the VRF of the VRRP instance or VRRP group that notifies a switch to the HA backup state. These rules drop undesired traffic potentially received, forwarded or sent by the device.
This feature is enabled by default, but may be disabled per VRF in the VRRP context, via the following command:
vsr running config# / vrf main vrrp
vsr running vrrp# drop-traffic-on-backup false
vsr running vrrp# commit
High-availability split-brain situation¶
A HA split-brain situation is a situation where both VRRP routers are in master
state because a network outage stops them from exchanging VRRP announcements
between them.
Note
If you are not using HA conntrack
or HA IKE
, HA split-brain situation will not affect Virtual Service Router packet
processing. You can skip reading until the Startup delay section. Configuring vrrp-startup-delay
has
some other benefits.
Possible causes¶
The cause of a broken communication between instances that causes a double master HA topology are the following:
An interface may only receive traffic several seconds after the link comes up. The common causes are:
a switch with STP configured in non-edge (or portfast) mode,
a LACP bonding coming up before its slaves interface are in LACP
collecting
state.
Since all instance interfaces are likely to face this issue at the same time at system startup, we recommend to use a vrrp-startup-delay of 30 seconds. To avoid an issue where all the instance interfaces would flap at the same time, it is advised as well to avoid the VRRP instance to rely on a unique logical interface or on several interfaces linked to the same switch. Consider adding a HA link if all the instances relies on one switch.
There is a defect on the network on a single point of failure between all the instances. See topology solutions.
Consequences¶
The consequences of a HA split-brain depend on the services the Virtual Service Router is running:
If the nodes are using HA conntrack or HA IKE , the sequence numbers of packets are synchronized from the master to the backup or fault router. The data plane traffic is only accepted and forwarded on the master router. If the routers are all in
master
state, they will both forward a part of the traffic and increment their sequence numbers independently. They will not synchronize their sequence numbers one with another because a master router cannot accept synchronized information by design. As soon as the unlegitimate master router returns to thebackup
state, the master router will receive its traffic. However the master’s sequence number table is outdated, so this traffic will be dropped.Without HA conntrack and HA IKE , having several master routers in the HA topology does not affect their packet processing because the routers forward packets without checking any sequence number. The split-brain situation is the result of a fault on the network that might lead to a traffic outage but that is not caused by the routers.
Solutions¶
Change the topology¶
We recommend to use different physical links and switches for the instances of a group to avoid this the split-brain situation.
Here is a network topology with two routers running Virtual Service Router. Each router has a HA group containing instances on link interfaces bond0.2 and bond0.3 that are respectively vlan 2 and 3 sub-interfaces of bond0.
The link between switch1 and switch2 is now broken. The communication is totally broken between routers. As a result, both routers are master.
A solution to avoid the problem is to dispatch the VRRP instances on several switching domains to ensure that VRRP announcements for different VRRP instances take a different path. It requires adding some switches.
If adding some switches is not possible, consider adding a HA link.
Use a HA link¶
The HA link is a dedicated link between two routers that is used to exchange VRRP
announcements on an additional path. In case of failure on the network, when the
main instances are prevented from exchanging their VRRP announcements, VRRP
announcements are still exchanged on this link. The backup router knows that the
other router is still up and does not switch to master
state. Without the HA
link, this condition would have lead to have two master routers.
The following diagram shows a HA link implemented on eth0. When communication is broken between bond0 interfaces of rtr1 and rtr2, VRRP announcements are still exchanged on the HA link between rtr1 eth0 and rtr2 eth0.
To check the availability of the other router on the HA link, a VRRP instance must be bound to the link and added to the same VRRP group as the other instances.
Like any other VRRP instance, a HA link VRRP instance has an associated virtual address. It must not be used for receiving data plane traffic.
Here are some recommendations to make the HA link reliable enough:
To be reliable enough, the HA link must not use a same path as the main interfaces.
If Virtual Service Router is running on a bare-metal server, it should be a dedicated physical link between the servers.
If Virtual Service Router is running on a virtual machine, the link should take a different physical path.
the HA link must be bound to a physical interface, which excludes for example a bonding or a virtual interface.
the HA interface must not be a fast path port.
the HA link should not go through a switch if Virtual Service Router is running on a bare-metal server.
Also, the HA link VRRP instance:
must use IPv4, even if the main instances are all of IPv6 type.
must use the same VRRP version as the other instances.
We recommend to use the HA link to synchronize stateful
information between the routers. local-address
and remote-address
would refer to the HA link ip addresses. Refer to HA neighbor,
HA conntrack and HA IKE
for details.
The track-link-interface
option must be disabled. It prevents the VRRP
HA link instance (and the instances of its group) from switching to
fault
state in case the HA link goes down.
vsr running vrf main# interface vrrp eth0
vsr running vrrp eth0# track-link-interface false
Configure a startup delay¶
vrrp-startup-delay
is a delay that is applied at system startup before
starting to listen for VRRP advertisement. This way, no VRRP election
happens before the router is ready.
We recommend to set a 30s vrrp-startup-delay
value.
vsr running vrf main#
vsr running vrf main# vrrp
vsr running vrrp# vrrp-startup-delay 30
Note
vrrp-startup-delay
only applies at system startup but not after a link
state is flapping.
Note
Before the vrrp-startup-delay
timer expires, the initial state set
by the init-state
leaf in the VRRP interface configuration applies. Make
sure to let it at the default backup
value.
VRRP settings for virtual environments¶
Using Virtual Service Router on VMs with VRRP may require to adjust the following settings:
Note
We recommend to keep default value of these settings if Virtual Service Router is running on bare-metal servers.
Virtual MAC address¶
By default, a vMAC is associated to the virtual IP address in the host ARP and NDP neighbor table. It is also used as a source MAC address for VRRP announcements.
Some virtual network switches (eg. VMware vSwitch) are unable to deal
with vMAC because they only know MAC addresses that the hypervisor has
assigned to the VMs. In this case, use-vmac
must be set to
false so that the MAC address of the link interface is used instead.
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# use-vmac false
Caution
When disabling use-vmac
, make sure the ARP or NDP protocols are
correctly propagated. Otherwise, some hosts
might not be informed of the new master MAC address after
a failover. We recommend to test some manual failovers.
The storm control feature, if it is enabled on switches, may affect the flooding of ARP broadcasts.
The reception of gratuitous ARP may be disabled on hosts for security reasons.
Note
Since IPv4 and IPv6 vMAC formats are different, it is not possible to define IPv4 and IPv6 virtual adresses on the same VRRP instance.
Note
vmac-xmit-based
setting is only effective if use-vmac
is true.
Its default value is false. If true, the MAC address of the link
interface is used for sending VRRP announcements. It should be left
configured to its default value.
Unicast peering¶
By default, the VRRP announcements are sent on multicast.
Some virtual network switches (eg. VMware vSwitch) are unable to deal
with multicast MAC addresses. In this case, a unicast-peer
must be configured
to send VRRP using unicast packets instead of multicast packets. The destination IP address
(192.168.222.2 in the following example) is the actual address of the same
instance on the other router.
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# unicast-peering 192.168.222.2
Note
When using values for advertisement-interval
lower than 1s and
unicast peering, we recommend to use static ARP / NDP entries to
avoid latency induced by ARP / NDP requests.