VRRP

Virtual Router Redundancy Protocol provides a way, for a set of routers, to control a virtual address and MAC address, including automatic fail-over mechanism. Such an address may be used by hosts for some service access, e.g. as static default gateway. The gain from using VRRP is a higher availability of the service without requiring automatic reconfiguration of end hosts.

The VRRP protocol is defined in RFC 3768 (VRRP v2) and RFC 5798 (VRRP v3). The major differences between VRRP v2 and VRRP v3 are the support of IPv6 and the support of low failover timer values which are only available in the latter.

Overview

There are two contexts involved in VRRP configuration:

  • the vrrp global context, from which options common to all VRRP interfaces are set, and from which VRRP fail-over groups are defined.

  • the interface type vrrp, from which a particular VRRP instance is configured.

Here is a simple example of VRRP, similar to Sample Configuration 1 described in section 4.1 of RFC 3768:

../../../_images/aafig-8328353f10823f476c5a8eb161e2613b41892c66.svg

The configuration of rtr1 is done with the following commands:

vsr running vrf main#
vsr running vrf main# vrrp
vsr running vrrp# enabled true
vsr running vrrp# router-id vrrp_router1
vsr running vrrp# vrrp-startup-delay 30
vsr running vrrp# ..
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51#! link-interface eth1
vsr running vrrp vrrp51#! vrid 5
vsr running vrrp vrrp51# priority 200
vsr running vrrp vrrp51# virtual-address 10.22.0.1/24
vsr running vrrp vrrp51# commit

Note

The link interface eth1 must be up and have an IP address configured.

Let’s fetch the state after committing this configuration:

vsr running vrf main# vrrp
vsr running vrrp# show state
vrrp
    enabled true
    router-id vrrp_router1
    traps-enabled false
    ..
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# show state
vrrp vrrp51
    vmac-xmit-base false
    preempt-delay 0
    init-state backup
    state master
    garp-delay 5
    link-interface eth1
    enabled true
    use-vmac true
    advertisement-interval 1000
    mtu 1500
    vrid 5
    oper-status UNKNOWN
    priority 200
    preempt true
    version 2
    counters
        in-errors 0
        out-discards 0
        out-octets 0
        in-octets 0
        out-unicast-pkts 9
        out-errors 0
        in-unicast-pkts 0
        in-discards 0
        ..
    ethernet
        mac-address 00:00:5e:00:01:05
        ..
    virtual-address 10.22.0.1/24
    ..

The same configuration can be made using this NETCONF XML configuration:

<config xmlns="urn:6wind:vrouter">
  <vrf>
    <name>main</name>
    <vrrp xmlns="urn:6wind:vrouter/vrrp">
      <enabled>true</enabled>
      <router-id>vrrp_router1</router-id>
      <traps-enabled>false</traps-enabled>
    </vrrp>
    <interface xmlns="urn:6wind:vrouter/interface">
      <vrrp xmlns="urn:6wind:vrouter/vrrp">
        <name>vrrp51</name>
        <enabled>true</enabled>
        <init-state>backup</init-state>
        <version>2</version>
        <garp-delay>5</garp-delay>
        <use-vmac>true</use-vmac>
        <vmac-xmit-base>false</vmac-xmit-base>
        <priority>200</priority>
        <preempt>true</preempt>
        <preempt-delay>0</preempt-delay>
        <advertisement-interval>1000</advertisement-interval>
        <authentication/>
        <link-interface>eth1</link-interface>
        <vrid>5</vrid>
        <virtual-address>
          <ip>10.22.0.1/24</ip>
        </virtual-address>
      </vrrp>
    </interface>
  </vrf>
</config>

This configuration is obtained by merging the output of the following commands:

vsr running config# show config xml absolute vrf main vrrp
vsr running config# show config xml absolute vrf main interface vrrp vrrp51

The configuration on the second router (backup) is similar, except the priority which should be lower than 200, and the router-id which is set to vrrp_router2.

See also

The command reference for details.

VRRP states

Here are the existing VRRP states and their meanings:

  • fault - this state is caused by a fault on a tracked element (link interface, route…). When the state switches to fault, the router sends a VRRP announcement with a priority of 0.

  • backup - the router is starting or it is ready to take the virtual address in case of failure of the master.

  • master - the only state where the router owns the virtual addresses.

VRRP interface settings

Version

version defines whether version 2 or 3 is used. Using version 3 allows faster failovers thanks to low timer values and is recommended. See information about timers.

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# version 3

Caution

Setting IPv6 VRRP does not imply version 3. Virtual Service Router allows using IPv6 on VRRP version 2 although its support is deprecated.

Priority

The router with the highest priority will be elected master. The other one will remain backup.

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# priority 150

We recommend to set a priority higher than the default value of 100 on one of the routers. Otherwise, on VRRP version 2, having the same priority on both routers makes the router with the highest IP to be the master. On VRRP version 3, the first router that appears on the network is elected as master.

Note

The priority value of 255 is a special value that can only be set when the virtual-address IP and the link-interface IP address are identical. It means that the router on which the priority 255 is set switches to the master state when the router comes up.

The value of 255 is only supported with the Vmac VRRP mode and when no HA services are configured.

Init state

init-state is a deprecated option that defines on which state the VRRP instance is initialized before the election occurs, at startup and after a link goes to up state.

We recommend to let it at the default value of backup. If you want the router election to converge quickly, use lower timers values.

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# init-state backup

Virtual router identifier

The virtual router identifier vrid identifies a set of VRRP routers on a LAN. Several groups of VRRP routers can coexist on the same broadcast domain as long as they all have a different vrid. For a given VRRP instance, the vrid value must be identical on both routers. We recommend to set the same vrid on all instances within a HA group of routers.

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# vrid 30

Caution

A vMAC is built from the vrid value. Setting the same vrid value on several groups of routers within the same broadcast domain would mean that the same vMAC is re-used at different places. See Virtual MAC address.

Preemption

“To preempt” in the context of VRRP means to take over as master when the other router is master. The condition for a router to preempt is basically to receive three times a lower priority announcement than its own.

The preemption feature is enabled by default and should remain enabled. For some special cases, it can be configured using the preempt option:

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# preempt (true|false)

See also

The timers section gives information about the preemption convergence time.

Gratuitous ARP delay

A gratuitous ARP is a special type of ARP packets that broadcasts without being requested an IP address resolution to all hosts.

After transiting to master state, the router sends immediately several gratuitous ARP for each virtual address it owns. Then, it waits for garp-delay before sending a second set of gratuitous ARP. This mechanism is actually useful when the use-vmac option is disabled. When use-vmac is enabled (which is the default mode), the virtual address resolution is always the same.

Note

Since it is a delay, reducing garp-delay will not speed up the failover. Its default value of 5 seconds should not be changed.

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# garp-delay 5

Trackers

A tracker is used to set a VRRP instance into fault state when a tracking condition fails.

Caution

The up state and the presence of an IP address on the link interface are tracked by the VRRP instance by default. When configuring a new VRRP instance in a failover group, make sure the link interface matches these conditions. Otherwise, the HA group would go to fault state.

IP address tracker

A VRRP instance can track IP addresses. When a tracked address is unreachable, the instance goes to fault state.

To enable IP tracking:

vsr running config# / tracker
vsr running tracker# icmp my-tracker vrf main address 10.100.0.1
vsr running tracker# / vrf main
vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# track my-tracker

See also

The Tracker guide.

Fast path status tracker

A VRRP instance can track the fast path status. If the fast path status does not match the configuration, the instance goes to fault state. This occurs for instance when the fast path is starting or stopping, or if the fast path configuration cannot be applied.

To enable fast path tracking:

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# track-fast-path true

ARP settings

Having arp-ignore set to its default value of any results in the VRRP instance and the link interface both responding to ARP requests addressed to either, meaning that each ARP request receives two conflicting responses. To avoid this, we recommend explicitly setting arp-ignore to check-interface on both the VRRP instance and the link interface.

To configure arp-ignore:

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# network-stack ipv4
vsr running ipv4# arp-ignore check-interface
vsr running ipv4# / vrf main
vsr running vrf main# interface physical eth1
vsr running physical eth1# network-stack ipv4
vsr running ipv4# arp-ignore check-interface

To prevent the link interface from responding to ARP requests addressed to itself, set arp-filter to false.

To configure arp-filter:

vsr running vrf main# interface physical eth1
vsr running physical eth1# network-stack ipv4
vsr running ipv4# arp-filter false

Synchronization of instance states

This section explains how to configure VRRP on Virtual Service Router to synchronize the instance states within a VRF and to work with the HA features such as HA conntrack and HA IKE .

Failover groups configuration

A VRRP group is used to group VRRP interfaces from a given VRF that should share the same VRRP state.

The VRRP instances of a group run VRRP independently but share their state with the group. This way, all group instances are in a consistent state and follow the following rules:

  • If an instance goes to fault state, all the instances of its group will go to fault state as well.

  • If any backup instance stops receiving announcements from the master router, the group will not go to the master state unless all instances stop receiving announcements as well.

Caution

All the instances of a group must have the same priority. If they don’t, it will lead to constant re-elections.

The following example shows how to group two VRRP instances:

vsr running vrf main#
vsr running vrf main# vrrp
vsr running vrrp# group my-group
vsr running group my-group# instance vrrp51
vsr running group my-group# instance vrrp52
vsr running group my-group# commit

Track a group in a different vrf

A VRRP group can track a group in another VRF, and apply a penalty to its own priority when the tracked group goes to fault state.

This feature is used to ensure that, when two VRRP groups are configured in two different vrfs, both groups will share the same fate.

The priority penalty should be calculated so that (master priority - penalty) < backup priority. Its value has to be between 1 and 253, and is set to 253 by default. We recommend to keep the default value.

Caution

A configuration where one side has a priority of 254 and the other has a priority of 1 is not compatible with this feature.

Here is an example of configuration:

../../../_images/aafig-a23a2f30bc45663e5d898de5a480b20501744c5d.svg 
rtr1 running config# vrf main
rtr1 running vrf main# vrrp
rtr1 running vrrp# group my-group
rtr1 running group my-group# instance vrrp51
rtr1 running group my-group# track-group group my-group vrf wan
rtr1 running group my-group# ..
rtr1 running vrrp# ..
rtr1 running vrf main# ..
rtr1 running config# vrf wan
rtr1 running vrf wan# vrrp
rtr1 running vrrp# group my-group
rtr1 running group my-group# instance vrrp52
rtr1 running group my-group# track-group group my-group vrf main

rtr2 running config#
rtr2 running config# vrf main
rtr2 running vrf main# vrrp
rtr2 running vrrp# group my-group
rtr2 running group my-group# instance vrrp51
rtr2 running group my-group# track-group group my-group vrf wan
rtr2 running group my-group# ..
rtr2 running vrrp# ..
rtr2 running vrf main# ..
rtr2 running config# vrf wan
rtr2 running vrf wan# vrrp
rtr2 running vrrp# group my-group
rtr2 running group my-group# instance vrrp52
rtr2 running group my-group# track-group group my-group vrf main
rtr2 running config#

When the vrrp51 instance goes to fault on rtr1, the main vrf VRRP group my-group will go fault as well. The VRRP instance in the main vrf on rtr2 will take over.

Thanks to the specified track-group group my-group vrf main command, the wan vrf VRRP group my-group will have its priority of 200 reduced by 253, to the minimum value of 1. This value is lower than the priority of the corresponding group on rtr2 (100). The VRRP instance in the wan vrf on rtr2 will take over.

Warning

This mechanism relies on the preemption feature which is enabled by default. In the case described above, disabling preemption will prevent the VRRP instance in the wan VRF on rtr2 from taking over.

Note

The vrf option accepts a all keyword. When vrf all is set, all the groups with the specified name in all the vrfs (except the current one) will be tracked. The priority will be reduced by (number of faulty instance) * (priority-penalty value). The minimum reduced priority is 1.

High-availability group notification

A VRRP group or VRRP instance can control the state of a high-availability group: when the VRRP state changes, all the subscribers of the high-availability group are notified and can act accordingly.

The following example configures a VRRP instance as a controller for the high-availability group my-ha-group.

See also

High-availability Groups for details.

vsr running config# ha group my-ha-group
vsr running group my-ha-group# / vrf main interface vrrp vrrp51
vsr running vrrp vrrp51#! link-interface eth1 vrid 5
vsr running vrrp vrrp51# notify-ha-group my-ha-group
vsr running vrrp vrrp51# commit

Other services support high-availability can declare themselves as subscribers for this group.

Note

If a HA group notification is configured, the following rules apply:

  • all the VRRP instances of a VRF must be in a unique VRRP group. Otherwise, the configuration would not be accepted at commit time.

  • only the master router can forward traffic. It synchronizes protocol information such as neighbor and packet sequence numbers to the backup. The backup router cannot synchronize information to the master. If it receives some data plane traffic by mistake, the traffic will be dropped.

Protection against undesired traffic on the backup node

When VRRP controls the state of an HA group (as described in High-availability group notification), it may set the HA group state to backup. The HA services then no longer expect to process traffic.

Undesired traffic may however temporarily be received, forwarded or sent by the device, which may cause adverse effect. For example, if the backup node encrypts traffic and sends IPsec packets to a remote host, the replay window of the remote host will slide forward and the remote peer will drop IPsec traffic sent by the master node due to the anti-replay mechanism, causing a potentially long-lasting traffic loss.

To protect the device from such issues, the VRRP service automatically installs filtering rules in the VRF of the VRRP instance or VRRP group that notifies a switch to the HA backup state. These rules drop undesired traffic potentially received, forwarded or sent by the device.

This feature is enabled by default, but may be disabled per VRF in the VRRP context, via the following command:

vsr running config# / vrf main vrrp
vsr running vrrp# drop-traffic-on-backup false
vsr running vrrp# commit

High-availability split-brain situation

A HA split-brain situation is a situation where both VRRP routers are in master state because a network outage stops them from exchanging VRRP announcements between them.

Note

If you are not using HA conntrack or HA IKE , HA split-brain situation will not affect Virtual Service Router packet processing. You can skip reading until the Startup delay section. Configuring vrrp-startup-delay has some other benefits.

Possible causes

The cause of a broken communication between instances that causes a double master HA topology are the following:

  • An interface may only receive traffic several seconds after the link comes up. The common causes are:

    • a switch with STP configured in non-edge (or portfast) mode,

    • a LACP bonding coming up before its slaves interface are in LACP collecting state.

    Since all instance interfaces are likely to face this issue at the same time at system startup, we recommend to use a vrrp-startup-delay of 30 seconds. To avoid an issue where all the instance interfaces would flap at the same time, it is advised as well to avoid the VRRP instance to rely on a unique logical interface or on several interfaces linked to the same switch. Consider adding a HA link if all the instances relies on one switch.

  • There is a defect on the network on a single point of failure between all the instances. See topology solutions.

Consequences

The consequences of a HA split-brain depend on the services the Virtual Service Router is running:

  • If the nodes are using HA conntrack or HA IKE , the sequence numbers of packets are synchronized from the master to the backup or fault router. The data plane traffic is only accepted and forwarded on the master router. If the routers are all in master state, they will both forward a part of the traffic and increment their sequence numbers independently. They will not synchronize their sequence numbers one with another because a master router cannot accept synchronized information by design. As soon as the unlegitimate master router returns to the backup state, the master router will receive its traffic. However the master’s sequence number table is outdated, so this traffic will be dropped.

  • Without HA conntrack and HA IKE , having several master routers in the HA topology does not affect their packet processing because the routers forward packets without checking any sequence number. The split-brain situation is the result of a fault on the network that might lead to a traffic outage but that is not caused by the routers.

Solutions

Change the topology

We recommend to use different physical links and switches for the instances of a group to avoid this the split-brain situation.

Here is a network topology with two routers running Virtual Service Router. Each router has a HA group containing instances on link interfaces bond0.2 and bond0.3 that are respectively vlan 2 and 3 sub-interfaces of bond0.

../../../_images/aafig-7ab14904c4ec3fc5785346ed48dfa3596aa48f4a.svg

The link between switch1 and switch2 is now broken. The communication is totally broken between routers. As a result, both routers are master.

../../../_images/aafig-8e5c1cc61105c1beda7519dc0b7a4ff42db8e3b0.svg

A solution to avoid the problem is to dispatch the VRRP instances on several switching domains to ensure that VRRP announcements for different VRRP instances take a different path. It requires adding some switches.

../../../_images/aafig-df618cf930bafdbb45ce503f40e8893f22557ebf.svg

If adding some switches is not possible, consider adding a HA link.

Configure a startup delay

vrrp-startup-delay is a delay that is applied at system startup before starting to listen for VRRP advertisement. This way, no VRRP election happens before the router is ready.

We recommend to set a 30s vrrp-startup-delay value.

vsr running vrf main#
vsr running vrf main# vrrp
vsr running vrrp# vrrp-startup-delay 30

Note

vrrp-startup-delay only applies at system startup but not after a link state is flapping.

Note

Before the vrrp-startup-delay timer expires, the initial state set by the init-state leaf in the VRRP interface configuration applies. Make sure to let it at the default backup value.

VRRP settings for virtual environments

Using Virtual Service Router on VMs with VRRP may require to adjust the following settings:

Note

We recommend to keep default value of these settings if Virtual Service Router is running on bare-metal servers.

Virtual MAC address

By default, a vMAC is associated to the virtual IP address in the host ARP and NDP neighbor table. It is also used as a source MAC address for VRRP announcements.

Some virtual network switches (eg. VMware vSwitch) are unable to deal with vMAC because they only know MAC addresses that the hypervisor has assigned to the VMs. In this case, use-vmac must be set to false so that the MAC address of the link interface is used instead.

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# use-vmac false

Caution

When disabling use-vmac, make sure the ARP or NDP protocols are correctly propagated. Otherwise, some hosts might not be informed of the new master MAC address after a failover. We recommend to test some manual failovers.

  • The storm control feature, if it is enabled on switches, may affect the flooding of ARP broadcasts.

  • The reception of gratuitous ARP may be disabled on hosts for security reasons.

Note

Since IPv4 and IPv6 vMAC formats are different, it is not possible to define IPv4 and IPv6 virtual adresses on the same VRRP instance.

Note

vmac-xmit-based setting is only effective if use-vmac is true. Its default value is false. If true, the MAC address of the link interface is used for sending VRRP announcements. It should be left configured to its default value.

Unicast peering

By default, the VRRP announcements are sent on multicast.

Some virtual network switches (eg. VMware vSwitch) are unable to deal with multicast MAC addresses. In this case, a unicast-peer must be configured to send VRRP using unicast packets instead of multicast packets. The destination IP address (192.168.222.2 in the following example) is the actual address of the same instance on the other router.

vsr running vrf main# interface vrrp vrrp51
vsr running vrrp vrrp51# unicast-peering 192.168.222.2

Note

When using values for advertisement-interval lower than 1s and unicast peering, we recommend to use static ARP / NDP entries to avoid latency induced by ARP / NDP requests.