VRRP

Virtual Router Redundancy Protocol provides a way, for a set of routers, to control a virtual address and MAC address, including automatic fail-over mechanism. Such an address may be used by hosts for some service access, e.g. as static default gateway. The gain from using VRRP is a higher availability of the service without requiring automatic reconfiguration of end hosts.

The VRRP protocol is defined in RFC 3768 (VRRP v2) and RFC 5798 (VRRP v3). The major differences between VRRP v2 and VRRP v3 are the support of IPv6 and the support of low failover timer values which are only available in the latter.

Overview

There are two contexts involved in VRRP configuration:

  • the vrrp global context, from which options common to all VRRP interfaces are set, and from which VRRP fail-over groups are defined.

  • the interface type vrrp, from which a particular VRRP instance is configured.

Here is a simple example of VRRP, similar to Sample Configuration 1 described in section 4.1 of RFC 3768:

../../../_images/aafig-8328353f10823f476c5a8eb161e2613b41892c66.svg

The configuration of rtr1 is done with the following commands:

vrouter running vrf main#
vrouter running vrf main# vrrp
vrouter running vrrp# enabled true
vrouter running vrrp# router-id vrrp_router1
vrouter running vrrp# vrrp-startup-delay 30
vrouter running vrrp# ..
vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51#! link-interface eth1
vrouter running vrrp vrrp51#! vrid 5
vrouter running vrrp vrrp51# priority 200
vrouter running vrrp vrrp51# virtual-address 10.22.0.1/24
vrouter running vrrp vrrp51# commit

Note

The link interface eth1 must be up and have an IP address configured.

Let’s fetch the state afer committing this configuration:

vrouter running vrf main# vrrp
vrouter running vrrp# show state
vrrp
    enabled true
    router-id vrrp_router1
    traps-enabled false
    ..
vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# show state
vrrp vrrp51
    vmac-xmit-base false
    preempt-delay 0
    init-state backup
    state master
    garp-delay 5
    link-interface eth1
    enabled true
    use-vmac true
    advertisement-interval 1000
    mtu 1500
    vrid 5
    oper-status UNKNOWN
    priority 200
    preempt true
    version 2
    counters
        in-errors 0
        out-discards 0
        out-octets 0
        in-octets 0
        out-unicast-pkts 9
        out-errors 0
        in-unicast-pkts 0
        in-discards 0
        ..
    ethernet
        mac-address 00:00:5e:00:01:05
        ..
    virtual-address 10.22.0.1/24
    ..

The same configuration can be made using this NETCONF XML configuration:

<config xmlns="urn:6wind:vrouter">
  <vrf>
    <name>main</name>
    <vrrp xmlns="urn:6wind:vrouter/vrrp">
      <enabled>true</enabled>
      <router-id>vrrp_router1</router-id>
      <traps-enabled>false</traps-enabled>
    </vrrp>
    <interface xmlns="urn:6wind:vrouter/interface">
      <vrrp xmlns="urn:6wind:vrouter/vrrp">
        <name>vrrp51</name>
        <enabled>true</enabled>
        <init-state>backup</init-state>
        <version>2</version>
        <garp-delay>5</garp-delay>
        <use-vmac>true</use-vmac>
        <vmac-xmit-base>false</vmac-xmit-base>
        <priority>200</priority>
        <preempt>true</preempt>
        <preempt-delay>0</preempt-delay>
        <advertisement-interval>1000</advertisement-interval>
        <authentication/>
        <link-interface>eth1</link-interface>
        <vrid>5</vrid>
        <virtual-address>
          <ip>10.22.0.1/24</ip>
        </virtual-address>
      </vrrp>
    </interface>
  </vrf>
</config>

This configuration is obtained by merging the output of the following commands:

vrouter running config# show config xml absolute vrf main vrrp
vrouter running config# show config xml absolute vrf main interface vrrp vrrp51

The configuration on the second router (backup) is similar, except the priority which should be lower than 200, and the router-id which is set to vrrp_router2.

See also

The command reference for details.

VRRP states

Here are the existing VRRP states and their meanings:

  • fault - this state is caused by a fault on a tracked element (link interface, route…). When the state switches to fault, the router sends a VRRP announcement with a priority of 0.

  • backup - the router is starting or it is ready to take the virtual address in case of failure of the master.

  • master - the only state where the router owns the virtual addresses.

VRRP interface settings

Version

version defines whether version 2 or 3 is used. Using version 3 allows faster failovers thanks to low timer values and is recommended. See information about timers.

vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# version 3

Caution

Setting IPv6 VRRP does not imply version 3. Turbo Router allows using IPv6 on VRRP version 2 although its support is deprecated.

Priority

The router with the highest priority will be elected master. The other one will remain backup.

vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# priority 150

We recommend to set a priority higher than the default value of 100 on one of the routers. Otherwise, on VRRP version 2, having the same priority on both routers makes the router with the highest IP to be the master. On VRRP version 3, the first router that appears on the network is elected as master.

Init state

init-state is a deprecated option that defines on which state the VRRP instance is initialized before the election occurs, at startup and after a link goes to up state.

We recommend to let it at the default value of backup. If you want the router election to converge quickly, use lower timers values.

vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# init-state backup

Virtual router identifier

The virtual router identifier vrid identifies a set of VRRP routers on a LAN. Several groups of VRRP routers can coexist on the same broadcast domain as long as they all have a different vrid. For a given VRRP instance, the vrid value must be identical on both routers. We recommend to set the same vrid on all instances within a HA group of routers.

vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# vrid 30

Caution

A vMAC is built from the vrid value. Setting the same vrid value on several groups of routers within the same broadcast domain would mean that the same vMAC is re-used at different places. See Virtual MAC address.

Preemption

“To preempt” in the context of VRRP means to take over as master when the other router is master. The condition for a router to preempt is basically to receive three times a lower priority announcement than its own.

The preemption feature is enabled by default and should remain enabled. For some special cases, it can be configured using the preempt option:

vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# preempt (true|false)

See also

The timers section gives information about the preemption convergence time.

Gratuitous ARP delay

A gratuitous ARP is a special type of ARP packets that broadcasts without being requested an IP address resolution to all hosts.

After transiting to master state, the router sends immediately several gratuitous ARP for each virtual address it owns. Then, it waits for garp-delay before sending a second set of gratuitous ARP. This mechanism is actually useful when the use-vmac option is disabled. When use-vmac is enabled (which is the default mode), the virtual address resolution is always the same.

Note

Since it is a delay, reducing garp-delay will not speed up the failover. Its default value of 5 seconds should not be changed.

vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# garp-delay 5

Trackers

A tracker is used to set a VRRP instance into fault state when a tracking condition fails.

Caution

The up state and the presence of an IP address on the link interface are tracked by the VRRP instance by default. When configuring a new VRRP instance in a failover group, make sure the link interface matches these conditions. Otherwise, the HA group would go to fault state.

IP address tracker

A VRRP instance can track IP addresses. When a tracked address is unreachable, the instance goes to fault state.

To enable IP tracking:

vrouter running config# / tracker
vrouter running tracker# icmp my-tracker vrf main address 10.100.0.1
vrouter running tracker# / vrf main
vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# track my-tracker

See also

The Tracker guide.

Fast path status tracker

A VRRP instance can track the fast path status. If the fast path status does not match the configuration, the instance goes to fault state. This occurs for instance when the fast path is starting or stopping, or if the fast path configuration cannot be applied.

To enable fast path tracking:

vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# track-fast-path true

Synchronization of instance states

This section explains how to configure VRRP on Turbo Router to synchronize the instance states within a VRF and to work with the HA features such as HA conntrack and HA IKE .

Failover groups configuration

A VRRP group is used to group VRRP interfaces from a given VRF that should share the same VRRP state.

The VRRP instances of a group run VRRP independently but share their state with the group. This way, all group instances are in a consistent state and follow the following rules:

  • If an instance goes to fault state, all the instances of its group will go to fault state as well.

  • If any backup instance stops receiving announcements from the master router, the group will not go to the master state unless all instances stop receiving announcements as well.

Caution

All the instances of a group must have the same priority. If they don’t, it will lead to constant re-elections.

The following example shows how to group two VRRP instances:

vrouter running vrf main#
vrouter running vrf main# vrrp
vrouter running vrrp# group my-group
vrouter running group my-group# instance vrrp51
vrouter running group my-group# instance vrrp52
vrouter running group my-group# commit

High-availability group notification

A VRRP group or VRRP instance can control the state of a high-availability group: when the VRRP state changes, all the subscribers of the high-availability group are notified and can act accordingly.

The following example configures a VRRP instance as a controller for the high-availability group my-ha-group.

See also

High-availability Groups for details.

vrouter running config# ha group my-ha-group
vrouter running group my-ha-group# / vrf main interface vrrp vrrp51
vrouter running vrrp vrrp51#! link-interface eth1 vrid 5
vrouter running vrrp vrrp51# notify-ha-group my-ha-group
vrouter running vrrp vrrp51# commit

Other services support high-availability can declare themselves as subscribers for this group.

Note

If a HA group notification is configured, the following rules apply:

  • all the VRRP instances of a VRF must be in a unique VRRP group. Otherwise, the configuration would not be accepted at commit time.

  • only the master router can forward traffic. It synchronizes protocol information such as neighbor and packet sequence numbers to the backup. The backup router cannot synchronize information to the master. If it receives some data plane traffic by mistake, the traffic will be dropped.

High-availability split-brain situation

A HA split-brain situation is a situation where both VRRP routers are in master state because a network outage stops them from exchanging VRRP announcements between them.

Note

If you are not using HA conntrack or HA IKE , HA split-brain situation will not affect Turbo Router packet processing. You can skip reading until the Startup delay section. Configuring vrrp-startup-delay has some other benefits.

Possible causes

The cause of a broken communication between instances that causes a double master HA topology are the following:

  • An interface may only receive traffic several seconds after the link comes up. The common causes are:

    • a switch with STP configured in non-edge (or portfast) mode,

    • a LACP bonding coming up before its slaves interface are in LACP collecting state.

    Since all instance interfaces are likely to face this issue at the same time at system startup, we recommend to use a vrrp-startup-delay of 30 seconds. To avoid an issue where all the instance interfaces would flap at the same time, it is advised as well to avoid the VRRP instance to rely on a unique logical interface or on several interfaces linked to the same switch. Consider adding a HA link if all the instances relies on one switch.

  • There is a defect on the network on a single point of failure between all the instances. See topology solutions.

Consequences

The consequences of a HA split-brain depend on the services the Turbo Router is running:

  • If the nodes are using HA conntrack or HA IKE , the sequence numbers of packets are synchronized from the master to the backup or fault router. The data plane traffic is only accepted and forwarded on the master router. If the routers are all in master state, they will both forward a part of the traffic and increment their sequence numbers independently. They will not synchronize their sequence numbers one with another because a master router cannot accept synchronized information by design. As soon as the unlegitimate master router returns to the backup state, the master router will receive its traffic. However the master’s sequence number table is outdated, so this traffic will be dropped.

  • Without HA conntrack and HA IKE , having several master routers in the HA topology does not affect their packet processing because the routers forward packets without checking any sequence number. The split-brain situation is the result of a fault on the network that might lead to a traffic outage but that is not caused by the routers.

Solutions

Change the topology

We recommend to use different physical links and switches for the instances of a group to avoid this the split-brain situation.

Here is a network topology with two routers running Turbo Router. Each router has a HA group containing instances on link interfaces bond0.2 and bond0.3 that are respectively vlan 2 and 3 sub-interfaces of bond0.

../../../_images/aafig-7ab14904c4ec3fc5785346ed48dfa3596aa48f4a.svg

The link between switch1 and switch2 is now broken. The communication is totally broken between routers. As a result, both routers are master.

../../../_images/aafig-8e5c1cc61105c1beda7519dc0b7a4ff42db8e3b0.svg

A solution to avoid the problem is to dispatch the VRRP instances on several switching domains to ensure that VRRP announcements for different VRRP instances take a different path. It requires adding some switches.

../../../_images/aafig-df618cf930bafdbb45ce503f40e8893f22557ebf.svg

If adding some switches is not possible, consider adding a HA link.

Configure a startup delay

vrrp-startup-delay is a delay that is applied at system startup before starting to listen for VRRP advertisement. This way, no VRRP election happens before the router is ready.

We recommend to set a 30s vrrp-startup-delay value.

vrouter running vrf main#
vrouter running vrf main# vrrp
vrouter running vrrp# vrrp-startup-delay 30

Note

vrrp-startup-delay only applies at system startup but not after a link state is flapping.

Note

Before the vrrp-startup-delay timer expires, the initial state set by the init-state leaf in the VRRP interface configuration applies. Make sure to let it at the default backup value.

VRRP settings for virtual environments

Using Turbo Router on VMs with VRRP may require to adjust the following settings:

Note

We recommend to keep default value of these settings if Turbo Router is running on bare-metal servers.

Virtual MAC address

By default, a vMAC is associated to the virtual IP address in the host ARP and NDP neighbor table. It is also used as a source MAC address for VRRP announcements.

Some virtual network switches (eg. VMware vSwitch) are unable to deal with vMAC because they only know MAC addresses that the hypervisor has assigned to the VMs. In this case, use-vmac must be set to false so that the MAC address of the link interface is used instead.

vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# use-vmac false

Caution

When disabling use-vmac, make sure the ARP or NDP protocols are correctly propagated. Otherwise, some hosts might not be informed of the new master MAC address after a failover. We recommend to test some manual failovers.

  • The storm control feature, if it is enabled on switches, may affect the flooding of ARP broadcasts.

  • The reception of gratuitous ARP may be disabled on hosts for security reasons.

Note

Since IPv4 and IPv6 vMAC formats are different, it is not possible to define IPv4 and IPv6 virtual adresses on the same VRRP instance.

Note

vmac-xmit-based setting is only effective if use-vmac is true. Its default value is false. If true, the MAC address of the link interface is used for sending VRRP announcements. It should be left configured to its default value.

Unicast peering

By default, the VRRP announcements are sent on multicast.

Some virtual network switches (eg. VMware vSwitch) are unable to deal with multicast MAC addresses. In this case, a unicast-peer must be configured to send VRRP using unicast packets instead of multicast packets. The destination IP address (192.168.222.2 in the following example) is the actual address of the same instance on the other router.

vrouter running vrf main# interface vrrp vrrp51
vrouter running vrrp vrrp51# unicast-peering 192.168.222.2

Note

When using values for advertisement-interval lower than 1s and unicast peering, we recommend to use static ARP / NDP entries to avoid latency induced by ARP / NDP requests.