Usage¶

Initialization¶

Load and initalize the VRF module:
```
# vrf.sh start
```
Note

Note that at Linux - Fast Path Synchronization start, the sysctl configuration of file linux-fp-sync.sh is applied on the main namespace. The sysctl net.core.devconf_inherit_init_net controls if a new network namespace should inherit all current settings under /proc/sys/net/{ipv4,ipv6} /conf/{all,default}/. This option exists in linux upstream since version 5.1. For kernels without this option, the default behavior applies. Please, refer to the kernel documentation: https://docs.kernel.org/admin-guide/sysctl/net.html#devconf-inherit-init-net

vrfctl¶

Basic functions¶

Description

Create, delete, list and monitor VRF instances.

Synopsis

vrfctl add < auto-vrfid | VRFID > [ vfrname NAME ] [ SCRIPT ]
vrfctl del < VRFID | vfrname NAME > [ SCRIPT ]
vrfctl list [ VRFID | vrfname NAME ]
vrfctl monitor

Parameters

auto-vrfid | VRFID: ID of the VRF instance to create or remove. If ‘auto-vrfid’ is specified (only for the command ‘add’), the highest existing vrfid plus one is used.
NAME: Optional. Name of the VRF instance.
SCRIPT: Optional. Custom script to execute when the VRF instance is created or deleted.

Note

By default, the script linux-fp-sync-vrf.sh must be used to manage Linux - Fast Path Synchronization. You can adapt it to your needs.

Advanced functions¶

vrfctl bind¶

Description

Bind an existing network namespace to a new VRF instance.

Note

Instances are deleted via ‘vrfctl del’, as for VRF instances created via ‘vrfctl add’.

Synopsis

vrfctl bind < file FILE | ipnetns IPNETNSNAME | pid PID > < auto-vrfid | VRFID > [ vfrname NAME ] [ SCRIPT ]

Parameters

FILE: A path to a network namespace file.
IPNETNSNAME: An iproute2 netns name.
PID: A PID.
auto-vrfid | VRFID: ID of the VRF instance to create. If ‘auto-vrfid’ is specified, the highest existing vrfid plus one is used.
NAME: Optional. Name of the VRF instance.
SCRIPT: Optional. Custom script to execute when the VRF instance is created or deleted.

Note

By default, the script linux-fp-sync-vrf.sh must be used to manage Linux - Fast Path Synchronization. You can adapt it to your needs.

ip netns exec¶

Description

Execute a command in a VRF instance.

Synopsis

ip netns exec <vrf name> <cmd>

Parameters

vrf name: Name of the VRF instance in which the command must be executed.
cmd: Command to be executed.

Configuration example¶

We will configure a router with four interfaces:

eth9_0 and eth9_1 are in the vrf0 VRF instance, and
eth2_0 and eth2_1 are in the vrf1 VRF instance.

For Cross-VRF routing, we reserve a special network namespace called xvrf: it hosts a bridge which is used to enable communication between VRF instances. Special values for the veth endpoint MAC address in vrfX are used, embedding the VRF instances’ identifiers: this is required by the fast path to help fast Cross-VRF routing. We also need to add private IP addresses on the veth endpoint in vrfX.

Create the vrf1 VRF instance:
```
# vrfctl add 1 linux-fp-sync-vrf.sh
```
Note

linux-fp-sync-vrf.sh is used to initialize Linux - Fast Path Synchronization.

Create a network namespace which will host a bridge to perform Cross-VRF routing:

# ip netns add xvrf
# ip netns exec xvrf brctl addbr xvrf-bridge
# ip netns exec xvrf ip link set xvrf-bridge up

Create a tunnel (veth) from vrf0 to xvrf:

# ip link add xvrf0 type veth peer name from-vrf0 netns xvrf

The last 2 bytes of the MAC address MUST match the VRF instance number:

# ip link set xvrf0 address 00:09:C0:00:00:00 up
# ip netns exec xvrf ip link set from-vrf0 up

We create a private address for Cross-VRF routing:

# ip address add 169.254.128.0/16 dev xvrf0

If VNB is loaded, we must destroy the ng_ether node:

# ngctl shutdown from-vrf0:

Link the veth tunnel to the xvrf bridge:

# ip netns exec xvrf brctl addif xvrf-bridge from-vrf0

Create a tunnel (veth) from vrf1 to xvrf:

# ip netns exec vrf1 ip link add xvrf1 type veth peer name from-vrf1 netns xvrf
# ip netns exec vrf1 ip link set xvrf1 address 00:09:C0:00:00:01 up
# ip netns exec xvrf ip link set from-vrf1 up
# ip netns exec vrf1 ip address add 169.254.128.1/16 dev xvrf1

If VNB is loaded, we must destroy the ng_ether node (VNB is netns agnostic, no need to execute ip netns exec).

# ngctl shutdown from-vrf1:

Link the veth tunnel to the xvrf bridge:

# ip netns exec xvrf brctl addif xvrf-bridge from-vrf1

Check the bridge in netns xvrf:

# ip netns exec xvrf brctl show
# bridge name     bridge id               STP enabled     interfaces
# xvrf-bridge     8000.8e4459c37102       no              from-vrf0
                                                          from-vrf1

Configure interfaces:

Set eth2_0 and eth2_1 in the vrf1 VRF instance:

# ip link set eth2_0 netns vrf1
# ip link set eth2_1 netns vrf1

Activate interfaces:

# ip netns exec vrf1 ip link set eth2_0 up
# ip netns exec vrf1 ip link set eth2_1 up
# ip link set eth9_0 up
# ip link set eth9_1 up

Add addresses:

# ip netns exec vrf1 ip address add 2.0.0.1/24 dev eth2_0
# ip netns exec vrf1 ip address add 2.1.0.1/24 dev eth2_1
# ip address add 9.0.0.1/24 dev eth9_0
# ip address add 9.1.0.1/24 dev eth9_1

Add a bidirectional route between 100.2.2.1 and 110.2.2.1 (eth2_0 <-> eth2_1):

# netns exec vrf1 ip route add 100.2.2.1/32 via 2.0.0.5
# netns exec vrf1 ip route add 110.2.2.1/32 via 2.1.0.5

Add a bidirectional route between 100.9.9.1 and 110.9.9.1 (eth9_0 <-> eth9_1):

# route add 100.9.9.1/32 via 9.0.0.5
# route add 110.9.9.1/32 via 9.1.0.5

Add a route from 100.2.9.1 to 110.2.9.1 (eth2_0 -> xvrf1 -> eth9_1):

# netns exec vrf1 ip route add 110.2.9.1/32 via 169.254.128.0
# route add 110.2.9.1/32 via 9.1.0.5

Add a route from 110.2.9.1 to 100.2.9.1 (eth2_1 -> xvrf1 -> eth9_0):

# netns exec vrf1 ip route add 100.2.9.1/32 via 169.254.128.0
# route add 100.2.9.1/32 via 9.0.0.5

Add a route from 100.9.2.1 to 110.9.2.1 (eth9_0 -> xvrf0 -> eth2_1):

# route add 110.9.2.1/32 via 169.254.128.1
# netns exec vrf1 ip route add 110.9.2.1/32 via 2.1.0.5

Add a route from 110.9.2.1 to 100.9.2.1 (eth9_1 -> xvrf0 -> eth2_0):

# route add 100.9.2.1/32 via 169.254.128.1
# netns exec vrf1 ip route add 100.9.2.1/32 via 2.0.0.5

Check the route in the vrf0 VRF instance:

# ip route
0.0.0/24 dev eth9_0  proto kernel  scope link  src 9.0.0.1
1.0.0/24 dev eth9_1  proto kernel  scope link  src 9.1.0.1
2.9.1 via 9.0.0.5 dev eth9_0
9.2.1 via 169.254.128.1 dev xvrf0
9.9.1 via 9.0.0.5 dev eth9_0
2.9.1 via 9.1.0.5 dev eth9_1
9.2.1 via 169.254.128.1 dev xvrf0
9.9.1 via 9.1.0.5 dev eth9_1
254.0.0/16 dev xvrf0  proto kernel  scope link  src 169.254.128.0

Check route in the vrf1 VRF instance:

# ip netns exec vrf1 ip route
0.0.0/24 dev eth2_0  proto kernel  scope link  src 2.0.0.1
1.0.0/24 dev eth2_1  proto kernel  scope link  src 2.1.0.1
2.2.1 via 2.0.0.5 dev eth2_0
2.9.1 via 169.254.128.0 dev xvrf1
9.2.1 via 2.0.0.5 dev eth2_0
2.2.1 via 2.1.0.5 dev eth2_1
2.9.1 via 169.254.128.0 dev xvrf1
9.2.1 via 2.1.0.5 dev eth2_1
254.0.0/16 dev xvrf1  proto kernel  scope link  src 169.254.128.1

Check the fast path configuration:

<fp-0> iface
37:eth2_1 [VR-1] ifuid=0x25d8c282 (port 1) <UP|RUNNING|FWD4|FWD6> (0x63)
        type=ether mac=00:02:02:00:00:21 mtu=1500 tcp4mss=0 tcp6mss=0
        IPv4 routes=3  IPv6 routes=1
59:lo [VR-1] ifuid=0x3b1c7058 (virtual) <UP|RUNNING|FWD4|FWD6> (0x63)
        type=loop mac=00:00:00:00:00:00 mtu=16436 tcp4mss=0 tcp6mss=0
        IPv4 routes=0  IPv6 routes=0
100:fpn0 [VR-0] ifuid=0x64247322 (virtual) <UP|RUNNING|FWD4|FWD6> (0x63)
        type=ether mac=00:00:46:50:4e:00 mtu=1500 tcp4mss=0 tcp6mss=0
        IPv4 routes=0  IPv6 routes=0
117:lo [VR-0] ifuid=0x754c6fa8 (virtual) <UP|RUNNING|FWD4|FWD6> (0x63)
        type=loop mac=00:00:00:00:00:00 mtu=16436 tcp4mss=0 tcp6mss=0
        IPv4 routes=0  IPv6 routes=0
141:eth9_0 [VR-0] ifuid=0x8d001382 (port 2) <UP|RUNNING|FWD4|FWD6> (0x63)
        type=ether mac=00:02:02:00:00:90 mtu=1500 tcp4mss=0 tcp6mss=0
        IPv4 routes=3  IPv6 routes=1
440:xvrf1 [VR-1] ifuid=0xb8d9e470 (virtual) <UP|RUNNING|FWD4|FWD6|LOCAL_OUT> (0x8063)
        type=xvrf mac=00:09:c0:00:00:01 mtu=1500 tcp4mss=0 tcp6mss=0
        IPv4 routes=3  IPv6 routes=0
504:eth2_0 [VR-1] ifuid=0xf8e170d2 (port 0) <UP|RUNNING|FWD4|FWD6> (0x63)
        type=ether mac=00:02:02:00:00:20 mtu=1500 tcp4mss=0 tcp6mss=0
        IPv4 routes=3  IPv6 routes=1
710:xvrf0 [VR-0] ifuid=0xc6129210 (virtual) <UP|RUNNING|FWD4|FWD6|LOCAL_OUT> (0x8063)
        type=xvrf mac=00:09:c0:00:00:00 mtu=1500 tcp4mss=0 tcp6mss=0
        IPv4 routes=3  IPv6 routes=0
953:eth9_1 [VR-0] ifuid=0xb9f76532 (port 3) <UP|RUNNING|FWD4|FWD6> (0x63)
        type=ether mac=00:02:02:00:00:91 mtu=1500 tcp4mss=0 tcp6mss=0
        IPv4 routes=3  IPv6 routes=1

<fp-0> route4
# - Preferred, * - Active, > - selected
2.9.1/32  [27]  ROUTE gw 9.0.0.5 via eth9_0(0x8d001382) (39)
9.2.1/32  [30]  ROUTE gw 169.254.128.1 via xvrf0(0xc6129210) (42)
9.9.1/32  [27]  ROUTE gw 9.0.0.5 via eth9_0(0x8d001382) (34)
2.9.1/32  [28]  ROUTE gw 9.1.0.5 via eth9_1(0xb9f76532) (37)
9.2.1/32  [30]  ROUTE gw 169.254.128.1 via xvrf0(0xc6129210) (40)
9.9.1/32  [28]  ROUTE gw 9.1.0.5 via eth9_1(0xb9f76532) (35)

New reference for the VRF instance: 1.

<fp-0> vrf-set 1

<fp-1> route4
# - Preferred, * - Active, > - selected
2.2.1/32  [25]  ROUTE gw 2.0.0.5 via eth2_0(0xf8e170d2) (32)
2.9.1/32  [29]  ROUTE gw 169.254.128.0 via xvrf1(0xb8d9e470) (38)
9.2.1/32  [25]  ROUTE gw 2.0.0.5 via eth2_0(0xf8e170d2) (43)
2.2.1/32  [26]  ROUTE gw 2.1.0.5 via eth2_1(0x25d8c282) (33)
2.9.1/32  [29]  ROUTE gw 169.254.128.0 via xvrf1(0xb8d9e470) (36)
9.2.1/32  [26]  ROUTE gw 2.1.0.5 via eth2_1(0x25d8c282) (41)

vrfd¶

Description

This daemon monitors iproute2 netns creation and deletion and automatically binds or deletes corresponding VRF instances.

Synopsis

vrfd [-Fvh] [-p iproute2-netns-prefix] [-s vrf-init-script]

Parameters

-F¶: Optional. Foreground mode.

-v¶: Optional. Copy logs to standard output.

-h¶: Optional. Display help.

-p <iproute2-netns-prefix>¶: Optional. Prefix of the iproute2 netns instance to monitor. This option can be set up to 64 times. vrfd matches all iproute2 netns instances except vrf0 and xvrf.

-s <vrf-init-script>¶: Optional. Custom script to execute when the VRF instance is bound.

Note

By default, the script linux-fp-sync-vrf.sh must be used to initialize Linux - Fast Path Synchronization. You can adapt it to your needs.

Userspace applications API usage examples¶

Opening a socket in the VRF3 VRF instance¶

if (libvrf_init() < 0) {
  /* error path */
  return -1;
}

if (libvrf_change(3) < 0) {
  /* error path */
  return -1;
}

/* now we are in netns vrf3 */

/* open /proc from vrf3 */
proc = open("/proc/sys/net/...", ..);

/* open a socket in vrf 3 */
socket = socket(x, y, z);

if (libvrf_back() < 0) {
  /* error path */
  return -1;
}

/* We are back to the initial network namespace, but proc and socket are still in network namespace
 * vrf3. Don't forget to close them when you are over; otherwise, it will not be
 * possible to remove this network namespace.
 */

Move an interface from the VRF2 to the VRF3 VRF instance¶

if (libvrf_init() < 0) {
      /* error path */
      return -1;
}

if (libvrf_change(2) < 0) {
      /* error path */
      return -1;
}

/* Now we are in network namespace vrf2 */

/* Open a netlink socket in vrf2 */
socket = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);

if (libvrf_back() < 0) {
      /* error path */
      return -1;
}

/* Now, we get a ref to the new network namespace, this ref will be used in the netlink
 * message.
 */
vrf3 = libvrf_get_fd(3);
if (vrf3 < 0) {
      /* error path */
      return -1;
}

/* Build a netlink message for the netdevice and then add the
 * information of the new network namespace.
 */
addattr_l(&req->n, sizeof(*req), IFLA_NET_NS_FD, &vrf3, 4);

/* Now, this message should be sent into a netlink socket of vrf2 */
sendmsg(socket, &req, 0);

/* Don't forget to close the fd at the end */
close(vrf3);