Fast path

The fast path is the Virtual Service Router component in charge of packet processing acceleration. There is only one instance of fast path, that can manage interfaces in several VRF.

Enable the fast path

To accelerate ethernet NICs, they must be dedicated to the fast path, and the fast path must be started:

vsr> edit running
vsr running config# system fast-path
vsr running fast-path#! port pci-b0s4
vsr running fast-path# port pci-b0s5
vsr running fast-path# show config
fast-path
    enabled true
    port pci-b0s4
    port pci-b0s5
    cp-protection
        budget 10
        ..
vsr running fast-path# commit

Note

use show state / network-port to see the list of available network ports with PCI ids; it can help choosing the right ports.

The same configuration can be made using this NETCONF XML configuration:

vsr running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
  <system xmlns="urn:6wind:vrouter/system">
    <fast-path xmlns="urn:6wind:vrouter/fast-path">
      <enabled>true</enabled>
      <cp-protection>
        <budget>10</budget>
      </cp-protection>
      <port>pci-b0s4</port>
      <port>pci-b0s5</port>
      <core-mask/>
      <crypto/>
      <advanced/>
      <limits/>
    </fast-path>
  </system>
</config>

Check the current state of the fast path:

vsr running fast-path# show state
fast-path
    port pci-b0s5
    port pci-b0s4
    enabled true
    core-mask
        fast-path 2-3
        exception 0
        linux-to-fp 2-3
        ..
    cpu-usage cpu2
        busy 0
        ..
    cpu-usage cpu3
        busy 0
        ..
    cp-protection
        budget 10
        ..
    crypto
        nb-session 0
        nb-buffer 0
        ..
    advanced
        nb-mbuf 32768
        offload false
        vlan-strip false
        power-mode performance
        frequency-scaling false
        intercore-ring-size 128
        software-txq 0
        reserve-hugepages true
        ipv4-netfilter-cache true
        ipv6-netfilter-cache true
        ipv4-pre-ipsec-fragmentation off
        ipv6-pre-ipsec-fragmentation off
        ..
    limits
        fp-max-vrf 16
        ..

Note

fast path starting can take several seconds.

Configuring the core masks

In the core-mask context, the assignation of cores can be customized. This includes:

  • The cores which are dedicated to the fast path for dataplane operations. The accepted values are either a policy (min, half, max) or a core mask. By default, half of the available cores on are dedicated to the fast path for dataplane operations.

  • Which dataplane cores (included in fast path mask) that receive packets from Linux. By default, all dataplane cores.

  • The control plane cores (disjoint of fast path mask) that receive exception packets. By default, the first control plane core.

  • The mapping between fast path cores and the ports, in other words which core polls which port. By default, each port is polled by each core of the same NUMA node.

Here is an example of configuration with a custom fast path core mask and exception mask:

vsr> edit running
vsr running config# system fast-path
vsr running fast-path#! port pci-b0s4
vsr running fast-path# core-mask
vsr running core-mask# fast-path 5,9-12
vsr running core-mask# exception 0-4
vsr running core-mask# ..
vsr running fast-path# show config
fast-path
    enabled true
    port pci-b0s4
    core-mask
        fast-path 5,9-12
        exception 0-4
        ..
    cp-protection
        budget 10
        ..
    ..
vsr running fast-path# commit

Note

use show state / system linux to see the list of available cores.

The same configuration can be made using this NETCONF XML configuration:

vsr running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
  <system xmlns="urn:6wind:vrouter/system">
    <fast-path xmlns="urn:6wind:vrouter/fast-path">
      <enabled>true</enabled>
      <core-mask>
        <fast-path>5,9-12</fast-path>
        <exception>0-4</exception>
      </core-mask>
      <cp-protection>
        <budget>10</budget>
      </cp-protection>
      <crypto/>
      <advanced/>
      <limits/>
      <port>pci-b0s4</port>
    </fast-path>
  </system>
</config>

Fast path limits configuration

The fast path capabilities can be tuned according to your requirements in terms of scalability and memory footprint. This is done through the fast path limits configuration.

Here is an example of configuration with a custom number of VRs and IPv4 routes:

vsr> edit running
vsr running config# system fast-path
vsr running fast-path#! port pci-b0s4
vsr running fast-path# limits
vsr running limits# fp-max-vrf 128
vsr running limits# ip4-max-route 1000000
vsr running limits# ..
vsr running fast-path# show config
fast-path
    enabled true
    port pci-b0s4
    cp-protection
        budget 10
        ..
    limits
        fp-max-vrf 128
        ip4-max-route 1000000
        ..
    ..
vsr running fast-path# commit

Warning

Similar changes may be required in system neighbor configuration and in system conntrack configuration.

Note

Default fast path scalability limits are automatically adjusted if memory is insufficient, to prevent startup failure due to lack of memory. show state / system fast-path limits can be used to check the actual values.

The same configuration can be made using this NETCONF XML configuration:

dut-vm running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
  <system xmlns="urn:6wind:vrouter/system">
    <fast-path xmlns="urn:6wind:vrouter/fast-path">
      <enabled>true</enabled>
      <core-mask/>
      <cp-protection>
        <budget>10</budget>
      </cp-protection>
      <crypto/>
      <advanced/>
      <limits>
        <fp-max-vrf>128</fp-max-vrf>
        <ip4-max-route>1000000</ip4-max-route>
      </limits>
      <port>pci-b0s4</port>
    </fast-path>
  </system>
</config>

Advanced fast path configuration

For advanced users, some fast path parameters can also be customized: the number of network packet buffers, the number of crypto buffers or sessions, the activation of advanced offload features, the exception core mask, the hardware queue mapping etc…

Please refer to the fast path crypto command reference and the fast path advanced command reference for details.

Reducing power consumption

The / system fast-path advanced power-mode configuration node selects a default power behavior:

  • performance (default): the fast path cores runs at their maximum speed to provide the lowest latency and highest throughput.

  • eco: in this mode, the recommended fast path settings are applied to reduce power consumption.

    It enables frequency scaling if it is supported in the environment where Virtual Service Router is running (currently, baremetal on Intel servers). The frequency of fast path cores is automatically adapted to fast path load.

    If frequency scaling is not available, the eco mode falls back to the usage of a mainloop delay: when the fast path load is low, the fast path cores are put to sleep during a short time (100 µs by default).

To take advantage of these power saving features, the machine BIOS has to be configured to:

  • let the operating system dynamically control individual core frequency (example: CPU Power Management = OS DPBM)

  • enable hardware CPU sleep states (example: C1E = Enabled, C States = Enabled)

  • scale uncore frequency dynamically (example: Uncore Frequency = Dynamic)

To enable power saving, you can change the power-mode value to eco:

vsr> edit running
vsr running config# system fast-path advanced
vsr running advanced# power-mode eco
vsr running advanced# commit

A custom value can be set for frequency-scaling and mainloop-delay-us. In this case, it overrides the default value corresponding to the configured power mode.

Please refer to the fast path advanced command reference for details.

Troubleshooting

Fast Path ports

Use show fast-path ports to list the ports that are currently used by the fast path.

vsr> show fast-path ports
Id Interface Port name VRF  Driver
== ========= ========= ===  ======
 0 eth1      pci-b0s4  main net_virtio
 1 eth2      pci-b0s5  main net_virtio
 2 eth3      pci-b0s6  main net_virtio

Fast Path statistics

Use show fast-path statistics to get the statistics recorded by the fast path:

vsr> show fast-path statistics
show-fast-path-statistics
  ipv4
      ip-forwarded-datagrams 56342521
      ip-in-received 56342521
      ..
  global
      fast-path-dropped 4
      fast-path-dropped-system 4
      ..
 [...]
  interface iface1-vr0
      accelerated true
      input-bytes 7857363852
      input-packets 22470833
      output-bytes 3940231251
      output-packets 11352582
      ..
 [...]
  ..

Note

The fast-path-dropped* and exceptions statistics can help diagnosing problems when too few or no packets are forwarded.

Fast Path CPU usage

Use show fast-path cpu-usage command to get the fast path usage per core:

vsr> show fast-path cpu-usage
cpu   busy
===   ====
cpu8  73%
cpu24 68%

It is a good indicator regarding how busy the fast path cores are, processing packets.

Control Plane Protection

In a network architecture, control packets are critical, since losing some of them has stronger consequences than losing data packets:

  • losing ARP packets can make a gateway unreachable

  • losing OSPF/BGP/… packets can make a network unreachable

  • losing IKE packets can prevent the setup of IPsec security associations

Control Plane Protection is a software mechanism that reduces the risk of dropping these control packets. It has an impact on performance, which can be tuned depending on the required throughput and criticity of losing control packets.

The software parser recognizes ARP, ICMP, ICMPv6, OSPF, VRRP, IKE, DHCP, DHCPv6, BGP, LACP, SSH packets. All can be encapsulated in vlan or QinQ.

Control Plane Protection is disabled by default. It can be enabled on a per-interface basis, for RX or TX, depending on the situation:

  • RX: the router is overloaded, the software is not able to dequeue the incoming packets fast enough, the hardware RX ring becomes full and the NIC starts to drop packets.

  • TX: the router tries to send more packets than what the network link supports, the hardware TX ring becomes full and the software starts to drop packets.

Control Plane Protection works according to a maximum CPU budget. If control plane packets are still dropped after enabling Control Plane Protection, it means that this budget has to be increased.

To enable Control Plane Protection on a physical interface:

vsr running config# system fast-path
vsr running fast-path#! port pci-b0s4
vsr running fast-path# cp-protection budget 10
vsr running fast-path# / vrf main interface physical eth0
vsr running physical eth0#! port pci-b0s4
vsr running physical eth0# rx-cp-protection true
vsr running physical eth0# tx-cp-protection true

The same configuration can be made using this NETCONF XML configuration:

vsr running config# show config xml absolute
<config xmlns="urn:6wind:vrouter">
  <system xmlns="urn:6wind:vrouter/system">
    <fast-path xmlns="urn:6wind:vrouter/fast-path">
      <enabled>true</enabled>
      <core-mask/>
      <cp-protection>
        <budget>10</budget>
      </cp-protection>
      <crypto/>
      <advanced/>
      <limits/>
      <port>pci-b0s4</port>
    </fast-path>
  </system>
  <vrf>
    <name>main</name>
    <interface xmlns="urn:6wind:vrouter/interface">
      <physical>
        <name>eth0</name>
        <enabled>true</enabled>
        <ipv4>
          <enabled>true</enabled>
        </ipv4>
        <ipv6>
          <enabled>true</enabled>
        </ipv6>
        <ethernet>
          <auto-negotiate>true</auto-negotiate>
        </ethernet>
        <port>pci-b0s4</port>
        <rx-cp-protection>true</rx-cp-protection>
        <tx-cp-protection>true</tx-cp-protection>
      </physical>
    </interface>
  </vrf>
</config>

Note

the Control Plane Protection feature only works when the fast path is enabled, if the feature is supported by the NIC driver.

Control Plane Protection provides statistics to monitor the number of filtered packets:

vsr running fast-path# show interface hardware-statistics eth0
  (...)
  fpn.rx_cp_passthrough: 0
  fpn.rx_cp_kept: 0
  fpn.rx_dp_drop: 0
  fpn.rx_cp_overrun: 0
  fpn.tx_cp_passthrough: 0
  fpn.tx_cp_kept: 0
  fpn.tx_dp_drop: 0
  fpn.tx_cp_overrun: 0
  (...)

When RX Control Plane Protection is enabled, fpn.rx_cp_passthrough is increased for each received packet when machine is not overloaded. These packets are processed normally without being analyzed.

If the machine is loaded (RX ring length exceeds the threshold) and the CPU budget is not reached, fpn.rx_cp_kept and fpn.rx_dp_drop will increase respectively for each control plane packet (kept) and for each data plane packet (drop).

If the CPU budget is exceeded, fpn.rx_cp_overrun is increased for each received packet. These packets are processed normally without being analyzed.

The same applies for TX.

See also

The command reference for details.

Isolation of dataplane cores

The cores that are in charge of processing the network packets (the data plane) are dedicated to this task. The other tasks (the control plane) run on the other cores.

To display the cores affected to control plane:

vsr> show state system cp-mask
cp-mask 0-2

To change the cores affected to control plane:

vsr> edit running
vsr running config# system cp-mask 0
vsr running config# commit
Configuration committed.

Note

It is not possible to add fast path cores in cp-mask.

Important

To get the best performance, fast path cores should be isolated thanks to the cmd set-next-boot-params isolate-cpus <fp-coremask> command. A reboot is needed after the set-next-boot-params command has been issued.

It is also recommended to mask IRQs on the fast path cores for optimal zero-loss performance. Please refer to the fast path advanced command reference for details.

Traffic mirroring

The fast path allows to mirror traffic from:

  • a source (vrf/interface)

  • to a destination (vrf/interface)

  • in a given direction (ingress/egress/both)

Here is the way of configuring it:

vsr> edit running
vsr running config# system fast-path mirror-traffic
vsr running mirror-traffic# from vrf main interface ntfp1 type both to vrf main interface ntfp2
vsr running mirror-traffic# show config
{
  "vrouter-fast-path:mirror-traffic": {
    "from": [
      {
        "interface": "ntfp1",
        "vrf": "main",
        "type": "both",
        "to": {
          "vrf": "main",
          "interface": "ntfp2"
        }
      }
    ]
  }
}

Note

In order to avoid mirroring traffic in a looping way, it is forbidden to set a configuration with a source (vrf/interface) that is also a destination (vrf/interface). For example, the following configuration will trigger an error:

vsr> edit running
vsr running config# system fast-path mirror-traffic
vsr running mirror-traffic# from vrf main interface ntfp1 type ingress to vrf main interface ntfp2
vsr running mirror-traffic# from vrf main interface ntfp3 type ingress to vrf main interface ntfp1
vsr running mirror-traffic# show config
mirror-traffic
    from interface ntfp1 vrf main type ingress to interface ntfp2 vrf main
    from interface ntfp3 vrf main type ingress to interface ntfp1 vrf main
vsr running mirror-traffic# commit
ERROR: / system fast-path mirror-traffic from interface ntfp1 vrf main type ingress
    Must condition "not (../from/to[interface=current()/interface and vrf=current()/vrf])" not satisfied.
ERROR: / system fast-path mirror-traffic from interface ntfp1 vrf main type ingress
    A source cannot be a destination
Invalid configuration