Fast path¶

The fast path is the Turbo Router component in charge of packet processing acceleration. There is only one instance of fast path, that can manage interfaces in several VRF.

Enable the fast path¶

To accelerate ethernet NICs, they must be dedicated to the fast path, and the fast path must be started:

vrouter> edit running
vrouter running config# system fast-path
vrouter running fast-path#! port pci-b0s4
vrouter running fast-path# port pci-b0s5
vrouter running fast-path# show config
fast-path
    enabled true
    port pci-b0s4
    port pci-b0s5
    cp-protection
        budget 10
        ..
vrouter running fast-path# commit

Note

use show state / network-port to see the list of available network ports with PCI ids; it can help choosing the right ports.

The same configuration can be made using this NETCONF XML configuration:

vrouter running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
  <system xmlns="urn:6wind:vrouter/system">
    <fast-path xmlns="urn:6wind:vrouter/fast-path">
      <enabled>true</enabled>
      <cp-protection>
        <budget>10</budget>
      </cp-protection>
      <port>pci-b0s4</port>
      <port>pci-b0s5</port>
      <core-mask/>
      <crypto/>
      <advanced/>
      <limits/>
    </fast-path>
  </system>
</config>

Check the current state of the fast path:

vrouter running fast-path# show state
fast-path
    port pci-b0s5
    port pci-b0s4
    enabled true
    core-mask
        fast-path 2-3
        exception 0
        linux-to-fp 2-3
        ..
    cpu-usage cpu2
        busy 0
        ..
    cpu-usage cpu3
        busy 0
        ..
    cp-protection
        budget 10
        ..
    crypto
        nb-session 0
        nb-buffer 0
        ..
    advanced
        nb-mbuf 32768
        offload false
        vlan-strip false
        intercore-ring-size 128
        software-txq 0
        reserve-hugepages true
        ipv4-netfilter-cache true
        ipv6-netfilter-cache true
        ipv4-pre-ipsec-fragmentation off
        ipv6-pre-ipsec-fragmentation off
        ..
    limits
        fp-max-vrf 16
        ..

Note

fast path starting can take several seconds.

Configuring the core masks¶

In the core-mask context, the assignation of cores can be customized. This includes:

The cores which are dedicated to the fast path for dataplane operations. The accepted values are either a policy (min, half, max) or a core mask. By default, half of the available cores on are dedicated to the fast path for dataplane operations.
Which dataplane cores (included in fast path mask) that receive packets from Linux. By default, all dataplane cores.
The control plane cores (disjoint of fast path mask) that receive exception packets. By default, the first control plane core.
The mapping between fast path cores and the ports, in other words which core polls which port. By default, each port is polled by each core of the same NUMA node.

Here is an example of configuration with a custom fast path core mask and exception mask:

vrouter> edit running
vrouter running config# system fast-path
vrouter running fast-path#! port pci-b0s4
vrouter running fast-path# core-mask
vrouter running core-mask# fast-path 5,9-12
vrouter running core-mask# exception 0-4
vrouter running core-mask# ..
vrouter running fast-path# show config
fast-path
    enabled true
    port pci-b0s4
    core-mask
        fast-path 5,9-12
        exception 0-4
        ..
    cp-protection
        budget 10
        ..
    ..
vrouter running fast-path# commit

Note

use show state / system linux to see the list of available cores.

The same configuration can be made using this NETCONF XML configuration:

vrouter running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
  <system xmlns="urn:6wind:vrouter/system">
    <fast-path xmlns="urn:6wind:vrouter/fast-path">
      <enabled>true</enabled>
      <core-mask>
        <fast-path>5,9-12</fast-path>
        <exception>0-4</exception>
      </core-mask>
      <cp-protection>
        <budget>10</budget>
      </cp-protection>
      <crypto/>
      <advanced/>
      <limits/>
      <port>pci-b0s4</port>
    </fast-path>
  </system>
</config>

Fast path limits configuration¶

The fast path capabilities can be tuned according to your requirements in terms of scalability and memory footprint. This is done through the fast path limits configuration.

Here is an example of configuration with a custom number of VRs and IPv4 routes:

vrouter> edit running
vrouter running config# system fast-path
vrouter running fast-path#! port pci-b0s4
vrouter running fast-path# limits
vrouter running limits# fp-max-vrf 128
vrouter running limits# ip4-max-route 1000000
vrouter running limits# ..
vrouter running fast-path# show config
fast-path
    enabled true
    port pci-b0s4
    cp-protection
        budget 10
        ..
    limits
        fp-max-vrf 128
        ip4-max-route 1000000
        ..
    ..
vrouter running fast-path# commit

Warning

Similar changes may be required in system neighbor configuration and in system conntrack configuration.

Note

Default fast path scalability limits are automatically adjusted if memory is insufficient, to prevent startup failure due to lack of memory. show state / system fast-path limits can be used to check the actual values.

The same configuration can be made using this NETCONF XML configuration:

dut-vm running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
  <system xmlns="urn:6wind:vrouter/system">
    <fast-path xmlns="urn:6wind:vrouter/fast-path">
      <enabled>true</enabled>
      <core-mask/>
      <cp-protection>
        <budget>10</budget>
      </cp-protection>
      <crypto/>
      <advanced/>
      <limits>
        <fp-max-vrf>128</fp-max-vrf>
        <ip4-max-route>1000000</ip4-max-route>
      </limits>
      <port>pci-b0s4</port>
    </fast-path>
  </system>
</config>

Advanced fast path configuration¶

For advanced users, some fast path parameters can also be customized: the number of network packet buffers, the number of crypto buffers or sessions, the activation of advanced offload features, the exception core mask, the hardware queue mapping etc…

Please refer to the fast path crypto command reference and the fast path advanced command reference for details.

Control Plane Protection¶

In a network architecture, control packets are critical, since losing some of them has stronger consequences than losing data packets:

losing ARP packets can make a gateway unreachable
losing OSPF/BGP/… packets can make a network unreachable
losing IKE packets can prevent the setup of IPsec security associations

Control Plane Protection is a software mechanism that reduces the risk of dropping these control packets. It has an impact on performance, which can be tuned depending on the required throughput and criticity of losing control packets.

The software parser recognizes ARP, ICMP, ICMPv6, OSPF, VRRP, IKE, DHCP, DHCPv6, BGP, LACP, SSH, OpenFlow, JSON RPC (TCP port 7406), Stats Collector (TCP port 39090), DPVI packets. All can be encapsulated in VLAN, QinQ or FPTUN.

Control Plane Protection is disabled by default. It can be enabled on a per-interface basis, for RX or TX, depending on the situation:

RX: the router is overloaded, the software is not able to dequeue the incoming packets fast enough, the hardware RX ring becomes full and the NIC starts to drop packets.
TX: the router tries to send more packets than what the network link supports, the hardware TX ring becomes full and the software starts to drop packets.

Control Plane Protection works according to a maximum CPU budget. If control plane packets are still dropped after enabling Control Plane Protection, it means that this budget has to be increased.

To enable Control Plane Protection on a physical interface:

vrouter running config# system fast-path
vrouter running fast-path#! port pci-b0s4
vrouter running fast-path# cp-protection budget 10
vrouter running fast-path# / vrf main interface physical eth0
vrouter running physical eth0#! port pci-b0s4
vrouter running physical eth0# rx-cp-protection true
vrouter running physical eth0# tx-cp-protection true

The same configuration can be made using this NETCONF XML configuration:

vrouter running config# show config xml absolute
<config xmlns="urn:6wind:vrouter">
  <system xmlns="urn:6wind:vrouter/system">
    <fast-path xmlns="urn:6wind:vrouter/fast-path">
      <enabled>true</enabled>
      <core-mask/>
      <cp-protection>
        <budget>10</budget>
      </cp-protection>
      <crypto/>
      <advanced/>
      <limits/>
      <port>pci-b0s4</port>
    </fast-path>
  </system>
  <vrf>
    <name>main</name>
    <interface xmlns="urn:6wind:vrouter/interface">
      <physical>
        <name>eth0</name>
        <enabled>true</enabled>
        <ipv4>
          <enabled>true</enabled>
        </ipv4>
        <ipv6>
          <enabled>true</enabled>
        </ipv6>
        <ethernet>
          <auto-negotiate>true</auto-negotiate>
        </ethernet>
        <port>pci-b0s4</port>
        <rx-cp-protection>true</rx-cp-protection>
        <tx-cp-protection>true</tx-cp-protection>
      </physical>
    </interface>
  </vrf>
</config>

Note

the Control Plane Protection feature only works when the fast path is enabled, if the feature is supported by the NIC driver.

Control Plane Protection provides statistics to monitor the number of filtered packets:

vrouter running fast-path# show interface hardware-statistics eth0
  (...)
  fpn.rx_cp_passthrough: 0
  fpn.rx_cp_kept: 0
  fpn.rx_dp_drop: 0
  fpn.rx_cp_overrun: 0
  fpn.tx_cp_passthrough: 0
  fpn.tx_cp_kept: 0
  fpn.tx_dp_drop: 0
  fpn.tx_cp_overrun: 0
  (...)

When RX Control Plane Protection is enabled, fpn.rx_cp_passthrough is increased for each received packet when machine is not overloaded. These packets are processed normally without being analyzed.

If the machine is loaded (RX ring length exceeds the threshold) and the CPU budget is not reached, fpn.rx_cp_kept and fpn.rx_dp_drop will increase respectively for each control plane packet (kept) and for each data plane packet (drop).

If the CPU budget is exceeded, fpn.rx_cp_overrun is increased for each received packet. These packets are processed normally without being analyzed.

The same applies for TX.

Isolation of dataplane cores¶

The cores that are in charge of processing the network packets (the data plane) are dedicated to this task. The other tasks (the control plane) run on the other cores.

To display the cores affected to control plane:

vrouter> show state system cp-mask
cp-mask 0-2

To change the cores affected to control plane:

vrouter> edit running
vrouter running config# system cp-mask 0
vrouter running config# commit
Configuration committed.

Note

It is not possible to add fast path cores in cp-mask.

Important

To get the best performance, fast path cores should be isolated thanks to the cmd set-next-boot-params isolate-cpus <fp-coremask> command. A reboot is needed after the set-next-boot-params command has been issued.