Fast path¶
The fast path is the Turbo Router component in charge of packet processing acceleration. There is only one instance of fast path, that can manage interfaces in several VRF.
Enable the fast path¶
To accelerate ethernet NICs, they must be dedicated to the fast path, and the fast path must be started:
vrouter> edit running
vrouter running config# system fast-path
vrouter running fast-path#! port pci-b0s4
vrouter running fast-path# port pci-b0s5
vrouter running fast-path# show config
fast-path
enabled true
port pci-b0s4
port pci-b0s5
cp-protection
budget 10
..
vrouter running fast-path# commit
Note
use show state / network-port
to see the list of available
network ports with PCI ids; it can help choosing the right ports.
The same configuration can be made using this NETCONF XML configuration:
vrouter running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
<system xmlns="urn:6wind:vrouter/system">
<fast-path xmlns="urn:6wind:vrouter/fast-path">
<enabled>true</enabled>
<cp-protection>
<budget>10</budget>
</cp-protection>
<port>pci-b0s4</port>
<port>pci-b0s5</port>
<core-mask/>
<crypto/>
<advanced/>
<limits/>
</fast-path>
</system>
</config>
Check the current state of the fast path:
vrouter running fast-path# show state
fast-path
port pci-b0s5
port pci-b0s4
enabled true
core-mask
fast-path 2-3
exception 0
linux-to-fp 2-3
..
cpu-usage cpu2
busy 0
..
cpu-usage cpu3
busy 0
..
cp-protection
budget 10
..
crypto
nb-session 0
nb-buffer 0
..
advanced
nb-mbuf 32768
offload false
vlan-strip false
intercore-ring-size 128
software-txq 0
reserve-hugepages true
ipv4-netfilter-cache true
ipv6-netfilter-cache true
..
limits
fp-max-vrf 16
..
Note
fast path starting can take several seconds.
Configuring the core masks¶
In the core-mask
context, the assignation of cores can be customized. This
includes:
The cores which are dedicated to the fast path for dataplane operations. The accepted values are either a policy (
min
,half
,max
) or a core mask. By default, half of the available cores on are dedicated to the fast path for dataplane operations.Which dataplane cores (included in fast path mask) that receive packets from Linux. By default, all dataplane cores.
The control plane cores (disjoint of fast path mask) that receive exception packets. By default, the first control plane core.
The mapping between fast path cores and the ports, in other words which core polls which port. By default, each port is polled by each core of the same NUMA node.
Here is an example of configuration with a custom fast path core mask and exception mask:
vrouter> edit running
vrouter running config# system fast-path
vrouter running fast-path#! port pci-b0s4
vrouter running fast-path# core-mask
vrouter running core-mask# fast-path 5,9-12
vrouter running core-mask# exception 0-4
vrouter running core-mask# ..
vrouter running fast-path# show config
fast-path
enabled true
port pci-b0s4
core-mask
fast-path 5,9-12
exception 0-4
..
cp-protection
budget 10
..
..
vrouter running fast-path# commit
Note
use show state / system linux
to see the list of available
cores.
The same configuration can be made using this NETCONF XML configuration:
vrouter running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
<system xmlns="urn:6wind:vrouter/system">
<fast-path xmlns="urn:6wind:vrouter/fast-path">
<enabled>true</enabled>
<core-mask>
<fast-path>5,9-12</fast-path>
<exception>0-4</exception>
</core-mask>
<cp-protection>
<budget>10</budget>
</cp-protection>
<crypto/>
<advanced/>
<limits/>
<port>pci-b0s4</port>
</fast-path>
</system>
</config>
Fast path limits configuration¶
The fast path capabilities can be tuned according to your requirements in terms of scalability and memory footprint. This is done through the fast path limits configuration.
Here is an example of configuration with a custom number of VRs and IPv4 routes:
vrouter> edit running
vrouter running config# system fast-path
vrouter running fast-path#! port pci-b0s4
vrouter running fast-path# limits
vrouter running limits# fp-max-vrf 128
vrouter running limits# ip4-max-route 1000000
vrouter running limits# ..
vrouter running fast-path# show config
fast-path
enabled true
port pci-b0s4
cp-protection
budget 10
..
limits
fp-max-vrf 128
ip4-max-route 1000000
..
..
vrouter running fast-path# commit
Warning
Similar changes may be required in system neighbor configuration and in system conntrack configuration.
Note
Default fast path scalability limits are automatically adjusted if memory is
insufficient, to prevent startup failure due to lack of memory. show state
/ system fast-path limits
can be used to check the actual values.
The same configuration can be made using this NETCONF XML configuration:
dut-vm running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
<system xmlns="urn:6wind:vrouter/system">
<fast-path xmlns="urn:6wind:vrouter/fast-path">
<enabled>true</enabled>
<core-mask/>
<cp-protection>
<budget>10</budget>
</cp-protection>
<crypto/>
<advanced/>
<limits>
<fp-max-vrf>128</fp-max-vrf>
<ip4-max-route>1000000</ip4-max-route>
</limits>
<port>pci-b0s4</port>
</fast-path>
</system>
</config>
Advanced fast path configuration¶
For advanced users, some fast path parameters can also be customized: the number of network packet buffers, the number of crypto buffers or sessions, the activation of advanced offload features, the exception core mask, etc…
Please refer to the fast path crypto command reference and the fast path advanced command reference for details.
Control Plane Protection¶
In a network architecture, control packets are critical, since losing some of them has stronger consequences than losing data packets:
losing ARP packets can make a gateway unreachable
losing OSPF/BGP/… packets can make a network unreachable
losing IKE packets can prevent the setup of IPsec security associations
Control Plane Protection is a software mechanism that reduces the risk of dropping these control packets. It has an impact on performance, which can be tuned depending on the required throughput and criticity of losing control packets.
The software parser recognizes ARP, ICMP, ICMPv6, OSPF, VRRP, IKE, DHCP, DHCPv6, BGP, LACP, SSH, OpenFlow, JSON RPC (TCP port 7406), Stats Collector (TCP port 39090), DPVI packets. All can be encapsulated in VLAN, QinQ or FPTUN.
Control Plane Protection is disabled by default. It can be enabled on a per-interface basis, for RX or TX, depending on the situation:
RX: the router is overloaded, the software is not able to dequeue the incoming packets fast enough, the hardware RX ring becomes full and the NIC starts to drop packets.
TX: the router tries to send more packets than what the network link supports, the hardware TX ring becomes full and the software starts to drop packets.
Control Plane Protection works according to a maximum CPU budget. If control plane packets are still dropped after enabling Control Plane Protection, it means that this budget has to be increased.
To enable Control Plane Protection on a physical interface:
vrouter running config# system fast-path
vrouter running fast-path#! port pci-b0s4
vrouter running fast-path# cp-protection budget 10
vrouter running fast-path# / vrf main interface physical eth0
vrouter running physical eth0#! port pci-b0s4
vrouter running physical eth0# rx-cp-protection true
vrouter running physical eth0# tx-cp-protection true
The same configuration can be made using this NETCONF XML configuration:
vrouter running config# show config xml absolute
<config xmlns="urn:6wind:vrouter">
<system xmlns="urn:6wind:vrouter/system">
<fast-path xmlns="urn:6wind:vrouter/fast-path">
<enabled>true</enabled>
<core-mask/>
<cp-protection>
<budget>10</budget>
</cp-protection>
<crypto/>
<advanced/>
<limits/>
<port>pci-b0s4</port>
</fast-path>
</system>
<vrf>
<name>main</name>
<interface xmlns="urn:6wind:vrouter/interface">
<physical>
<name>eth0</name>
<enabled>true</enabled>
<ipv4>
<enabled>true</enabled>
</ipv4>
<ipv6>
<enabled>true</enabled>
</ipv6>
<ethernet>
<auto-negotiate>true</auto-negotiate>
</ethernet>
<port>pci-b0s4</port>
<rx-cp-protection>true</rx-cp-protection>
<tx-cp-protection>true</tx-cp-protection>
</physical>
</interface>
</vrf>
</config>
Note
the Control Plane Protection feature only works when the fast path is enabled, if the feature is supported by the NIC driver.
Control Plane Protection provides statistics to monitor the number of filtered packets:
vrouter running fast-path# show interface hardware-statistics eth0
(...)
fpn.rx_cp_passthrough: 0
fpn.rx_cp_kept: 0
fpn.rx_dp_drop: 0
fpn.rx_cp_overrun: 0
fpn.tx_cp_passthrough: 0
fpn.tx_cp_kept: 0
fpn.tx_dp_drop: 0
fpn.tx_cp_overrun: 0
(...)
When RX Control Plane Protection is enabled, fpn.rx_cp_passthrough
is increased for
each received packet when machine is not overloaded. These packets are processed
normally without being analyzed.
If the machine is loaded (RX ring length exceeds the threshold) and the CPU
budget is not reached, fpn.rx_cp_kept
and fpn.rx_dp_drop
will increase
respectively for each control plane packet (kept) and for each data plane packet (drop).
If the CPU budget is exceeded, fpn.rx_cp_overrun
is increased for each
received packet. These packets are processed normally without being analyzed.
The same applies for TX.
See also
The command reference for details.
Isolation of dataplane cores¶
The cores that are in charge of processing the network packets (the data plane) are dedicated to this task. The other tasks (the control plane) run on the other cores.
To display the cores affected to control plane:
vrouter> show state system cp-mask
cp-mask 0-2
To change the cores affected to control plane:
vrouter> edit running
vrouter running config# system cp-mask 0
vrouter running config# commit
Configuration committed.
Note
It is not possible to add fast path cores in cp-mask.