Fast path¶
The fast path is the Virtual Service Router component in charge of packet processing acceleration. There is only one instance of fast path, that can manage interfaces in several VRF.
Enable the fast path¶
To accelerate ethernet NICs, they must be dedicated to the fast path, and the fast path must be started:
vsr> edit running
vsr running config# system fast-path
vsr running fast-path#! port pci-b0s4
vsr running fast-path# port pci-b0s5
vsr running fast-path# show config
fast-path
enabled true
port pci-b0s4
port pci-b0s5
cp-protection
budget 10
..
vsr running fast-path# commit
Note
use show state / network-port to see the list of available
network ports with PCI ids; it can help choosing the right ports.
The same configuration can be made using this NETCONF XML configuration:
vsr running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
<system xmlns="urn:6wind:vrouter/system">
<fast-path xmlns="urn:6wind:vrouter/fast-path">
<enabled>true</enabled>
<cp-protection>
<budget>10</budget>
</cp-protection>
<port>pci-b0s4</port>
<port>pci-b0s5</port>
<core-mask/>
<crypto/>
<advanced/>
<limits/>
</fast-path>
</system>
</config>
Check the current state of the fast path:
vsr running fast-path# show state
fast-path
port pci-b0s5
port pci-b0s4
enabled true
core-mask
fast-path 2-3
exception 0
linux-to-fp 2-3
..
cpu-usage cpu2
busy 0
..
cpu-usage cpu3
busy 0
..
cp-protection
budget 10
..
crypto
nb-session 0
nb-buffer 0
..
advanced
nb-mbuf 32768
offload false
vlan-strip false
intercore-ring-size 128
software-txq 0
reserve-hugepages true
ipv4-netfilter-cache true
ipv6-netfilter-cache true
ipv4-pre-ipsec-fragmentation off
ipv6-pre-ipsec-fragmentation off
..
limits
fp-max-vrf 16
..
Note
fast path starting can take several seconds.
Configuring the core masks¶
In the core-mask context, the assignation of cores can be customized. This
includes:
The cores which are dedicated to the fast path for dataplane operations. The accepted values are either a policy (
min,half,max) or a core mask. By default, half of the available cores on are dedicated to the fast path for dataplane operations.Which dataplane cores (included in fast path mask) that receive packets from Linux. By default, all dataplane cores.
The control plane cores (disjoint of fast path mask) that receive exception packets. By default, the first control plane core.
The mapping between fast path cores and the ports, in other words which core polls which port. By default, each port is polled by each core of the same NUMA node.
Here is an example of configuration with a custom fast path core mask and exception mask:
vsr> edit running
vsr running config# system fast-path
vsr running fast-path#! port pci-b0s4
vsr running fast-path# core-mask
vsr running core-mask# fast-path 5,9-12
vsr running core-mask# exception 0-4
vsr running core-mask# ..
vsr running fast-path# show config
fast-path
enabled true
port pci-b0s4
core-mask
fast-path 5,9-12
exception 0-4
..
cp-protection
budget 10
..
..
vsr running fast-path# commit
Note
use show state / system linux to see the list of available
cores.
The same configuration can be made using this NETCONF XML configuration:
vsr running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
<system xmlns="urn:6wind:vrouter/system">
<fast-path xmlns="urn:6wind:vrouter/fast-path">
<enabled>true</enabled>
<core-mask>
<fast-path>5,9-12</fast-path>
<exception>0-4</exception>
</core-mask>
<cp-protection>
<budget>10</budget>
</cp-protection>
<crypto/>
<advanced/>
<limits/>
<port>pci-b0s4</port>
</fast-path>
</system>
</config>
Fast path limits configuration¶
The fast path capabilities can be tuned according to your requirements in terms of scalability and memory footprint. This is done through the fast path limits configuration.
Here is an example of configuration with a custom number of VRs and IPv4 routes:
vsr> edit running
vsr running config# system fast-path
vsr running fast-path#! port pci-b0s4
vsr running fast-path# limits
vsr running limits# fp-max-vrf 128
vsr running limits# ip4-max-route 1000000
vsr running limits# ..
vsr running fast-path# show config
fast-path
enabled true
port pci-b0s4
cp-protection
budget 10
..
limits
fp-max-vrf 128
ip4-max-route 1000000
..
..
vsr running fast-path# commit
Warning
Similar changes may be required in system neighbor configuration and in system conntrack configuration.
Note
Default fast path scalability limits are automatically adjusted if memory is
insufficient, to prevent startup failure due to lack of memory. show state
/ system fast-path limits can be used to check the actual values.
The same configuration can be made using this NETCONF XML configuration:
dut-vm running config# show config xml absolute system fast-path
<config xmlns="urn:6wind:vrouter">
<system xmlns="urn:6wind:vrouter/system">
<fast-path xmlns="urn:6wind:vrouter/fast-path">
<enabled>true</enabled>
<core-mask/>
<cp-protection>
<budget>10</budget>
</cp-protection>
<crypto/>
<advanced/>
<limits>
<fp-max-vrf>128</fp-max-vrf>
<ip4-max-route>1000000</ip4-max-route>
</limits>
<port>pci-b0s4</port>
</fast-path>
</system>
</config>
Advanced fast path configuration¶
For advanced users, some fast path parameters can also be customized: the number of network packet buffers, the number of crypto buffers or sessions, the activation of advanced offload features, the exception core mask, the hardware queue mapping etc…
Please refer to the fast path crypto command reference and the fast path advanced command reference for details.
Control Plane Protection¶
In a network architecture, control packets are critical, since losing some of them has stronger consequences than losing data packets:
losing ARP packets can make a gateway unreachable
losing OSPF/BGP/… packets can make a network unreachable
losing IKE packets can prevent the setup of IPsec security associations
Control Plane Protection is a software mechanism that reduces the risk of dropping these control packets. It has an impact on performance, which can be tuned depending on the required throughput and criticity of losing control packets.
The software parser recognizes ARP, ICMP, ICMPv6, OSPF, VRRP, IKE, DHCP, DHCPv6, BGP, LACP, SSH, OpenFlow, JSON RPC (TCP port 7406), Stats Collector (TCP port 39090), DPVI packets. All can be encapsulated in VLAN, QinQ or FPTUN.
Control Plane Protection is disabled by default. It can be enabled on a per-interface basis, for RX or TX, depending on the situation:
RX: the router is overloaded, the software is not able to dequeue the incoming packets fast enough, the hardware RX ring becomes full and the NIC starts to drop packets.
TX: the router tries to send more packets than what the network link supports, the hardware TX ring becomes full and the software starts to drop packets.
Control Plane Protection works according to a maximum CPU budget. If control plane packets are still dropped after enabling Control Plane Protection, it means that this budget has to be increased.
To enable Control Plane Protection on a physical interface:
vsr running config# system fast-path
vsr running fast-path#! port pci-b0s4
vsr running fast-path# cp-protection budget 10
vsr running fast-path# / vrf main interface physical eth0
vsr running physical eth0#! port pci-b0s4
vsr running physical eth0# rx-cp-protection true
vsr running physical eth0# tx-cp-protection true
The same configuration can be made using this NETCONF XML configuration:
vsr running config# show config xml absolute
<config xmlns="urn:6wind:vrouter">
<system xmlns="urn:6wind:vrouter/system">
<fast-path xmlns="urn:6wind:vrouter/fast-path">
<enabled>true</enabled>
<core-mask/>
<cp-protection>
<budget>10</budget>
</cp-protection>
<crypto/>
<advanced/>
<limits/>
<port>pci-b0s4</port>
</fast-path>
</system>
<vrf>
<name>main</name>
<interface xmlns="urn:6wind:vrouter/interface">
<physical>
<name>eth0</name>
<enabled>true</enabled>
<ipv4>
<enabled>true</enabled>
</ipv4>
<ipv6>
<enabled>true</enabled>
</ipv6>
<ethernet>
<auto-negotiate>true</auto-negotiate>
</ethernet>
<port>pci-b0s4</port>
<rx-cp-protection>true</rx-cp-protection>
<tx-cp-protection>true</tx-cp-protection>
</physical>
</interface>
</vrf>
</config>
Note
the Control Plane Protection feature only works when the fast path is enabled, if the feature is supported by the NIC driver.
Control Plane Protection provides statistics to monitor the number of filtered packets:
vsr running fast-path# show interface hardware-statistics eth0
(...)
fpn.rx_cp_passthrough: 0
fpn.rx_cp_kept: 0
fpn.rx_dp_drop: 0
fpn.rx_cp_overrun: 0
fpn.tx_cp_passthrough: 0
fpn.tx_cp_kept: 0
fpn.tx_dp_drop: 0
fpn.tx_cp_overrun: 0
(...)
When RX Control Plane Protection is enabled, fpn.rx_cp_passthrough is increased for
each received packet when machine is not overloaded. These packets are processed
normally without being analyzed.
If the machine is loaded (RX ring length exceeds the threshold) and the CPU
budget is not reached, fpn.rx_cp_kept and fpn.rx_dp_drop will increase
respectively for each control plane packet (kept) and for each data plane packet (drop).
If the CPU budget is exceeded, fpn.rx_cp_overrun is increased for each
received packet. These packets are processed normally without being analyzed.
The same applies for TX.
See also
The command reference for details.
Isolation of dataplane cores¶
The cores that are in charge of processing the network packets (the data plane) are dedicated to this task. The other tasks (the control plane) run on the other cores.
To display the cores affected to control plane:
vsr> show state system cp-mask
cp-mask 0-2
To change the cores affected to control plane:
vsr> edit running
vsr running config# system cp-mask 0
vsr running config# commit
Configuration committed.
Note
It is not possible to add fast path cores in cp-mask.
Important
To get the best performance, fast path cores should be isolated thanks
to the cmd set-next-boot-params isolate-cpus <fp-coremask> command.
A reboot is needed after the set-next-boot-params command has been issued.
It is also recommended to mask IRQs on the fast path cores for optimal zero-loss performance. Please refer to the fast path advanced command reference for details.
Traffic mirroring¶
- The fast path allows to mirror traffic from:
a source (vrf/interface)
to a destination (vrf/interface)
in a given direction (ingress/egress/both)
Here is the way of configuring it:
vsr> edit running
vsr running config# system fast-path mirror-traffic
vsr running mirror-traffic# from vrf main interface ntfp1 type both to vrf main interface ntfp2
vsr running mirror-traffic# show config
{
"vrouter-fast-path:mirror-traffic": {
"from": [
{
"interface": "ntfp1",
"vrf": "main",
"type": "both",
"to": {
"vrf": "main",
"interface": "ntfp2"
}
}
]
}
}
Note
In order to avoid mirroring traffic in a looping way, it is forbidden to set a configuration with a source (vrf/interface) that is also a destination (vrf/interface). For example, the following configuration will trigger an error:
vsr> edit running
vsr running config# system fast-path mirror-traffic
vsr running mirror-traffic# from vrf main interface ntfp1 type ingress to vrf main interface ntfp2
vsr running mirror-traffic# from vrf main interface ntfp3 type ingress to vrf main interface ntfp1
vsr running mirror-traffic# show config
mirror-traffic
from interface ntfp1 vrf main type ingress to interface ntfp2 vrf main
from interface ntfp3 vrf main type ingress to interface ntfp1 vrf main
vsr running mirror-traffic# commit
ERROR: / system fast-path mirror-traffic from interface ntfp1 vrf main type ingress
Must condition "not (../from/to[interface=current()/interface and vrf=current()/vrf])" not satisfied.
ERROR: / system fast-path mirror-traffic from interface ntfp1 vrf main type ingress
A source cannot be a destination
Invalid configuration