Control Plane Protection¶

Overview¶

This guide describes how to enable and configure the Control Plane Protection mechanism. Enabling this feature reduces the risk of dropping control packets when the target is under high load, or when the transmission link is overloaded.

In a network architecture, control packets are critical, since losing some of them has stronger consequences than losing data packets. For instance:

losing ARP packets can make a gateway unreachable
losing OSPF/BGP/… packets can make a network unreachable
losing IKE packets can prevent the setup of IPsec security associations
losing LACP packets can bring a link down

Control Plane Protection is a mechanism that reduces the risk of dropping these control packets. It has an impact on performance, which can be tuned depending on the required throughput and criticity of losing control packets.

This guide describes how to enable and configure this Control Plane Protection mechanism.

Recognized packet types¶

The parser recognizes ARP, ICMP, ICMPv6, OSPF, VRRP, IKE, DHCP, DHCPv6, BGP, LACP, SSH, OpenFlow, JSON RPC (TCP port 7406), Stats Collector (TCP port 39090), BFD. All can be encapsulated in VLAN, QinQ or FPTUN.

Supported platforms¶

This feature is only available on products based on DPDK. It requires a specific PMD API (called RX/TX descriptor status) which is available for the following PMDs: i40e, ixgbe, mlx5, e1000.

Design¶

Packet loss can occur at several places.

RX: when the router is overloaded, the software may not able to dequeue the incoming packets fast enough. In that case, the hardware RX ring becomes full and the NIC starts to drop packets.
TX: if the router tries to send more packets than what the network link support, the hardware TX ring becomes full and the software starts to drop packets.
Exceptions (packets sent to Linux): when the exception rate is too high, the software ring becomes full and the software starts to drop packets.

Control Plane Protection is disabled by default in RX and TX, and always enabled for exceptions.

When enabled, the Control Plane Protection mechanism checks the filling level of the rings (RX, TX, or Exception) before enqueuing or dequeuing.

If the filling level is below a threshold, nothing else is done and the packets are processed normally. Therefore, the additional CPU cost is very low in this situation.
Else, if the filling level is above the threshold, the retrieved/transmitted packets are filtered: the data plane packets are dropped and the control plane are kept. Since most packets are data plane, the ring is emptied quickly and its filling level falls back below the threshold.

The filtering of packets is done by a parser. This gives more flexibility, but it implies a cost in terms of CPU cycles. To avoid spending all the CPU time to filter and drop without doing any effective processing:

It is advised to have at least one core per 10Gb link: this ensures that even in the worst case, most CPU power is available for the processing application.
A maximum CPU budget can be assigned to the Control Plane Protection to limit the impact on throughput performance. If the limit is exceeded, only a fraction of the packets will be filtered and the Control Plane Protection will be disabled for the other part. Therefore, control plane packets can be dropped in this situation.

Usage¶

Control Plane Protection is disabled by default. If control plane packets are lost on a given port, enable Control Plane Protection using the following command:

root@host# fp-cli dpdk-cp-filter-set eth0 rx_mode software-filter tx_mode software-filter
rx cp filter is enabled: rx_mode=software-filter, rxd_thres=256, rxd_count=512
tx cp filter is enabled: tx_mode=software-filter, txd_thres=256, txd_count=512

To enable Control Plane Protection automatically on start, you can specify FPN-SDK option in fast-path.env:

root@host# vi /etc/fast-path.env
  [...]
  FPNSDK_OPTIONS="--rx-cp-filter-mode=phys:software-filter --tx-cp-filter=phys:software-filter"
  [...]

Control Plane Protection works according to a maximum CPU budget (default value is 10%). If control plane packets are still dropped after enabling Control Plane Protection, it means that this budget has to be increased using the following command:

root@host# fp-cli dpdk-cp-filter-budget-set 15
cpu budget is 15%

As described in the above section, the Control Plane Protection works according to a queue threshold. The threshold can be configured by specifying the following in the FPN-SDK option in fast-path.env:

root@host# vi /etc/fast-path.env
  [...]
  FPNSDK_OPTIONS="--rx-cp-filter-threshold=phys:2048,virt:50%,excp:10%"
  [...]

Note: The value can be provided with a fixed value or a percentage by using the % keyword. The default value is 50%.

Customizing Packet Detection¶

tcp/udp services¶

The behavior of the control plane packet parser can be customized with ‘–cp-filter-services’. This option needs to be configured in the FPNSDK_OPTIONS variable of the fast-path.env configuration file.

This option takes as argument the list of tcp or udp services that should be marked as control packets.

Example:

: ${FPNSDK_OPTIONS:=--cp-filter-services=default,udp-src-68,udp-dst-67,tcp-srcdst-80}

By default, the cp-filter-services value is:

udp-dst-500,udp-dst-4500,udp-dst-67,udp-dst-68,udp-dst-546,udp-dst-547,udp-dst-3784,
udp-dst-3785,tcp-srcdst-179,tcp-srcdst-22,tcp-srcdst-6633,tcp-srcdst-7406,tcp-srcdst-39090

to recognize ike, DHCP, DHCPv6, BGP, SSH, OpenFlow, JSON RPC, Stats Collector packets.

The default keyword can be set in cp-filter-services to still match these packets.

Note: ARP, ICMP, ICMPv6, OSPF, VRRP are always recognized as control plane packets. L2 or L3 packets detection can not be customized.

vxlan¶

By default, the Control Plane Protection protects control plane packets encapsulated in a vxlan tunnel. Only the outer packets with an udp layer with the dst ports equals to 4789 are considered as VXLAN packets by the Control Plane Protection.

The VXLAN udp port can be configured to a different value, with the --cp-filter-vxlan-port-filter option of FPNSDK_OPTIONS variable in the fast-path.env configuration.

To not protect control plane packets encapsulated in a vxlan tunnel, the --cp-filter-vxlan-port-filter needs to be set to 0.

RX mode¶

The Control Plane Protection supports three differents mode in RX: software filter, hardware filter and dedicated queue.

The software or hardware filter works like described in the design section by checking the filling level of the RX rings. When the filling is over a threshold, the data plane packets are dropped and the control plane are kept. The only difference between these two modes is how the packets are parsed. In the first mode, the parsing is done in software and in the second mode is done by the hardware NIC. In the hardware filter mode, hardware flows are programmed into the NIC to mark the control plane packets.

The dedicated queue mode works differently. In this mode, the NIC is programmed to enqueue the control plane packets into a specific RX queue. The data plane packets are enqueued by the NIC on the other RX queues.

Thus, the dedicated queue mode impacts how the fast path cores polls a port. When no core/port mapping is provided in fast-path.env (i.e. CORE_PORT_MAPPING=auto), an extra queue is added for dedicated queue mode. If a core/port mapping has been provided, the last queue is used for control plane packets. In this case, be sure to have at least 2 queues configured.

The default RX mode is none (i.e. cp filter is disabled). The hardware filter and dedicated queue mode are only supported on Mellanox NIC. Other hardware NICs don’t support to offload control plane packet detection. For these NICs, the software-filter mode is always used.

When the fast path has started, it’s possible to check which RX mode has been configured on a specific device with the following command fp-cli dpdk-cp-filter <port_id>.

Hardware flows configured on the NIC can also be dumped with the the following command fp-cli dpdk-cp-filter-hw-offload-flows <port_id>.

It’s also possible to change of RX mode dynamically with the following commands:

# fp-cli dpdk-cp-filter-set <iface_name> rx_mode hardware-filter
  (or software-filter or dedicated-queue or none to disable it)

TX mode¶

The Control Plane Protection supports only one mode in TX: software filter.

The TX mode of the Control Plane Protection can be chosen by setting the --tx-cp-filter-mode=all|phys|excp|virt:none|software-filter option in FPNSDK_OPTIONS variable of the fast-path.env configuration file.

The default TX mode is software-filter for exception devices, or none for other devices (i.e. cp filter is disabled).

When the fast path has started, it is possible to check which TX mode has been configured on a specific device with the following command fp-cli dpdk-cp-filter <port_id>.

It is also possible to change of TX mode dynamically with the following commands:

# fp-cli dpdk-cp-filter-set <iface_name> tx_mode software-filter
  (or none to disable it)

Statistics¶

Control Plane Protection provides statistics to monitor the number of filtered packets.

RX and TX Control Plane Protection statistics¶

The RX and TX Control Plane Protection statistics are available through the ethtool command.

root@host# ethtool -S eth0
  {...}
  fpn.rx_cp_passthrough: 0
  fpn.rx_cp_kept: 0
  fpn.rx_dp_drop: 0
  fpn.rx_cp_overrun: 0
  fpn.tx_cp_passthrough: 0
  fpn.tx_cp_kept: 0
  fpn.tx_dp_drop: 0
  fpn.tx_cp_overrun: 0
  {...}

When RX Control Plane Protection is enabled, fpn.rx_cp_passthrough is increased for each received packet when machine is not overloaded. These packets are processed normally without being analyzed.

If the machine is loaded (RX ring length exceeds the threshold) and the CPU budget is not reached, fpn.rx_cp_kept and fpn.rx_dp_drop will increase respectively for each control plane packet (kept) and for each data plane packet (drop).

If the CPU budget is exceeded, fpn.rx_cp_overrun is increased for each received packet. These packets are processed normally without being analyzed.

The same applies for TX.

Exception Control Plane Protection statistics¶

For exceptions Control Plane Protection, the statistics are available through fp-cli dpdk-port-stats.

root@host# fp-cli dpdk-port-stats
     fpn.rx_cp_passthrough: 25
     fpn.rx_cp_kept: 0
     fpn.rx_dp_drop: 0
     fpn.rx_cp_overrun: 0
     fpn.tx_cp_passthrough: 25
     fpn.tx_cp_kept: 0
     fpn.tx_dp_drop: 0
     fpn.tx_cp_overrun:

The Control Plane Protection on exceptions is always enabled, without CPU limit:

enq is increased for each packet enqueued in the exception ring
when the ring is more than half-full, enq_dp_drop is increased for each data plane packet dropped
when the ring is more than half-full, enq_cp_kept is increased for each control plane packet enqueued in the ring