Usage

Start-up configuration of the fast path

When starting the fast path without configuration, the default behavior is to use all the supported physical network ports. On a Virtual Accelerator, one physical core per socket is enabled.

Each network port is polled by all logical cores from the same CPU socket.

The fast-path.sh script reads the /etc/fast-path.env configuration file before starting the fast path and associated modules.

A default /etc/fast-path.env configuration file template is provided. It defines the set of available options.

To customize the fast path configuration, you can either:

• use the fast path configuration wizard:

  $ fast-path.sh config -i
  

• manually edit /etc/fast-path.env fast path configuration file.

Note

When changing the configuration, the fast path must be stopped.

The fast path configuration file

Syntax

In the fast path configuration file, you can use one of the following syntaxes:

PARAMETER=value

Supersedes the corresponding global environment variable, if any. For instance: FP_MASK=2-5

: ${PARAMETER:=value}

Ignored if the corresponding global environment variable exists. For instance: : ${FP_MASK:=2-5}

The template configuration file describes the list of available options and their syntax.

# Copyright 2015-2021 6WIND S.A.
#
# This is the configuration file template for the fast path.
# Edit it prior to starting the fast path.
#   default location: /etc
#   default file name: fast-path.env

#
# NOTE: variables containing spaces must be quoted.
#

#####################
#### EAL OPTIONS ####
#####################

## FP_MASK defines which logical cores run the fast path.
##
## The value can be:
## - 'min': one physical core per socket is enabled.
## - 'half': half of the cores are enabled, spread across sockets.
## - 'max': all physical cores except one are enabled.
## - 'auto' or commented out: equivalent to 'half' on a Turbo Router,
##   or 'min' on a Virtual Accelerator.
##
## - a list of logical cores ranges.
##   Example: "1-4,6,8" means logical cores 1,2,3,4,6,8.
## - an hexadecimal mask starting with '0x': it represents the mask of logical
##   cores to be enabled.
##   Example: "0x1e" means logical cores 1,2,3,4.
##
## Note: the core 0 is usually reserved for Linux processes, so
## it's not advised to use it in FP_MASK.
##
## In expert mode (EXPERT_FPCONF=on), this parameter is mandatory and must not
## be auto.
##
#: ${FP_MASK:=auto}

## FP_MEMORY defines how much memory from the hugepages is reserved for
## the fast path in MegaBytes.
##
## The default architecture hugepage size will be used: 2MB on x86_64,
## 1GB on x86_64 if "hugepagesz=1G default_hugepagesz=1G" is passed to
## the kernel command line.
##
## The value can be:
## - 'auto' or commented out: the proper amount of memory will automatically
##   be reserved at fast path start.
## - an integer: representing the amount of memory accross all NUMA nodes.
##   The memory is automatically spread among nodes according to the
##   configuration.
##   Example: "1024" asks to reserve 1GB of memory for the fast path.
## - an integer, prefixed by '+': allocate the specified amount of memory
##   in addition to the automatically determined amount.
##   Example: "+512" allocates 512MB in addition to the amount required
##   by the fast path.
## - a list of integers, representing the amount of memory in MB to use on each
##   NUMA node.
##   Example: "256,512" asks to reserve 256MB on node 0 and 512MB on node 1.
##   Each element of the list can be prefixed with a "+": in this case,
##   this amount will be added to the auto value for this node.
##   Example: "+512,+512" asks to allocate 512MB on each socket in addition
##   to the automatic value.
##
## In expert mode (EXPERT_FPCONF=on), the parameter is mandatory and its format
## must be a list of integer, one per socket.
##
#: ${FP_MEMORY:=auto}

## HUGEPAGES_DIR defines the hugepages mountpoint (hugetlbfs). The hugepages
## are used by the DPDK to allocate its memory, and they are reserved by the
## fast path startup script. They are available from a specific filesystem
## called hugetlbfs that has to be mounted.
## Refer to the DPDK documentation for details about hugepages.
##
## The value can be:
## - 'auto' or commented out: the default path (/dev/hugepages) is used.
## - a path to a directory in the filesystem.
##
## In expert mode (EXPERT_FPCONF=on), the parameter is mandatory.
##
#: ${HUGEPAGES_DIR:=/dev/hugepages}

## This parameter is deprecated. It is replaced by EXT_MEMORY.
##
## VM_MEMORY defines how much memory from the hugepages to allocate for
## virtual machines.
##
## Note: to properly replace VM_MEMORY by EXT_MEMORY, a '+' should be
## prepended to the value. For instance:
##   VM_MEMORY=123 should be replaced by EXT_MEMORY=+123
##   VM_MEMORY=123,456 should be replaced by EXT_MEMORY=+123,+456
#: ${VM_MEMORY:=auto}

## EXT_MEMORY defines how much memory from the hugepages to allocate
## for DPDK external memory, virtual machines and other applications.
##
## When external memory is used for DPDK mbuf pool, the fast path
## startup script is able to reserve additional memory stored in huge
## pages.
##
## When running the fast path as a host managing VMs, the fast path
## startup script is able to reserve additional memory stored in huge
## pages. This memory can be used by Qemu or libvirt for the virtual
## machines.
##
## The value can be:
## - auto or commented out: if external mbuf pool is enabled, the proper
##   amount of memory will automatically be reserved at fast path start;
##   on a Virtual Accelerator, 4GB per socket will be reserved for VM,
##   on other products no VM memory will be reserved.
## - an integer: it represents the amount of memory in MB to reserve.
##   This amount will be spread equally on all NUMA nodes.
##   Example: "4096" asks to reserve 4GB for the virtual machines, distributed
##   on all the NUMA nodes of the machine (2GB per node if the machine has
##   2 nodes).
## - an integer, prefixed by '+': allocate the specified amount of memory
##   in addition to the automatically determined amount.
##   Example: "+512" allocates 512MB in addition to the amount required
##   by the fast path.
## - a list of integers, representing the amount of memory in MB
##   to reserve on each NUMA node.
##   Example: "2048,2048" asks to reserve 2GB on node0 and 2GB on node1
##   in huge pages.
##   Each element of the list can be prefixed with a "+": in this case,
##   this amount will be added to the auto value for this node.
##   Example: "+512,+512" asks to allocate 512MB on each socket in addition
##   to the automatic value.
##
## It is mandatory to stop or pause all VMs managed by fastpath to change this
## parameter otherwise the old value is kept.
## In expert mode (EXPERT_FPCONF=on), the parameter is mandatory and its format
## must be a list of integer, one per socket.
##
#: ${EXT_MEMORY:=auto}

## RESERVE_FP_HUGEPAGES enables or disables the allocation of
## huge pages used for the fast path by the startup script.
##
## When enabled (set to 'on' or commented out), the memory required to
## run the fast path is automatically reserved in the hugetlbfs at
## fast path start.
## Otherwise (set to 'off'), it is assumed that there are enough free huge
## pages reserved by the user to start the fast path. The user can
## allocate hugepages at boot time or dynamically (which is not supported
## on all kernels for 1G hugepages).
##
## In expert mode (EXPERT_FPCONF=on), the parameter is mandatory.
##
#: ${RESERVE_FP_HUGEPAGES:=on}

## FP_PORTS defines the list of ports enabled in the fast path.
##
## The value can be:
## - 'all' or commented out: all supported physical ports are enabled.
## - a space-separated list of keys, defining the list of ports. A key
##   adds one or several ports to the current list, or removes them if
##   it is prefixed by '-'. The valid keys are: 'all', a pci identifier,
##   a linux interface name.
##
##   Example: "" means no port
##   Example: "all -mgmt0" means all ports except the one associated to
##   the interface called mgmt0 in Linux.
##   Example: "0000:03:00.0 0000:03:00.1" means the ports whose pci bus
##   addresses match.
##   Example: "dpmac.1 dpmac.2" means the ports whose FSL MC bus MAC match.
##
##   A PCI bus can be suffixed by driver-specific arguments. For instance:
##   "0000:03:00.0,role=right,verbose=1".
##
##   The list is evaluated from left to right, so that
##   "eth0 0000:03:00.0 -all" means no port are enabled.
##
##   Note: be careful when using Linux interface names in configuration,
##   as they are subject to changes, depending on system configuration.
##
## This parameter is evaluated at fast path start and is converted into
## a whitelist or blacklist of PCI devices, that is passed to the fast
## path command line. Therefore, it is not possible to enable only some
## ports of a PCI device.
##
## All the ports within the same iommu group must be enabled.
##
## Example:
## "eth0": {
## "expected_pci": "0000:01:00.0",
## "iommu_group": 1
## },
## "eth1": {
## "expected_pci": "0000:01:00.1",
## "iommu_group": 1
## },
## "eth20": {
## "expected_pci": "0000:07:00.0",
## "iommu_group": 15
## }
## In the above example, we can exclude port eth20 with no issue:
## : ${FP_PORTS:=all -eth20}
## However, the following will not work:
## : ${FP_PORTS:=all -eth0}
##
## In expert mode (EXPERT_FPCONF=on), this parameter is mandatory and
## must contain a list of PCI devices only.
##
#: ${FP_PORTS:=all}

## FP_VDEVS defines the list of static virtual devices enabled in the
## fast path.
##
## The value can be:
## - "" or commented out: no static virtual device is configured
## - a space-separated list of virtual devices, as in the *--vdev* DPDK EAL
##   option.
##   The format is <driver-name><id>,cfg1=val1,cfg2=val2,...
##   Example: "pmd-vhost0,sockname=/tmp/pmd-vhost0,verbose=1".
##
## Refer to DPDK documentation for details about creation of virtual devices.
##
## Note: virtual devices created using hotplug once the fast path is running do
## not appear in the configuration.
##
#: ${FP_VDEVS:=''}

## EAL_ADDONS defines the list of EAL addons loaded by the fast path.
## An addon is a shared library that is loaded at fast path start. These
## addons usually provide new ethernet or crypto drivers.
##
## The value can be:
## - 'auto' or commented out: the required addons are automatically
##   loaded.
## - a string: the given list of addons are loaded.
##   Example: "librte_pmd_vhost.so"
## - a string prefixed by '+': the list is appended to the automatic
##   value.
##   Example: "+myaddon1.so myaddon2.so"
##
## In expert mode (EXPERT_FPCONF=on), this parameter is mandatory and
## should contain the list of EAL addons to be loaded at fast path start.
##
#: ${EAL_ADDONS:=auto}

## NB_MEM_CHANNELS defines the total number of memory channels on the machine.
## It is used by the fast path to spread the pool objects (like mbufs) among
## the different memory channels, thus increasing performance.
##
## The value can be:
## - 'auto' or commented out: the number of memory channels is determined
##   automatically when the fast path starts.
## - an integer between 1 and 32, representing the number of memory channels.
##
## This argument corresponds to the '-n' EAL option which is described
## in DPDK documentation.
##
## In expert mode (EXPERT_FPCONF=on), the parameter is mandatory.
##
#: ${NB_MEM_CHANNELS:=auto}

## CRYPTO_TYPE defines the type of crypto driver to use.
##
## The value can be:
## - 'auto' or commented out: use hardware crypto if available, else use
##   fastest available software crypto
## - the driver name: "Generic Crypto", "Intel Multi-Buffer",
##   "Intel QuickAssist", "Cavium Octeon TX crypto", "Cavium Octeon TX2 crypto"
## - the driver config name: software, multibuffer, quickassist, octeontxcpt,
##   octeontx2cpt
##
## In expert mode (EXPERT_FPCONF=on), this parameter is unused. The
## crypto driver is selected from the addons list (EAL_ADDONS).
##
#: ${CRYPTO_TYPE:=auto}

## MACHINE_MEMORY is a hint used to limit all memory used by the fast path
## (hugepages, shm, mallocs...), so it can run on a machine with this amount
## of physical memory (in megabytes).
## The wizard tries to scale down the runtime parameters according to this
## value to decrease its memory consumption, in order to keep enough free memory
## for other purposes, like VM memory.
## The value can be:
## - 'auto' or commented out: the fast path memory will be limited by available physical memory.
## - an integer: the fast path memory will be limited to the specified value.
##
#: ${MACHINE_MEMORY:=auto}

## EAL_OPTIONS specifies additional EAL options that are passed as-is
## to the fast path command line. They should not conflict with the rest
## of the configuration.
#: ${EAL_OPTIONS:='--log-level 4'}

#########################
#### FPN-SDK OPTIONS ####
#########################

## CORE_PORT_MAPPING maps fast path cores with network ports,
## specifying which logical cores poll which ports.
##
## The value can be:
## - 'auto' or commented out: each port is polled by all the logical
##   cores of the same socket.
## - a mapping string: it associates each logical core to a list
##   of ports.
##   Example: "c1=0:1/c2=2/c3=0:1:2" means the logical core 1 polls
##   the port 0 and 1, the core 2 polls the port 2, and the core 3
##   polls the ports 0, 1, and 2.
##
## Note: the port identifiers are assigned in the following order:
## - static virtual devices (FP_VDEVS), in the order they appear.
## - physical ports, in the PCI bus order.
## - hotplug ports created once fast path is running.
##
#: ${CORE_PORT_MAPPING:=auto}

## NB_MBUF defines the total number of mbufs to add in the mbufs pools.
## The mbufs store the network packets inside the fast path. The optimal
## performance is reached when there are as few mbufs as possible. However,
## mbuf allocation failure can lead to unexpected behavior.
##
## The value can be:
## - 'auto' or commented out: the optimal number of mbufs is determined
##   automatically at fast path start.
## - an integer: in this case, it specifies the number of mbufs to allocate
##   accross all nodes. The minimum value is 16384, the maximum value is 16777215.
##   Example: NB_MBUF=16384
## - an integer prefixed with '+': in this case, it specifies an additional
##   number of mbufs which is added to the automatic value.
##   Example: NB_MBUF=+16384
## - a list of integers: it specifies the amount of mbufs to allocate on
##   each NUMA node. An entry can be prefixed by '+', meaning this number
##   is added to the auto value for this node.
##   Example: NB_MBUF=20000,50000
##
## In expert mode (EXPERT_FPCONF=on), the parameter is mandatory and its format
## must be a list of integer, one per socket.
##
#: ${NB_MBUF:=auto}

## MBUF_RX_SIZE specifies the size of Rx data (excluding headroom) inside
## each mbuf.
##
## The value is an integer between 512 and 64000. The default value is 2176.
##
## In expert mode (EXPERT_FPCONF=on), the parameter is mandatory.
##
#: ${MBUF_RX_SIZE:=2176}

## MAINLOOP_DELAY_US defines the mainloop sleeping period in microseconds.
#: ${MAINLOOP_DELAY_US:=0}

## LINUX2FP_MASK defines the fast path cores that can receive packets
## from Linux. It must be included in FP_MASK.
##
## The value can be:
## - 'auto' or commented out: all fast path cores can receive packets from Linux.
## - a list of logical cores ranges.
##   Example: "2-4,6" means logical cores 2,3,4,6
## - an hexadecimal mask starting with '0x': it represents the mask of logical
##   cores to be enabled.
##   Example: "0x18" means logical cores 3,4
##
#: ${LINUX2FP_MASK:=auto}

## QOS_SCHEDULER_MASK defines the fast path cores dedicated for qos schedulers.
## It must be included in FP_MASK.
##
## The value can be:
## - 'none' or commented out: No fast path cores are reserved.
##   Example: "2-4,6" means logical cores 2,3,4,6
## - an hexadecimal mask starting with '0x': it represents the mask of logical
##   cores to be enabled.
##   Example: "0x18" means logical cores 3,4
##
#: ${QOS_SCHEDULER_MASK:=none}

## FP_OFFLOAD enables or disables the offload support in the
## fast path.
##
## To support offload features such as TSO or L4 checksum offloading to
## the NIC, or forwarding offload information from a guest to the NIC
## through a virtual interface, you must enable offloading in the fast
## path. You can then tune the offload features more precisely using
## "ethtool -K <iface> <feature> on|off".
##
## If set to 'auto', the effective configuration depends on the
## product: it is enabled on a Virtual Accelerator and disabled
## on Turbo Router and 6WINDGate.
## Else, it can be forced to "on" or "off".
##
## In expert mode (EXPERT_FPCONF=on), the parameter is mandatory
## and must be set to "on" or "off".
##
#: ${FP_OFFLOAD:=auto}

## CRYPTO_OFFLOAD_MASK defines the fast path cores that can do crypto operations
## for other fast path cores. It must be included in FP_MASK.
## NB: the crypto offloading is always done on cores in the same numa node.
##
## The value can be:
## - 'all' or commented out: all fast path cores can do crypto operations for
##   other fast path cores.
## - 'none': no fast path cores can do crypto operations for other fast path
##   cores
## - a list of logical cores ranges.
##   Example: "2-4,6" means logical cores 2,3,4,6
## - an hexadecimal mask starting with '0x': it represents the mask of logical
##   cores to be enabled.
##   Example: "0x18" means logical cores 3,4
##
#: ${CRYPTO_OFFLOAD_MASK:=all}

## MAX_GRO_MBUFS specifies the maximum number of mbufs that can be stored in
## the GRO reassembly queues for physical and virtual ports. This variable
## helps to control and allocate precisely the right amount of mbufs required
## by the GRO.
##
## This is the value per port and per core. It means that the total number of
## mbufs required for the GRO reassembly will be the total number of cores *
## (the number of physical ports * MAX_GRO_MBUFS[0] + the number of virtual
## ports * MAX_GRO_MBUFS[1]).
##
## The value should contain 2 integers separate by a coma. The first integer
## is for physical ports and the seconde one is for the virtual ports.
##
## Each integer must be between 0 and 65535. The default value
## for physical ports is 4096 and 128 for virtual ports.
##
#: ${MAX_GRO_MBUFS:=4096,128}

## MAX_VPORTS specifies the maximum number of virtual ports. This variable is
## used to define how many additionnal mbuf would be required.
##
## The value is an integer between 0 and 300. The default value is 'auto'.
##
#: ${MAX_VPORTS:=auto}

## NB_RX_DESCRIPTORS defines the default number of descriptors configured
## for each Rx queue of Ethernet ports.
##
## If set to 'auto', the effective configuration is determined automatically
## depending on the devices.
##
#: ${NB_RX_DESCRIPTORS:=auto}

## NB_TX_DESCRIPTORS defines the default number of descriptors configured
## for each Tx queue of Ethernet ports.
##
## If set to 'auto', the effective configuration is determined automatically
## depending on the devices.
##
#: ${NB_TX_DESCRIPTORS:=auto}

## NB_RXDTXD_FPVI defines the number of descriptors for each FPVI on
## virtual and physical ports.
##
## Each port in fast path has a FPVI vhost port used to send/receive packets
## between fast path and linux kernel.
##
## The number of descriptors of the FPVI vhost port can be configured with a
## different value depending on the associated fast path port devtype. The
## different devtypes for the fast path ports are:
## `phys` (i.e. any physical network device like Intel or Mellanox NICS),
## `virt` (i.e. any virtual network device like vhost-user ports),
## `excp` (i.e. internal exception port: fpn0).
## Example: phys:1024,virt:512
##
## Default value is all:512
#: ${NB_RXDTXD_FPVI:=all:512}

## SOFT_TXQ_SIZE defines the size of the software Tx queue for each
## Ethernet ports. The value must be a power of 2.
##
#: ${SOFT_TXQ_SIZE:=0}

## NB_CRYPTO_SESSION the maximum number of cryptographic sessions.
##
## The value can be:
## - an integer
## - unset or 'auto' to use the default value.
##
#: ${NB_CRYPTO_SESSION:=auto}

## NB_CRYPTO_BUFFER the maximum number of cryptographic buffers.
##
## Maximum number of 'in flight' operations, either being processed by the
## asynchronous crypto engine, or waiting in crypto device queues.
##
## The value can be:
## - an integer
## - unset or 'auto' to use the default value.
##
#: ${NB_CRYPTO_BUFFER:=auto}

## INTERCORE_RING_SIZE sets the size of intercore ring.
##
## The intercore ring isused by dataplane cores to send messages to
## another dataplane core.
## The default value is CONFIG_MCORE_INTERCORE_RING_SIZE, as specified in
## /etc/6WINDGate/fpnsdk.config.
##
## The value can be:
## - an integer
## - unset or 'auto' to use the default value.
##
#: ${INTERCORE_RING_SIZE:=auto}

## VLAN_STRIP forces to strip the VLAN header on incoming frames, if supported
## by the hardware.
##
## The value can be:
## - 'on' to enable the VLAN stripping.
## - unset or 'auto' to disable the VLAN stripping.
##
#: ${VLAN_STRIP:=off}

## NETDEV_NAME provides user request to name FPVI interfaces with a specific
## name.
##
## The value is a list of <fp-port>:<netdev-name>,...
## Ex: 0:fp1,1:fp2,10:fp10
## When a port is not in the list, the name of the FPVI netdevice will
## be the same as the Linux netdevice if it exists before starting the
## fast path. Else, the kernel will use its default interface naming
## convention ("eth%d").
##
#: ${NETDEV_NAME:=''}

## FPNSDK_OPTIONS specifies additional FPNSDK options.
#: ${FPNSDK_OPTIONS:=--logmode=console}

####################
#### FP OPTIONS ####
####################

## FP_OPTIONS specifies additional fast path options.
#: ${FP_OPTIONS:=''}

######################
#### FPVI OPTIONS ####
######################

## FPVI_VHOST_QUEUES defines the number of queues per FPVI vhost port.
## Each port in fast path has a FPVI vhost port used to send/receive packets between fast path and
## linux kernel.
## The number of queues of the FPVI vhost port can be configured with a different value in function
## of the fast path port devtype associated. The different devtypes for the fast path are:
## configured are `phys` (i.e. any physical network device like Intel or Mellanox NICS),
## `virt` (i.e. any virtual network device like vhost-user ports) and `excp' (i.e. fpn0).
## Example: phys:2,virt:1,excp:4
##
## Default value is all:1
#: ${FPVI_VHOST_QUEUES:=all:1}

#####################
#### ADV OPTIONS ####
#####################

## ADV_OPTIONS specifies advanced options specified through the wizard.
#: ${ADV_OPTIONS:=''}

## If EXPERT_FPCONF is set to 'on', the automatic calculation of some parameters
## is disabled. While not recommended, this can be needed if the configuration
## script is not able to parse the machine specifications and generates a
## wrong configuration.
##
#: ${EXPERT_FPCONF:=off}

## If DEBUG_FPCONF is set to 'on', additional debug logs are issued when the
## configuration is parsed at fast path start.
##
#: ${DEBUG_FPCONF:=off}

## If FP_NAMESPACE is set to 'off', the creation of a netns to store fast-path
## interfaces is disabled. This can be needed for some specific interface types.
##
#: ${FP_NAMESPACE:=on}

## If RESTART_DEPS is set to 'off', the external services (openvswitch, ...)
## that must be restarted each time fastpath is restarted will not be handled
## automatically by fast-path.sh or service scripts and should be handled manually
## by user.
##
#: ${RESTART_DEPS:=on}

fast path capabilities

You can change some fast path capabilities to tune the fast path according to your requirements in terms of scalability, for example, the maximum number of VRs or the maximum number of netfilter rules managed by the fast path.

These parameters are set in the fast-path.env configuration file through the FP_OPTIONS variable.

See also

• Start-up configuration of the fast path

Configurable fast path capabilities are described in Fast path capabilities section.

Microsoft Hyper-V & Microsoft Azure VMs

Running the fast path under Microsoft Hyper-V and Microsoft Azure VMs requires NetVSC NICs to be configured manually in /etc/fast-path.env due to their split device characteristics.

Warning

When NetVSC acceleration is enabled on the hypervisor, failure to configure the fast path properly may lead to inoperative ports or loss of communication with a VM as the fast path unknowingly takes over the wrong devices by default.

1. FP_PORTS must be left empty to prevent fast path from directly managing any physical devices found on the system (typically Mellanox ConnectX-3 EN series or Mellanox ConnectX-4 EN series devices on Microsoft Azure VMs):

   # Do not probe physical devices directly.
   : ${FP_PORTS:=}
   

2. FP_VDEV must describe a number of NetVSC-based netdevices to handle with the vdev_netvsc driver. To use eth1 for instance:

   # Use NetVSC interface eth1.
   : ${FP_VDEVS:=net_vdev_netvsc,iface=eth1}
   

   Several devices can be provided as follows:

   # Use NetVSC interfaces eth1, eth2 and eth3.
   : ${FP_VDEVS:=net_vdev_netvsc,iface=eth1
                 net_vdev_netvsc,iface=eth2
                 net_vdev_netvsc,iface=eth3}
   

3. Once fast path is running, use ethtool to verify that the network device is bound to the proper driver:

   ethtool -i eth1
   

   Expected driver name is net_failsafe instead of vdev_netvsc or the original hv_netvsc (Linux kernel driver):

   driver: net_failsafe
   [...]
   

Starting the fast path

1. Isolate the fast path from other system tasks using cpusets to avoid disrupting the fast path process.

2. Start the fast path with the default configuration file:

   # systemctl start fast-path.service
   

   Warning

   If you are connected to the target using the network, take care of removing this interface from the selected fast path ports, as any previous network configuration (IP addresses, MTU, routes, etc) is destroyed on interfaces assigned to the fast path.

See also

• Configuring cpusets.

Stopping the fast path

Stop the fast path:

# systemctl stop fast-path.service

If Linux - Fast Path Synchronization is active, it is stopped before the fast path.

Displaying the fast path status

• Display the current status of running fast path threads:

  # systemctl status fast-path.service
  

Applying a new fast path configuration

You can edit the .env fast path configuration file and restart the Virtual Accelerator to apply the new configuration.

# systemctl restart virtual-accelerator.target

Important

If you use Control Plane OVS, you must restart Open vSwitch after restarting the fast path.

See also

The Control Plane OVS documentation.

Saving/restoring linux networking configuration

If nothing is done, the linux networking configuration is lost after a fast path start or stop. To solve this problem, a tool named networking-config is available to save the linux networking configuration, diff it and restore it, using the netlink python-pyroute2 package.

The objects that can be restored are:

• interface state and mtu

• virtual interfaces (vlan, vxlan, macvlan, gre, gretap, ip6gre, ip6gretap, ipip, sit, ip6tnl, vti)

• IPv4 and IPv6 addresses

• IPv4 and IPv6 routes

• permanent neighbours

• bridge and bonding interfaces

Unsupported features:

• net namespaces

• policy based routing rules

To use the tool, you should save the configuration before starting or stopping the fast path, and restore it after.

Note

If you are using iptables, you should use iptables-save / iptables-restore to save and restore those rules in addition to networking-config.

Example of use:

# networking-config --help
usage: networking-config [-h] {save,diff,restore} ...

Manage the linux network configuration

positional arguments:
  {save,diff,restore}  sub-command help
    save               Save current configuration
    diff               Diff configuration with saved one
    restore            Restore saved configuration

optional arguments:
  -h, --help           show this help message and exit
# networking-config save
# head /tmp/networking-config.json
{
  "routes": {
    "v4": {
      "192.168.0.0/24": [
        {
          "RTA_DST": "192.168.0.0",
          "family": 2,
          "dst_len": 24,
          "proto": 2,
          "ifaces": [
# ip addr add 192.168.0.1/24 dev eth1
# ip link set eth1 up
# networking-config diff
eth1|IFLA_CARRIER                                     0 => 1
eth1|IFLA_CARRIER_CHANGES                             1 => 2
eth1|IFLA_OPERSTATE                                   DOWN => UP
eth1|IFLA_QDISC                                       noop => pfifo_fast
eth1|flags                                            0x1002 => 0x11043
eth1|ipaddr|192.168.0.1/24|IFA_ADDRESS                {} => 192.168.0.1
eth1|ipaddr|192.168.0.1/24|IFA_FLAGS                  {} => 128
eth1|ipaddr|192.168.0.1/24|IFA_LABEL                  {} => eth1
eth1|ipaddr|192.168.0.1/24|IFA_LOCAL                  {} => 192.168.0.1
eth1|ipaddr|192.168.0.1/24|address                    {} => 192.168.0.1
eth1|ipaddr|192.168.0.1/24|prefixlen                  {} => 24
eth1|ipaddr|192.168.0.1/24|scope                      {} => 0
eth1|ipaddr|fe80::dced:1ff:fe41:4ca3/64|IFA_ADDRESS   {} => fe80::dced:1ff:fe41:4ca3
eth1|ipaddr|fe80::dced:1ff:fe41:4ca3/64|IFA_FLAGS     {} => 128
eth1|ipaddr|fe80::dced:1ff:fe41:4ca3/64|address       {} => fe80::dced:1ff:fe41:4ca3
eth1|ipaddr|fe80::dced:1ff:fe41:4ca3/64|prefixlen     {} => 64
eth1|ipaddr|fe80::dced:1ff:fe41:4ca3/64|scope         {} => 253
# networking-config save
# ip add del 192.168.0.1/24 dev eth1
# ip addr show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether de:ed:01:41:4c:a3 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dced:1ff:fe41:4ca3/64 scope link
       valid_lft forever preferred_lft forever
# networking-config restore eth1  -v
networking-config:   INFO  starting system networking restoration
networking-config:   INFO  #### starting interfaces ####
networking-config:   INFO  ==== eth1 ====
networking-config:  DEBUG  + state DOWN
networking-config:  DEBUG  + fe80::dced:1ff:fe41:4ca3/64
networking-config:  DEBUG  + mtu 1500, state UP
networking-config:   INFO  ==== eth1 restored ====
networking-config:   INFO  #### finished interfaces ####
networking-config:   INFO  #### starting routes ####
networking-config:   INFO  ==== eth1 routes ====
networking-config:   INFO  ==== eth1 routes restored ====
networking-config:   INFO  #### finished routes ####
networking-config:   INFO  system networking restoration ended

Configuring NUMA awareness

Some protocols can cause packets to be transmitted on a processor socket different from the one where they were received, which has an impact on performance (see Performance optimization of FPN-SDK Add-on for DPDK for more details).

For some protocols, it is possible to configure the fast path to avoid processor socket crossing and therefore avoid performance loss. This feature is called NUMA awareness.

As NUMA awareness may break compatibility with Linux and/or RFCs, it is disabled by default.

NUMA awareness can be enabled with the fast path CLI:

# fp-cli numa-aware-set on

For more details on the CLI commands to manage NUMA awareness, see detailed NUMA awareness configuration.

Configuring cpusets

cpuset is a kernel mechanism which is used to control the processor and memory placement of processes. It is accessible from userspace through the cpuset pseudo filesystem.

The fast path starting scripts support the isolation of the fast path process on DPDK architectures. It is implemented in cpuset.sh, which is called by fast-path.sh.

After the fast path is started, a new cpuset “system” is created. It includes all CPUs but fast path ones. All processes except the fast path are moved from the root cpuset to the cpuset “system”.

On fast path stop, this cpuset is removed and all processes are moved back to the root cpuset.

You can configure or disable the cpuset feature in /etc/cpuset.env.

1. Display the status of cpusets:

   $ cpuset.sh status
   

See also

cpuset(7)

Managing virtual devices

Adding a new virtual port

You can dynamically create a new virtual device with the fp-vdev script.

fp-vdev add [-d pmd-vhost] [-m mtu] [-f {txt,json}] [driver options] IFNAME

IFNAME

Interface name

Optional arguments

-d vhost-pmd, --driver vhost-pmd

Driver to use (default: `vhost-pmd`).

-m mtu, --mtu mtu

The mtu value to set to the new virtual device (default: 0).

-f {txt,json}, --format {txt,json}

Which format to use to display the new virtual device (default: `txt`).

Driver options for Virtio Host PMD

Default behavior is specified in the /etc/fp-vdev.ini file under the [pmd-vhost] section.

# Copyright 2018 6WIND S.A.

# Default configuration for pmd-vhost ports created with fp-vdev.
# available options are:
# profile = ['endpoint', 'nfv']
# sockmode = ['client', 'server']
# txhash = ['l3', 'l3l4']
# verbose = <debug level>
# sockfolder = /tmp
[pmd-vhost]
profile = endpoint
verbose = 0
sockmode = client
txhash = l3l4
sockfolder = /tmp

This behavior can be changed on the fp-vdev command line by providing one of the following options:

--sockpath SOCKPATH

Set the UNIX socket path. This socket is used to negotiate features with QEMU and configure Virtio Host PMD (default: /tmp/<IFNAME>.sock).

--sockmode {client,server}

Set the UNIX socket mode. This socket is used to negotiate features with QEMU and configure Virtio Host PMD (default: client). This default can be overwritten by the sockmode option in the /etc/fp-vdev.ini file.

--nb_rings NB_RINGS

Set the number of rings to use. The default value is 1 (min=1, max=32 for Virtio Host PMD). This option is deprecated and will be removed in the next release.

--profile {endpoint,nfv}

Define the usage profile (default: endpoint). In nfv mode, mergeable buffers are disabled, and auto txhash mapping tries to lower the lock contention, at the price of deteriorating the spreading among guest cores. This default can be overwritten by the profile option in the /etc/fp-vdev.ini file.

--txhash {l3l4,l3}

Select which layers are used to calculate the Tx hash (default is l3l4). This default can be overwritten by the txhash option in the /etc/fp-vdev.ini file.

--verbose

Enable debug log (default is 0). This default can be overwritten by the verbose option in the /etc/fp-vdev.ini file.

Example

$ fp-vdev add tap1
 devargs:
   profile: endpoint
   sockmode: client
   sockname: /tmp/tap1.sock
   txhash: l3l4
   verbose: 0
 driver: pmd-vhost
 ifname: tap1
 rx_cores: all

$ fp-vdev add tap2 --profile nfv --verbose 1
 devargs:
   profile: nfv
   sockmode: client
   sockname: /tmp/tap2.sock
   txhash: l3l4
   verbose: 1
 driver: pmd-vhost
 ifname: tap2
 rx_cores: all

Deleting a virtual port

You can delete a virtual device with the fp-vdev script.

fp-vdev del IFNAME

IFNAME

Interface name

Example

$ fp-vdev del tap1