Usage¶

Principles¶

The Open vSwitch daemons should be restarted after starting the fast path. Indeed, the fast path generates new network interfaces. If the daemons are not stopped, the bridges will have to be destroyed and recreated to re-apply the virtual switch configuration.
The Open vSwitch daemons can be restarted. The interfaces that are put in a bridge must be set up and promiscuous.

Using an Open vSwitch distribution package¶

This section implies that 6WIND packages were already installed (at least Fast Path Baseline, Fast Path OVS Acceleration and Linux - Fast Path Synchronization).

Example

Here is an example of configuration for a virtual bridge between two physical ports:

Install the Open vSwitch package:
Configure and start the fast path.
Start the linux synchronization.

Restart openvswitch service:

# /etc/init.d/openvswitch-switch stop
# /etc/init.d/openvswitch-switch start

Configure a bridge between two ports:

# ovs-vsctl add-br br0
# ovs-vsctl add-port br0 eth0
# ovs-vsctl add-port br0 eth1

Add an OpenFlow controller (optional, the installation of such controller is not covered by this document):
```
# ovs-vsctl set-controller br0 tcp:192.168.0.27:6633
```

Set the interfaces up and promiscuous:

# ip link set eth0 up
# ip link set eth0 promisc on
# ip link set eth1 up
# ip link set eth1 promisc on
# ip link set br0 up

Managing a VXLAN port¶

The Open vSwitch commands below allow you to manage VXLAN ports.

Creating a VXLAN port¶

# ovs-vsctl add-port BRIDGENAME PORTNAME -- set interface PORTNAME type=vxlan \
  options:remote_ip=REMOTE_IP options:key=VNI options:dst_port=DSTPORT

BRIDGENAME: Open vSwitch bridge’s name.
PORTNAME: VXLAN port’s name.
VNI: VXLAN Network Id.
REMOTE_IP: remote ip address.
DSTPORT: Port number of the VXLAN port (Linux default value is 8472).

Example

Create new Open vSwitch VXLAN port (vxlan1) in Open vSwitch bridge br0 with vni 1 and remote ip address 10.125.0.2.

# ovs-vsctl add-port br0 vxlan1 -- set interface vxlan1 type=vxlan \
  options:remote_ip=10.125.0.2 options:key=1 options:dst_port=8472

Creating a VXLAN GBP port¶

# ovs-vsctl add-port BRIDGENAME PORTNAME -- set interface PORTNAME type=vxlan \
  options:remote_ip=REMOTE_IP options:key=VNI options:dst_port=DSTPORT options:exts=gbp

Example

Create a new VXLAN GBP port and set the mark 0xabcd in gbp field. And also drop any incomming VXLAN GBP packet to the VXLAN GBP port with the matching mark 0x1234.

# ovs-vsctl add-port br0 vxlan1 -- set interface vxlan1 type=vxlan \
  options:remote_ip=10.125.0.2 options:key=1 options:dst_port=8472 options:exts=gbp

# ovs-ofctl add-flow br0 'in_port=1,actions=load:0xabcd->NXM_NX_TUN_GBP_ID[],NORMAL'

# ovs-ofctl add-flow br0 'priority=10,in_port=3,tun_gbp_id=0x1234,actions=drop'

Adapting maximum retention time for idle flows¶

By default, idle flows will be kept for 10 seconds in ovs-vswitchd flow table before disappearing. You may want to prolong the maximum time (in ms) idle flows will remain cached.

For instance, in order to set 60s as the maximum before idle flows removal, use:

# ovs-vsctl set Open_vSwitch . other_config:max-idle=60s

Note

This is a maximum value. You don’t have an absolute guarantee that idle flows will be kept that long. Refer to the official ovs-vswitchd documentation for details.

Note

When performing zero-loss performance tests, it is interesting to have a high value like 60s. Indeed, between two iterations, flows may otherwise disappear and packets would then go through the slow path before synchronization with the fast path. The slow path being what it is, packets would get lost and the resulting performance tests would be wrong.

About the flow cache¶

The Fast Path OVS Acceleration provides a cache to speed-up the packet processing. The cache can provide a boost of performance between 10% to 60% in function of the number of flows.

The cache is disabled by default. It can be dynamically enabled with a fp-cli command (fp-cli-cache-set).

The default cache size is calculated in function of the cpu cache size, to avoid any memory access for performance reason. Example of maximal number of flows handled by the cache is function of the last level cpu cache size:

last level cpu cache=10MB, default cache size is limited to 32768 flows.
last level cpu cache=20MB, default cache size is limited to 65536 flows.

The cache has several limitations that can impact the performance negatively in some use cases:

The cache size is limited in number of flows. If Fast Path OVS Acceleration needs to process more than this maximal size, the cache impacts the performance negatively.
The cache uses the RSS hash computed by the network device. If the NIC doesn’t support this feature (i.e. virtual device like Virtio Host PMD, Virtio Guest XEN-KVM PMD), or the driver is unable to copy the hash into the packet descriptor (i.e. Mellanox ConnectX-3 EN series PMD), the cache can not be used for the packets coming from this interface.
If two packets have the same RSS hash (i.e. non ip packet, packet with same src/dst ip and src/dst port), only one is processed by the cache. In this case, the second packet processing costs more CPU cycles with the cache than without it.
The cache is not used for GRE packets. As all packets from a GRE tunnel have the same RSS hash (i.e. the ip addresses and ports of the outer packets are identical), it will generate many cache conflicts.
The cache is not used for recirc packets. As the RSS hash of a recirc packet is the same that the initial packet injected in Fast Path OVS Acceleration, it will always conflict with the cache entry of the initial packet.

Providing options¶

You can dynamically set 3 parameters when starting the fast path:

Number of flows with option --mod-opt=fp-vswitch:--flows=<flow-number>. Default is 65536.
Number of ports with option --mod-opt=fp-vswitch:--ports=<port-count>. Default is 256.
Number of masks with option --mod-opt=fp-vswitch:--masks=<mask-number>. Default is 32768.
Hash order with option --mod-opt=fp-vswitch:--hash-order=<hash-order>. Default is 16.
Cache hash order with option --mod-opt=fp-vswitch:--cache-hash-order=<hash-order>. Default is computed in function of the cpu cache size. For more information, see about-flow-cache .

Tip

To get optimal performance, apply the following ratios to the three parameters:

Parameter	Value
–hash-order	`N`
–flows	`2 ** N`
–masks	`2 ** (N - 1)`

`fp-cli` commands¶

Enabling Fast Path OVS Acceleration provides the following additional fp-cli commands.

fp-vswitch-cache¶

Description

Show if the Fast Path OVS Acceleration cache is enabled or disabled.

Synopsis

fp-vswitch-cache

Example

<fp-0> fp-vswitch-cache
cache is off

fp-vswitch-cache-set¶

Description

Enable/Disable the Fast Path OVS Acceleration cache.

Synopsis

fp-vswitch-cache-set on|off

Example

<fp-0> fp-vswitch-cache-set on
cache is on (was off)

fp-vswitch-ports¶

Description

Print the list of ports synchronized in the fast path.

Synopsis

fp-vswitch-ports [percore] [all]

Parameters

percore: Display statistic values for each core.
all: Display all statistics (even those that are null).

Example

<fp-0> fp-vswitch-ports all
0: ovs-system (internal)
  rx_pkts:0
  tx_pkts:0
  rx_bytes:0
  tx_bytes:0
1: br0 (internal)
  rx_pkts:0
  tx_pkts:0
  rx_bytes:0
  tx_bytes:0
2: eth1 (netdev)
  rx_pkts:56
  tx_pkts:53
  rx_bytes:6529
  tx_bytes:6253
3: eth2 (netdev)
  rx_pkts:53
  tx_pkts:56
  rx_bytes:6253
  tx_bytes:6529

fp-vswitch-stats¶

Description

Print statistics about packets and flows in fp-vswitch module. All statistics below are number of packets, except control plane flow statistics, which are number of flows

Synopsis

fp-vswitch-stats

Flow statistics

flow_not_found: The flow was not found in the shared memory. Packet goes to exception.
flow_pullup_failed: Flow extraction failed, data in mbuf could not be made contiguous. Packet goes to exception.
flow_pullup_too_small: Flow extraction failed, not enough data was made contiguous. Packet goes to exception.
flow_frag_lookup_fail: Lookup of the so-called “later” fragment flow (respectively the “first” fragment flow) failed when matching the first fragment of a packet (respectively non-first fragments of a packet). Packet holding the fragment goes to exception.

Output statistics

output_ok: Successfully sent out of fp-vswitch plugin.
output_failed_no_mbuf: Failed to duplicate packet before sending it.
output_failed_no_ifp: Failed to find an interface to send it to.
output_failed_operative: Interface to send is down.
output_toobig_dropped: Non-IP packet does not fit in the interface’s MTU.
output_failed: Output on a GRE port failed.
output_failed_unknown_type: OVS port type is not supported.

Action statistics

output_dropped: No action found for the flow. Packet is dropped.
userspace: Packet processed by the ovs-vswitchd daemon.
push_vlan: VLAN header was added.
pop_vlan: VLAN header was removed.
push_mpls: MPLS header was added.
pop_mpls: MPLS header was removed.
recirc: Recirculation action was executed.
set_ethernet: Ethernet header was changed.
set_mpls: MPLS header was changed.
set_priority: Unused.
set_tunnel_id: Packet was encapsulated in a tunnel.
set_ipv4: IPv4 header was changed.
set_ipv6: IPv6 header was changed.
set_tcp: TCP header was changed.
set_udp: UDP header was changed.
set_mark: SKB mark was changed.

Control plane flow statistics

flow_add_failed: Flow could not be added to shared memory.
flow_update_failed: Flow could not be updated in shared memory.
flow_delete_failed: Flow could not be deleted from shared memory.

Example

<fp-0> fp-vswitch-stats
  cache_hit:60
  cache_miss:5
  flow_not_found:3
  flow_pullup_failed:0
  flow_pullup_too_small:0
  output_ok:65
  output_failed_no_mbuf:0
  output_failed_no_ifp:0
  output_failed_operative:0
  output_toobig_dropped:0
  output_failed:0
  output_failed_unknown_type:0
  output_dropped:0
  userspace:0
  push_vlan:0
  pop_vlan:0
  push_mpls:0
  pop_mpls:0
  recirc:0
  set_ethernet:0
  set_mpls:0
  set_priority:0
  set_tunnel_id:0
  set_ipv4:0
  set_ipv6:0
  set_tcp:0
  set_udp:0
  flow_add_failed:0
  flow_update_failed:0
  flow_delete_failed:0

fp-vswitch-flows¶

Description

Dump the current flow table as a human-readable C-like structure. Only flows with traffic are displayed: flows are removed as soon as traffic stops. The output is similar to ovs-dpctl display for key, mask and action (default value).

Note

This command doesn’t dump the controller’s flow table.

Synopsis

fp-vswitch-flows [help|[{+|-}]{item}] [...]

Parameters

To display the items below, prefix them with a plus sign (+).

To hide the items below, prefix them with a minus sign (-).

help: List all available items.
flow: Affect flow.* items globally.
next: Next flow index (enabled by default).
flow.key: Affect flow.key.* items globally.
flow.actions: Defined flow actions.
flow.actions_len: Size of flow.actions[] in bytes.
flow.dup: Number of output and recirc in flow.actions[].
flow.except: Indicate if the packets matching this flow are sent as exceptions.
flow.index: Flow index.
flow.hash: Flow hash.
flow.state: Flow state (unspecified = 0, active = 1).
flow.key.l1.ovsport: Input port.
flow.key.l2.src: Ethernet source address.
flow.key.l2.dst: Ethernet destination address.
flow.key.l2.ether_type: Ethernet frame type.
flow.key.l2.vlan_tci: If 802.1Q, TCI | VLAN_CFI; otherwise 0.
flow.key.l3.frag: FLOW_FRAG_* flags.
flow.key.l3.tos: IP ToS (including DSCP and ECN).
flow.key.l3.ttl: IP TTL/Hop limit.
flow.key.l3.proto: IP protocol or lower 8 bits of ARP opcode.
flow.key.l3.ip.src: IPv4 source address.
flow.key.l3.ip.dst: IPv4 destination address.
flow.key.l3.ip.arp.sha: ARP source hardware address.
flow.key.l3.ip.arp.tha: ARP target hardware address.
flow.key.l3.ip6.src: IPv6 source address.
flow.key.l3.ip6.dst: IPv6 destination address.
flow.key.l3.ip6.label: IPv6 flow label.
flow.key.l3.ip6.ndp.target: IPv6 neighbor discovery (ND) target.
flow.key.l3.ip6.ndp.sll: IPv6 neighbor discovery (ND) source hardware address.
flow.key.l3.ip6.ndp.tll: IPv6 neighbor discovery (ND) target hardware address.
flow.key.l4.flags: TCP flags.
flow.key.l4.sport: TCP/UDP/SCTP source port.
flow.key.l4.dport: TCP/UDP.SCTP destination port.
flow.key.tunnel.id: Encapsulating tunnel ID.
flow.key.tunnel.src: Tunnel outer IPv4 src addr.
flow.key.tunnel.dst: Tunnel outer IPv4 dst addr.
flow.key.tunnel.flags: Tunnel flags.
flow.key.tunnel.tos: Tunnel ToS.
flow.key.tunnel.ttl: Tunnel TTL

Examples

<fp-0> fp-vswitch-flows
FPVS flow table (max 65536 flows):
  sizeof(fpvs_flow_entry_t): 2432
  sizeof(struct fpvs_flow): 2400

.table = {
  [4] = { .ufid = 0x3bd045ef-b29b-4249-8525-aa0f4a9b444b, .pkts = 35, .bytes = 4541,
  .flow.key = recirc(0),in_port(2),eth(src=de:ed:01:1d:3f:0f,dst=de:ed:02:08:d7:ad),eth_type(0x0800),ipv4(src=0.0.0.0,dst=0.0.0.0,proto=0,tos=0,ttl=0,frag=0),l4(sport=0,dport=0,flags=0),
  .flow.mask = recirc(00000000),in_port(ffffffff),eth(src=ff:ff:ff:ff:ff:ff,dst=ff:ff:ff:ff:ff:ff),eth_type(0xffff),vlan(id=ffff),ipv4(src=0.0.0.0,dst=0.0.0.0,proto=0,tos=0,ttl=0,frag=ff),l4(sport=0,dport=0,flags=0),
  .flow.actions = actions(output:3),
  },
  [5] = { .ufid = 0x1595126b-3554-4d47-ae52-a9d38ed11698, .pkts = 32, .bytes = 4271,
  .flow.key = recirc(0),in_port(3),eth(src=de:ed:02:08:d7:ad,dst=de:ed:01:1d:3f:0f),eth_type(0x0800),ipv4(src=0.0.0.0,dst=0.0.0.0,proto=0,tos=0,ttl=0,frag=0),l4(sport=0,dport=0,flags=0),
  .flow.mask = recirc(00000000),in_port(ffffffff),eth(src=ff:ff:ff:ff:ff:ff,dst=ff:ff:ff:ff:ff:ff),eth_type(0xffff),vlan(id=ffff),ipv4(src=0.0.0.0,dst=0.0.0.0,proto=0,tos=0,ttl=0,frag=ff),l4(sport=0,dport=0,flags=0),
  .flow.actions = actions(output:2),
  },
}

fp-vswitch-masks¶

Description

Dump the current mask table as a human-readable C-like structure. The output is similar to ovs-dpctl display.

Synopsis

fp-vswitch-masks

Examples

<fp-0> fp-vswitch-masks
FPVS flow mask table (max 32768 masks):
  sizeof(fpvs_mask_entry_t): 192
  sizeof(struct fpvs_mask): 160

.table = {
  [2] = {
    ref_count = 2,
    range = [0x18, 0x38],
    key = recirc(00000000),in_port(ffffffff),eth(src=ff:ff:ff:ff:ff:ff,dst=ff:ff:ff:ff:ff:ff),eth_type(0xffff),vlan(id=ffff),l3(value=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,proto=0,tos=0,ttl=0,frag=ff),l4(sport=0,dport=0,flags=0)
  },
}

Usage¶

Principles¶

Using an Open vSwitch distribution package¶

Managing a VXLAN port¶

Creating a VXLAN port¶

Creating a VXLAN GBP port¶

Adapting maximum retention time for idle flows¶

About the flow cache¶

Providing options¶

fp-cli commands¶

fp-vswitch-cache¶

fp-vswitch-cache-set¶

fp-vswitch-ports¶

fp-vswitch-stats¶

fp-vswitch-flows¶

fp-vswitch-masks¶

`fp-cli` commands¶