Usage¶

In this section, it is assumed that Virtual Accelerator has been properly installed and configured. See Getting Started for more details.

PBR can be divided into 2 parts:

Policy routing rules: a set of rules that select packets based on various information (source address, input interface…) and apply a routing action, typically perform a route lookup in a routing table.
Routing tables: Linux creates 2 of them by default:
- the local table (table 255), automatically populated by the kernel and containing routes local to the machine (local loopback, broadcast, local addresses…)
- the main table (table 254), where all routes are added by default
Additionally, a user may create and populate custom tables.

With Linux - Fast Path Synchronization, PBR may be managed via standard Linux commands:

configuration of PBR rules
configuration of routes in various routing tables

Rules¶

The routing policy database is the first stage of route lookup. An ordered list of PBR rules exists in each VRF and for each IP version.

The following IPv4 rules are created by default:

# ip rule show
     from all lookup local
 from all lookup main
 from all lookup default

These rules mean:

for all packets, do a LPM lookup in the local table. If a route is found, return it, otherwise jump to next rule
for all packets, do a LPM lookup in the main table. If a route is found, return it, otherwise jump to next rule
for all packets, do a LPM lookup in the default table. If a route is found, return it, otherwise return an error

The following IPv6 rules are created by default:

# ip -6 rule show
0:      from all lookup local
32766:  from all lookup main

These rules mean:

for all packets, do a LPM lookup in the local table. If a route is found, return it, otherwise jump to next rule
for all packets, do a LPM lookup in the main table. If a route is found, return it, otherwise return an error

All these rules may be deleted, and new rules may be added.

The Linux commands below enable to manage PBR rules. All these commands will be synchronized to the fast path.

Rules are managed by the ip rule command:

# ip rule help
Usage: ip rule [ list | add | del | flush | save ] SELECTOR ACTION
       ip rule restore
SELECTOR := [ not ] [ from PREFIX ] [ to PREFIX ] [ tos TOS ]
              ~~~     ~~~~            ~~
            [ fwmark FWMARK[/MASK] ]
              ~~~~~~
            [ iif STRING ] [ oif STRING ] [ pref NUMBER ]
              ~~~                           ~~~~
ACTION := [ table TABLE_ID ]
            ~~~~~
          [ realms [SRCREALM/]DSTREALM ]
          [ goto NUMBER ]
          SUPPRESSOR
SUPPRESSOR := [ suppress_prefixlength NUMBER ]
              [ suppress_ifgroup DEVGROUP ]
TABLE_ID := [ local | main | default | NUMBER ]

The underlined selector and action types are supported. Some of them have aliases:

selector types:
- not: flag that inverts the match result
- from: source address or prefix
- to: destination address or prefix
- fwmark: mark of the packet
- iif: input interface (a.k.a. dev)
- pref: priority of the rule (a.k.a. priority or order)
action types:
- table: LPM lookup in a routing table (a.k.a. lookup)

Adding a rule¶

iproute2 enables to add new rules. Fast Path Policy-Based Routing supports selector fields priority, from, to, iif, fwmark and not, and action lookup. All selector fields are optional.

The routing table in the lookup action may be referenced by its 32 bit ID or by a literal name listed in /etc/iproute2/rt_tables.

Example

Examples of IPv4 rules:

# ip rule add priority 100 from 10.100.0.1 lookup 180
# ip rule add priority 150 from 10.100.0.0/24 lookup main
# ip rule add priority 170 iif eth1 lookup 200
# ip rule add priority 170 from 10.50.0.0/16 iif eth2 lookup 210
# ip rule add priority 190 not iif vxlan1 lookup 220

Examples of IPv6 rules:

# ip -6 rule add priority 100 from 3ffe:304:124:100::1 lookup 180
# ip -6 rule add priority 150 from 3ffe:304:124:100::/64 lookup main
# ip -6 rule add priority 170 iif eth1 lookup 200
# ip -6 rule add priority 170 from 3ffe:304:124:50::/64 iif eth2 lookup 210
# ip -6 rule add priority 190 not iif vxlan1 lookup 220

Note

There is no unique identifier for a rule. Several rules may share the same priority and duplicate rules are even supported.

It is recommended to assign a unique priority to each rule and use it as an identifier. This precaution is important if one wants to later delete the right PBR rule.

Note

The interface specified in the iif field is a physical or logical interface identified by its ifname. It does not need to exist at the time the rule is created. It may appear later, be deleted, be created again with a different ifindex. Linux and the fast path will properly recognize the interface and apply the rules as expected.

Displaying rules in the Linux kernel¶

Example

Display IPv4 rules:

# ip rule show
     from all lookup local
   from 10.100.0.1 lookup 180
   from 10.100.0.0/24 lookup main
   from all iif eth1 lookup 200
   from 10.50.0.0/16 iif eth2 lookup 210
   not from all iif vxlan1 lookup 220
 from all lookup main
 from all lookup default

Display IPv6 rules:

# ip -6 rule show
     from all lookup local
   from 3ffe:304:124:100::1 lookup 180
   from 3ffe:304:124:100::/64 lookup main
   from all iif eth1 lookup 200
   from 3ffe:304:124:50::/64 iif eth2 lookup 210
   not from all iif vxlan1 lookup 220
 from all lookup main

Displaying rules in the fast path¶

Example

Display IPv4 rules:

<fp-0> pbr4-rule
from all lookup 255
from 10.100.0.1 lookup 180
from 10.100.0.0/24 lookup 254
from all iif eth1 lookup 200
from 10.50.0.0/16 iif eth2 lookup 210
not from all iif vxlan1 lookup 220
from all lookup 254
from all lookup 253

Display IPv6 rules:

<fp-0> pbr6-rule
from all lookup 255
from 3ffe:304:124:100::1 lookup 180
from 3ffe:304:124:100::/64 lookup 254
from all lookup 200
from 3ffe:304:124:50::/64 lookup 210
not from all lookup 220
from all lookup 254

Deleting a rule¶

Note

As stated in Adding a rule, there is no unique identifier for a rule. The Linux implementation of PBR rule deletion is peculiar, even puzzling. For example, if the current rules are:

100: from 10.100.0.1 iif eth1 lookup 180
110: from all iif eth1 lookup 200

Then, the following command will delete the first rule, which is probably not what the user expected:

ip rule delete from all iif eth1

This is the reason why it is recommended to assign a different priority to each rule. It can then be deleted by specifying its priority:

ip rule delete priority 110

Anyhow, the fast path manager will always delete the same PBR rule as the kernel.

Example

Delete the IPv4 rule with priority 150:

# ip rule del priority 150

Delete the IPv6 rule with priority 150:

# ip -6 rule del priority 150

Note

All rules may be deleted, including default rules. It is even possible, though not recommended, to delete rule 0 (from all lookup local)

# ip rule del priority 0

Default rules may be manually restored, like any PBR rule:

# ip rule add priority 0 lookup local
# ip rule add priority 32766 lookup main
# ip rule add priority 32767 lookup default

Flushing rules¶

iproute2 enables to flush all rules in one command. It will delete all rules, except rules with priority 0.

Example

Flush all IPv4 rules, except rules with priority 0:

# ip rule flush

Delete all IPv6 rules, except rules with priority 0:

# ip -6 rule flush

Behavior for unsupported rules¶

Some rules maybe be unsupported by fast path, either because of unsupported match or unsupported action. In case of unsupported match, a packet hitting this rule will automatically be sent as exception to be processed by Linux kernel. In case of unsupported action, a packet hitting the rule will be sent as exception only if there is a match. The unsupported parts are displayed at fast path level.

<fp-0> pbr4-rule
from all lookup 255
from 1.0.0.0/24 [unsup. action]
[unsup. match] from 1.0.0.0/24 lookup 29
[unsup. match] from all [unsup. action]
from all lookup 254
from all lookup 253

<fp-0> pbr6-rule
from all lookup 255
from 3ffe::/64 [unsup. action]
from 3ffe::/64 [unsup. action]
from 3ffe::/64 lookup 29
[unsup. match] from all lookup 7
[unsup. match] from all [unsup. action]
from 3ffe::/64 lookup 254
from all lookup 254

Routes¶

The Linux commands below enable to manage routes in various routing tables. All these commands will be synchronized to the fast path.

Note

Routes from Linux local table are not synchronized to the fast path. The fast path manager populates the fast path local table with special routes mainly deduced from addresses configured on the machine.

Adding a route into a specific routing table¶

By default, iproute2 configures routes in the main table (table 254). But other routing tables may be configured, such as the local table (table 255), the default table (table 253) or custom tables created and populated by the user.

A routing table may be referenced by its 32 bit ID or by a literal name listed in /etc/iproute2/rt_tables. This file may be updated to add new table names.

A route may be added into a specific table by specifying table TABLEID in the ip route command, were TABLEID is its numerical ID or a literal name. Adding a route to a nonexistent table will create it.

Linux routing tables and routes are then synchronized in the fast path (except the local table).

Example

Here, we add IPv4 routes into the main table, then in table 200. This will create the IPv4 table 200:

# ip link set eth2 up
# ip addr add 10.23.1.101/24 dev eth2

# ip route add 10.24.1.0/24 via 10.23.1.201
# ip route add 10.25.1.0/24 via 10.23.1.201 table 200

Here, we add IPv6 routes into the main table, then in table 200. This will create the IPv6 table 200:

# ip link set eth2 up
# ip addr add 3ffe:304:124:2301::101/64 dev eth2

# ip route add 3ffe:304:124:2401::101/64 via 3ffe:304:124:2301::201
# ip route add 3ffe:304:124:2501::101/64 via 3ffe:304:124:2301::201 table 200

Displaying routes in Linux kernel¶

By default, iproute2 displays routes in the main table (table 254):

Example

# ip route show
10.23.1.0/24 dev eth2  proto kernel  scope link  src 10.23.1.101
10.24.1.0/24 via 10.23.1.201 dev eth2

# ip -6 route show
3ffe:304:124:2301::/64 dev eth2  proto kernel  metric 256  pref medium
3ffe:304:124:2401::/64 via 3ffe:304:124:2301::201 dev eth2  metric 1024  pref medium
fe80::/64 dev eth2  proto kernel  metric 256  pref medium

To display routes in a specific table, specify table TABLEID:

# ip route show table 200
10.25.1.0/24 via 10.23.1.201 dev eth2

root@dut-vm:~# ip -6 route show table 200
3ffe:304:124:2501::/64 via 3ffe:304:124:2301::201 dev eth2  metric 1024  pref medium

# ip route show table local
local 10.0.2.15 dev mgmt0  proto kernel  scope host  src 10.0.2.15
broadcast 10.23.1.0 dev eth2  proto kernel  scope link  src 10.23.1.101
local 10.23.1.101 dev eth2  proto kernel  scope host  src 10.23.1.101
broadcast 10.23.1.255 dev eth2  proto kernel  scope link  src 10.23.1.101
broadcast 127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1

# ip -6 route show table local
local ::1 dev lo  proto none  metric 0  pref medium
local 3ffe:304:124:2301:: dev lo  proto none  metric 0  pref medium
local 3ffe:304:124:2301::101 dev lo  proto none  metric 0  pref medium
local fe80:: dev lo  proto none  metric 0  pref medium
local fe80:: dev lo  proto none  metric 0  pref medium
local fe80::200:46ff:fe50:4e00 dev lo  proto none  metric 0  pref medium
local fe80::dced:2ff:fe1c:341a dev lo  proto none  metric 0  pref medium
ff00::/8 dev eth2  metric 256  pref medium

To display routes in all tables, specify table all:

# ip route show table all
10.25.1.0/24 via 10.23.1.201 dev eth2  table 200
10.23.1.0/24 dev eth2  proto kernel  scope link  src 10.23.1.101
10.24.1.0/24 via 10.23.1.201 dev eth2
local 10.0.2.15 dev mgmt0  table local  proto kernel  scope host  src 10.0.2.15
broadcast 10.23.1.0 dev eth2  table local  proto kernel  scope link  src 10.23.1.101
local 10.23.1.101 dev eth2  table local  proto kernel  scope host  src 10.23.1.101
broadcast 10.23.1.255 dev eth2  table local  proto kernel  scope link  src 10.23.1.101
broadcast 127.0.0.0 dev lo  table local  proto kernel  scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo  table local  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  table local  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  table local  proto kernel  scope link  src 127.0.0.1
fe80::/64 dev eth2  proto kernel  metric 256  pref medium
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101 pref medium
local ::1 dev lo  table local  proto none  metric 0  pref medium
local fe80:: dev lo  table local  proto none  metric 0  pref medium
local fe80:: dev lo  table local  proto none  metric 0  pref medium
local fe80::200:46ff:fe50:4e00 dev lo  table local  proto none  metric 0  pref medium
local fe80::dced:2ff:fe1c:341a dev lo  table local  proto none  metric 0  pref medium
ff00::/8 dev eth2  table local  metric 256  pref medium
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101 pref medium

# ip -6 route show table all
3ffe:304:124:2501::/64 via 3ffe:304:124:2301::201 dev eth2  table 200  metric 1024  pref medium
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101 pref medium
3ffe:304:124:2301::/64 dev eth2  proto kernel  metric 256  pref medium
3ffe:304:124:2401::/64 via 3ffe:304:124:2301::201 dev eth2  metric 1024  pref medium
fe80::/64 dev eth2  proto kernel  metric 256  pref medium
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101 pref medium
local ::1 dev lo  table local  proto none  metric 0  pref medium
local 3ffe:304:124:2301:: dev lo  table local  proto none  metric 0  pref medium
local 3ffe:304:124:2301::101 dev lo  table local  proto none  metric 0  pref medium
local fe80:: dev lo  table local  proto none  metric 0  pref medium
local fe80:: dev lo  table local  proto none  metric 0  pref medium
local fe80::200:46ff:fe50:4e00 dev lo  table local  proto none  metric 0  pref medium
local fe80::dced:2ff:fe1c:341a dev lo  table local  proto none  metric 0  pref medium
ff00::/8 dev eth2  table local  metric 256  pref medium
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101 pref medium

Displaying routes in the fast path¶

The fp-cli route4 command displays IPv4 routes in all tables, or in a specific table if table TABLEID is specified.

Example

<fp-0> route4
# - Preferred, * - Active, > - selected
(254) 10.24.1.0/24  [08]  ROUTE gw 10.23.1.201 via eth2-vr0 (13)
(200) 10.25.1.0/24  [08]  ROUTE gw 10.23.1.201 via eth2-vr0 (14)

<fp-0> route4 table 200
# - Preferred, * - Active, > - selected
(200) 10.25.1.0/24  [08]  ROUTE gw 10.23.1.201 via eth2-vr0 (14)

<fp-0> route4 type all
# - Preferred, * - Active, > - selected
(255) 0.0.0.0/32  [5001]  LOCAL (1)
(255) 10.0.2.15/32  [01]  ADDRESS via mgmt0-vr0 (5)
(255) 10.23.1.101/32  [05]  ADDRESS via eth2-vr0 (10)
(255) 127.0.0.0/8  [5002]  BLACKHOLE (4)
(255) 224.0.0.0/4  [5001]  LOCAL (3)
(255) 255.255.255.255/32  [5001]  LOCAL (2)
(254) 10.23.1.0/24  [06]  CONNECTED via eth2-vr0 (11)
(254) 10.24.1.0/24  [08]  ROUTE gw 10.23.1.201 via eth2-vr0 (13)
(200) 10.25.1.0/24  [08]  ROUTE gw 10.23.1.201 via eth2-vr0 (14)

The fp-cli route6 command displays IPv6 routes in all tables, or in a specific table if table TABLEID is specified:

<fp-0> route6
# - Preferred, * - Active, > - selected
(254) 3ffe:304:124:2401::/64  [03]  ROUTE gw 3ffe:304:124:2301::201 via eth2-vr0 (6)
(200) 3ffe:304:124:2501::/64  [03]  ROUTE gw 3ffe:304:124:2301::201 via eth2-vr0 (7)

<fp-0> route6 type all table 200
# - Preferred, * - Active, > - selected
(200) 3ffe:304:124:2501::/64  [03]  ROUTE gw 3ffe:304:124:2301::201 via eth2-vr0 (7)

<fp-0> route6 type all
# - Preferred, * - Active, > - selected
(255) ::/80  [5002]  BLACKHOLE (3)
(255) 3ffe:304:124:2301::101/128  [02]  ADDRESS via eth2-vr0 (5)
(255) fe80::/10  [5001]  LOCAL (1)
(255) ff00::/8  [5001]  LOCAL (2)
(254) 3ffe:304:124:2301::/64  [01]  CONNECTED via eth2-vr0 (4)
(254) 3ffe:304:124:2401::/64  [03]  ROUTE gw 3ffe:304:124:2301::201 via eth2-vr0 (6)
(200) 3ffe:304:124:2501::/64  [03]  ROUTE gw 3ffe:304:124:2301::201 via eth2-vr0 (7)

Note

The fast path maintains an extra table (table 0), that holds neighbor entries (ARP and NDP entries) in the form of /32 or /128 routes of type NEIGH. These entries are hidden by default, except if you request to display routes of type neigh or all.

Deleting a route from a specific routing table¶

By default, iproute2 deletes routes in the main table (table 254). But another routing table may be specified by adding table TABLEID to the command.

Example

# ip route del 10.24.1.0/24

# ip route del 10.25.1.0/24 table 200

# ip route del 3ffe:304:124:2401::101/64

# ip route del 3ffe:304:124:2501::101/64 table 200

Flushing routes from a specific routing table¶

iproute2 enables to flush all routes in one command, with mandatory parameters.

Among parameters, a table ID may be specified, all standing for all tables.

Example

Flush all IPv4 rules in table 200:

# ip route flush table 200

Flush all IPv4 routes in table main:

# ip route flush table main

Flush all IPv6 routes in table 200:

# ip -6 route flush table 200

Flush all IPv6 routes in table main:

# ip -6 route flush table main

Note

It is of course not recommended to flush the local table, which is automatically populated by the kernel. In case you did it though, the routes may be restored by setting all interfaces down then up again, including lo.

# for i in /sys/class/net/*; do \
  DEV=$(basename $i); \
  ip link set $DEV down; \
  ip link set $DEV up; \
done

Providing options¶

--max-rules¶

The maximum number of PBR rules can be configured using the variable FP_OPTIONS

Example

FP_OPTIONS="--mod-opt=pbr:--max-rules=512"

At least 5 |pbr| rules (3 for IPv4 and 2 for IPv6) are created per |vr|

Default value
   1024
Memory footprint per |pbr| rule
   100 B
Range
   0 .. 400K

Note

See Fast Path Capabilities documentation for impact of the available memory on the default value of configurable capabilities