Scheduling

Scheduling is applied to egress traffic on physical interfaces.

Outgoing packets are queued before actually being sent on the interface. The scheduling consists in selecting in which order and at which time packets are dequeued and submitted to the network adapter.

../../../_images/scheduler.svg

The order and time depend on the chosen scheduling algorithm and its configuration.

The following classful algorithms are supported:

  • PQ

  • PB-DWRR

  • HTB

Packets are dispatched into different queues according to the class of service they belong to. Then, the scheduling algorithm and its configuration determine from which queue and at which moment the next packet is dequeued.

Each queue has several parameters:

  • The size of the queue defines how many packets can be stored in the queue. Longer queues mean longer delays.

  • The list of traffic classes that are submitted to the queue. Traffic classes are defined in the firewall section.

  • An input policer to rate limit incoming traffic.

  • An output shaper to rate limit outgoing traffic (except for HTB algorithm)

  • Specific parameters related to the selected queueing processing (PQ, PB-DWRR or HTB)

The following classless algorithm is supported:

  • SFQ

Contrary to classful algorithms, classless algorithms do not classify traffic into classes of service.

Traffic classes

A class specifies a set of traffic flows, identified by their mark and/or the value of the CP flag.

A class is attached to the queue in which traffic flows will be scheduled. One or more classes may be attached to the same queue.

The critical control plane traffic flag

The CP flag is a flag set on critical packets sent by the control plane, i.e. the same traffic as the one protected by the Control Plane Protection feature, as described in the control plane protection.

This flag is automatically set when the Linux kernel outputs such a packet. It can be matched by the QoS classification stage.

Flow matching and marking

Classes are defined by the mark of the packets and/or the value of the CP flag. Packet marking is done before QoS processing and must be configured at the IP packet filtering level.

QoS classification is usually based on the DSCP field of the IP header. In practice, this field is set for incoming packets by the border routers of a QoS network, allowing core routers to work with it without giving up too much processing power.

This example shows how to mark packets based on DSCP field:

vsr running config# / vrf main
vsr running vrf main# firewall ipv4
vsr running ipv4# mangle
vsr running mangle# prerouting
vsr running prerouting# rule 1000 dscp af41 inbound-interface eth1 action mark 0x23

This example shows how to mark packets based on DSCP field, and change the DSCP value. Only one action is allowed per rule, therefore this requires to either use a chain or repeat the same rule with different actions.

vsr running config# / vrf main
vsr running vrf main# firewall ipv4
vsr running ipv4# mangle
vsr running mangle# chain g3 rule 1 action mark 0x3
vsr running mangle# chain g3 rule 2 action dscp af41
vsr running mangle# chain g3 policy return
vsr running mangle# prerouting
vsr running prerouting# rule 3000 dscp af42 inbound-interface eth1 action chain g3

Here, the chain policy is return, which means that matching packets will be processed by the remaining rules after being marked. Use policy accept if marked packets should not be processed further by the firewall.

Note

Refer to the mark action of the rule command in the command reference.

Classification

Classes are created in the global qos context with the class command. They can then be referenced by any scheduler.

A class is defined by a packet mark and/or the value of the CP flag.

Enter the global qos context and create classes:

vsr running config# / qos
vsr running qos# class voip
vsr running class voip#! mark 0x1
vsr running class voip# ..
vsr running qos# class mail
vsr running class mail#! mark 0x2
vsr running class mail# ..
vsr running qos#

By default, all bits of the mark are used to specify classes. Therefore, up to 2**32 different marks are supported. It is possible to specify which bits are used for QoS in order to use the mark for different purposes. In this case, the number of different marks is 2**n where n is the number of bits reserved for the QoS in the mark.

To modify the mask used by the QoS enter the global qos context and edit the class-mask:

vsr running config# / qos
vsr running qos# class-mask 0xff

With this configuration, the first 8 bits of the mark are used to specify classes for QoS, so that 256 different marks can be used.

Note

The class-mask bitmask indicates which bits of the packet and class marks are taken into account. Other bits are ignored.

A packet belongs to a class if:

(packet-mark XOR class-mark) AND class-mask = 0

For example, with the following configuration:

/ qos class-mark 0xff
/ qos class class42 mark 0x42
/ qos class class542 mark 0x542

the 2 classes class42 and class542 match the same packets, those with a mark whose last byte is 0x42; for example packets with marks 0x42, 0x542, 0xff42 or 0x424242. Which class these packets will be assigned is undefined.

Therefore, care must be taken to avoid defining two classes for which (class-mark AND class-mask) equals the same value.

A class can also be configured to match (or to not match) the output critical control plane traffic.

The following class matches all critical control plane traffic:

vsr running config# / qos
vsr running qos# class control
vsr running class control#! cp true
vsr running class control# ..
vsr running qos#

The following class matches all critical control plane traffic with mark 0x30:

vsr running qos# / qos
vsr running qos# class control30
vsr running class control30#! cp true
vsr running class control30# mark 0x30
vsr running class control30# ..
vsr running qos#

The following class matches traffic with mark 0x30, except critical control plane traffic:

vsr running qos# / qos
vsr running qos# class nocontrol30
vsr running class nocontrol30#! cp false
vsr running class nocontrol30#! mark 0x30
vsr running class nocontrol30# ..

Note

At most one mark and the value of the CP flag may be specified in a class. A packet belongs to a class if it matches all parameters specified in the class. The following combinations are supported, and evaluated in this order:

  • cp true + mark

  • cp true

  • cp false + mark

  • mark

Note

A packet that does not belong to any class or whose class is not bound to any queue will be submitted to the last queue.

Scheduling algorithms

PQ

When the scheduling algorithm is PQ, N queues are defined. Each queue has a different priority. The first queue has the highest priority, the last one has the lowest. Queues are served by order of priority: the scheduler first takes packets from the highest priority queue and submits them to the network hardware. When the queue is empty, it starts processing the next queue and so on.

PB-DWRR

When the scheduling algorithm is PB-DWRR, N queues and two priority levels are defined: high and low.

Among the N queues, one has the high priority, and the N-1 others the low priority. Each low priority queue has a quantum that defines the share of the remaining bandwidth it will receive.

The high priority queue is served first. Once it is empty, other queues are served in a round robin fashion: the scheduler performs DWRR rounds between low priority queues. At each round, it checks each queue in sequence and enables it to send bytes up to its quantum. Then it serves the next queue, and so on.

When queue priorities are not set, all queues are served according to their quantum. This is the simple DWRR mode, which prevents starvation.

HTB

When the scheduling algorithm is HTB, the queues are organized in a tree structure hierarchy, that determines how available bandwidth is distributed among queues.

Only leaf queues actually queue packets. Inner (non-leaf) queues are virtual queues that only maintain bandwidth credits, in order to hierarchically redistribute available bandwidth.

The unique root queue determines the total bandwidth of the interface.

To create the tree structure hierarchy, each inner queue specifies a list of child queues, that may themselves be inner queues or leaf queues.

Each queue (inner or leaf) is assigned two key parameters:

  • bandwidth: minimum guaranteed rate/bandwidth of the queue (given sufficient demand)

  • ceiling: maximum authorized rate/bandwidth of the queue

  • priority: high priority queues (smaller values) are served before low priority queues (bigger values)

First, leaf queues with packets are served up to their allocated guaranteed rate (bandwidth). Once all queues with sufficient demand have received at least their allocated guaranteed bandwidth, the remaining bandwidth is redistributed among demanding queues up to their maximum bandwidth (ceiling), by borrowing bandwidth from their parent queue, following rules that take the queues priority, bandwidth and ceiling into account.

The algorithm for borrowing available bandwidth above the guaranteed rate is applied recursively from leaf queues up to the root queue.

SFQ

SFQ is a fair queueing algorithm. Its goal is to fairly share the bandwidth between several flows. The packets are dispatched into flows, based on a hash on the packet IP information. The traffic is then scheduled in a round robin fashion.

The parameters of an SFQ scheduler are:

  • flows: number of flows in which the traffic is distributed

  • limit: total maximum number of packets that can be enqueued. For example, by default, we have 1024 flows and a value for limit equal to 10240 packets. We will get approximately 10 packets per flow. If the limit is exceeded, the flow with the greatest number of enqueued packets is identified, and half of its packets are dropped.

  • quantum: amount of bytes a flow is allowed to dequeue before dequeueing the next flow

Scheduler templates

Scheduler templates are created in the global qos context with the scheduler command. They can then be referenced by a physical interface for egress.

Enter the global qos context and create a scheduler using PQ:

vsr running config# / qos
vsr running qos# scheduler sched1
vsr running scheduler sched1#! pq
vsr running pq#! nb-queue 3
vsr running pq# queue 1
vsr running queue 1# class control
vsr running queue 1# class voip
vsr running queue 1# shaper shaper1
vsr running queue 1# ..
vsr running pq# queue 2
vsr running queue 2# class mail
vsr running queue 2# .. .. ..

Enter the global qos context and create a scheduler using PB-DWRR:

vsr running config# / qos
vsr running qos# scheduler sched2
vsr running scheduler sched2#! pb-dwrr
vsr running pb-dwrr#! nb-queue 3
vsr running pb-dwrr# queue 1
vsr running queue 1# class control
vsr running queue 1# class voip
vsr running queue 1# shaper shaper1
vsr running queue 1# priority high
vsr running queue 1# ..
vsr running pb-dwrr# queue 2
vsr running queue 2# class mail
vsr running queue 2# quantum 3000
vsr running queue 2# .. .. ..

Note

To send several types of traffic flows to the same queue, you can define a class for each traffic flow, and attach all classes to the queue. (e.g. classes control and voip attached to queue 1 in examples above).

Enter the global qos context and create a scheduler using HTB:

vsr running config# / qos
vsr running qos# scheduler sched3
vsr running scheduler sched3#! htb
vsr running htb# divisor 20
vsr running htb# queue 1
vsr running queue 1#! bandwidth 2000000000
vsr running queue 1# burst 150000000
vsr running queue 1# ceiling 1750000000
vsr running queue 1# ceiling-burst 21875000
vsr running queue 1# priority 1
vsr running queue 1# size 256
vsr running queue 1# quantum 100000000
vsr running queue 1# child-queue 2
vsr running queue 1# child-queue 3
vsr running queue 1# ..
vsr running htb# queue 2
vsr running queue 2#! bandwidth 3000000000
vsr running queue 2# priority 2
vsr running queue 2# ..
vsr running htb# queue 3
vsr running queue 3#! bandwidth 4000000000
vsr running queue 3# priority 3
vsr running queue 3# policer pol1
vsr running queue 3#! .. .. ..
vsr running qos#! policer pol1
vsr running policer pol1#! bandwidth 6000000000
vsr running policer pol1# burst 1500
vsr running policer pol1# excess-bandwidth 500000000
vsr running policer pol1# excess-burst 1500

Note

A scheduler using HTB must only have one root queue.

Note

The bandwidth value of a queue should be greater than the sum of all the bandwidth values of the sub-tree queues underneath. The same rule applies for the ceiling value.

Note

A shaper can’t be added to a queue since the rate is instead limited by the ceiling value.

Enter the global qos context and create a scheduler using SFQ:

vsr running config# / qos
vsr running qos# scheduler sched4
vsr running scheduler sched4#! sfq
vsr running sfq# flows 2048
vsr running sfq# limit 10000
vsr running sfq# quantum 3000

Review the QoS configuration:

vsr running config# / qos
vsr running qos# show config
qos
    shaper shaper1
        bandwidth 1G
        burst 2K
        layer1-overhead 0
        queue-size 128
        ..
    shaper shaper2
       bandwidth 10G
       burst 48K
       layer1-overhead 24
       queue-size 256
       ..
    scheduler sched1
        pq
            nb-queue 3
            queue 1
                size 256
                shaper shaper1
                class control
                class voip
                ..
            queue 2
                size 256
                class mail
                ..
            ..
        ..
    scheduler sched2
        pb-dwrr
            nb-queue 3
            queue 1
                size 256
                shaper shaper1
                class control
                class voip
                quantum 1500
                priority high
                ..
            queue 2
                size 256
                class mail
                quantum 3000
                priority low
                ..
            ..
        ..
    class-mask 0xff
    class voip
        mark 0x1
        ..
    class mail
        mark 0x2
        ..
    class control
        cp true
        ..
    class control30
        mark 0x30
        cp true
        ..
    class nocontrol30
        mark 0x30
        cp false
        ..
    ..

The same settings can be applied using the following NETCONF XML configuration:

vsr running config# / qos
vsr running config qos# show config xml absolute
<config xmlns="urn:6wind:vrouter">
  <qos xmlns="urn:6wind:vrouter/qos">
    <class-mask>0xff</class-mask>
    <class>
      <name>voip</name>
      <mark>0x1</mark>
    </class>
    <class>
      <name>mail</name>
      <mark>0x2</mark>
    </class>
    <class>
      <name>control</name>
      <cp>true</cp>
    </class>
    <class>
      <name>control30</name>
      <cp>true</cp>
      <mark>0x30</mark>
    </class>
    <class>
      <name>nocontrol30</name>
      <cp>false</cp>
      <mark>0x30</mark>
    </class>
    <shaper>
      <name>shaper1</name>
      <burst>2000</burst>
      <layer1-overhead>0</layer1-overhead>
      <queue-size>128</queue-size>
      <bandwidth>1000000000</bandwidth>
    </shaper>
    <shaper>
      <name>shaper2</name>
      <burst>48000</burst>
      <layer1-overhead>24</layer1-overhead>
      <queue-size>256</queue-size>
      <bandwidth>10000000000</bandwidth>
    </shaper>
    <scheduler>
      <name>sched1</name>
      <pq>
        <nb-queue>3</nb-queue>
        <queue>
          <id>1</id>
          <size>256</size>
          <class>
            <name>mail</name>
          </class>
        </queue>
      </pq>
    </scheduler>
    <scheduler>
      <name>sched2</name>
      <pb-dwrr>
        <nb-queue>3</nb-queue>
        <queue>
          <id>1</id>
          <size>256</size>
          <quantum>1500</quantum>
          <priority>high</priority>
          <class>
            <name>control</name>
          </class>
          <class>
            <name>voip</name>
          </class>
          <shaper>shaper1</shaper>
        </queue>
        <queue>
          <id>2</id>
          <size>256</size>
          <quantum>3000</quantum>
          <priority>low</priority>
          <class>
            <name>mail</name>
          </class>
        </queue>
      </pb-dwrr>
    </scheduler>
  </qos>
</config>

Configuring a scheduler on an interface

Schedulers are configured in the qos context of physical interfaces.

Enter the qos context of the eth0 physical interface:

vsr running config# vrf main
vsr running vrf main# interface physical eth0
vsr running physical eth0# qos

Configure sched1 as the scheduler for egress traffic:

vsr running qos# egress scheduler sched1
vsr running qos# egress rate-limit shaper shaper2
vsr running qos# ..
vsr running physical eth0#

Note

When a scheduler is configured on an interface, it is mandatory to also configure a rate limit shaper on the same interface.

Review eth0 configuration:

vsr running physical eth0# show config nodefault
physical eth0
    (...)
    qos
        egress
            rate-limit
                shaper shaper2
                ..
            scheduler sched1
            ..
        ..
    ..

Note

When the scheduler applied on an interface is configured with the algorithm HTB, we can’t apply a rate-limit shaper in egress of the same interface. In fact, there is no need for a shaper, because the whole traffic bandwidth is limited by the ceiling attribute of the root queue of the hierarchy.

Commit the configuration:

vsr running physical eth0# commit
Configuration committed.
vsr running physical eth0# ..
vsr running config#

Review the QoS state of the interface:

qos
    egress
        rate-limit
            shaper
                bandwidth 10G
                burst 48K
                layer1-overhead 24
                queue-size 256
                stats
                    pass-packets 0
                    drop-packets 0
                    ..
                ..
            ..
        scheduler
            pq
                nb-queue 3
                queue 1
                    size 256
                    shaper
                        bandwidth 1G
                        burst 2K
                        stats
                            pass-packets 0
                            drop-packets 0
                            ..
                        ..
                    class cp
                        stats
                            match-packets 0
                            ..
                        ..
                    class 0x00000001
                        stats
                            match-packets 0
                            ..
                        ..
                    stats
                        enqueue-packets 0
                        xmit-packets 0
                        drop-queue-full 0
                        ..
                    ..
                queue 2
                    size 256
                    class 0x00000002
                        stats
                            match-packets 0
                            ..
                        ..
                    stats
                        enqueue-packets 0
                        xmit-packets 0
                        drop-queue-full 0
                        ..
                    ..
                queue 3
                    size 256
                    class default
                        stats
                            match-packets 0
                            ..
                        ..
                    stats
                        enqueue-packets 0
                        xmit-packets 0
                        drop-queue-full 0
                        ..
                    ..
                ..
            ..
        ..
    ..

The same settings can be applied using the following NETCONF XML configuration:

vsr running config# show config xml absolute vrf main interface physical eth0
<config xmlns="urn:6wind:vrouter">
  <vrf>
    <name>main</name>
    <interface xmlns="urn:6wind:vrouter/interface">
      <physical>
        <name>eth0</name>
        (...)
        <qos>
          <egress>
            <scheduler>sched1</scheduler>
            <rate-limit>
              <shaper>shaper2</shaper>
            </rate-limit>
          </egress>
        </qos>
      </physical>
    </interface>
  </vrf>
</config>

Chaining a scheduler

Schedulers can be attached to a physical interface but also to the queue of another scheduler. Such a scheduler is called a child-scheduler of this queue. A configuration with a child scheduler is called chaining. The child scheduler will be created when the parent scheduler is attached to a physical interface.

In the example below, the parent scheduler uses a PQ algorithm and the queues 1 and 3 have a child scheduler using an SFQ algorithm.

vsr running config# / qos
vsr running qos# scheduler sched-pq
vsr running scheduler sched-pq#! pq
vsr running pq#! nb-queue 3
vsr running pq# queue 1
vsr running queue 1# class control
vsr running queue 1#! class voip
vsr running queue 1#! shaper shaper1
vsr running queue 1#! priority 1
vsr running queue 1#! child-scheduler sched-sfq1
vsr running queue 1#! ..
vsr running pq#! queue 2
vsr running queue 2#! class mail
vsr running queue 2#! size 1024
vsr running queue 2#! priority 2
vsr running queue 2#! ..
vsr running pq#! queue 3
vsr running queue 3#! class video
vsr running queue 3#! priority 3
vsr running queue 3#! child-scheduler sched-sfq2
vsr running queue 3#! .. .. ..
vsr running qos#! scheduler sched-sfq1
vsr running scheduler sched-sfq1#! sfq
vsr running sfq#! flows 2048
vsr running sfq#! limit 10000
vsr running sfq#! quantum 3000
vsr running sfq#! .. ..
vsr running qos#! scheduler sched-sfq2
vsr running scheduler sched-sfq2#! sfq
vsr running sfq#! flows 2048
vsr running sfq#! limit 10000
vsr running sfq#! quantum 3000
vsr running sfq#! .. ..
vsr running qos#! class control mark 0x2
vsr running qos#! class voip mark 0x4
vsr running qos#! class video mark 0x5
vsr running qos#! class mail mark 0x10
vsr running qos#!
vsr running qos#! shaper shaper1
vsr running shaper shaper1#! bandwidth 10G
vsr running shaper shaper1# burst 4000
vsr running shaper shaper1# layer1-overhead 24

Apply the parent scheduler to a physical interface:

vsr running shaper shaper1# / vrf main
vsr running vrf main# interface physical ntfp1
vsr running physical ntfp1# qos
vsr running qos# egress
vsr running egress# rate-limit
vsr running rate-limit# shaper shaper1
vsr running rate-limit# ..
vsr running egress# scheduler sched-pq

Note

For now, a child scheduler can only be set with a classless algorithm SFQ.

Note

For a scheduler using HTB, a child scheduler can only be attached to its leaf queues.

Shaping the output while protecting critical control plane traffic

When configuring a simple shaper on the output of an interface that is constantly fed with traffic over the limit, a part of the traffic is necessarily dropped. There is a risk that critical control plane traffic be dropped.

A simple solution is to configure a PQ scheduler with 2 queues, one for the critical control plane traffic, the other for the rest of the traffic.

In this example, we configure an output shaper at 100Mbps, and the 2 queue PQ scheduler:

Configure the shaper wanshaper:

vsr running config# / qos
vsr running qos# shaper wanshaper
vsr running shaper wanshaper#! bandwidth 100M
vsr running shaper wanshaper# burst 125K
vsr running shaper wanshaper# ..
vsr running qos#

Configure the class for critical control plane traffic control:

vsr running qos# class control
vsr running class control#! cp true
vsr running class control# ..
vsr running qos#

Configure the scheduler wansched, and attach class control to the high priority queue:

vsr running qos# scheduler wansched
vsr running scheduler wansched#! pq
vsr running pq#! nb-queue 2
vsr running pq# queue 1
vsr running queue 1# class control
vsr running queue 1# ..
vsr running pq# ..
vsr running scheduler wansched# ..
vsr running qos#

Attach the shaper and scheduler to the output interface:

vsr running qos# / vrf main
vsr running vrf main# interface physical eth0
vsr running physical eth0# qos
vsr running qos# egress
vsr running egress# rate-limit
vsr running rate-limit# shaper wanshaper
vsr running rate-limit# ..
vsr running egress# scheduler wansched
vsr running egress# ..
vsr running qos# ..
vsr running physical eth0# ..

Review eth0 configuration:

vsr running physical eth0# show config nodefault
physical eth0
    (...)
    qos
        egress
            rate-limit
                shaper wanshaper
                ..
            scheduler wansched
            ..
        ..
    ..

Commit the configuration:

vsr running physical eth0# commit
Configuration committed.
vsr running physical eth0# /
vsr running config#

Review the QoS state of the interface:

vsr running config# show state vrf main interface physical eth0
qos
    egress
        rate-limit
            shaper
                bandwidth 1M
                burst 1250
                layer1-overhead 0
                queue-size 256
                stats
                    pass-packets 311640
                    drop-packets 329125
                    ..
                ..
            ..
        scheduler
            pq
                nb-queue 2
                queue 1
                    size 256
                    class cp
                        stats
                            match-packets 90
                            ..
                        ..
                    stats
                        enqueue-packets 90
                        xmit-packets 90
                        drop-queue-full 0
                        ..
                    ..
                queue 2
                    size 256
                    class default
                        stats
                            match-packets 640583
                            ..
                        ..
                    stats
                        enqueue-packets 311550
                        xmit-packets 311550
                        drop-queue-full 329125
                        ..
                    ..
                ..
            ..
        ..
    ..
..

The same settings can be applied using the following NETCONF XML configuration:

<config xmlns="urn:6wind:vrouter">
  (...)
  <vrf>
    <name>main</name>
    <interface xmlns="urn:6wind:vrouter/interface">
      <physical>
        <name>eth0</name>
        <port>pci-b0s5</port>
        <qos>
          <egress>
            <rate-limit>
              <shaper>wanshaper</shaper>
            </rate-limit>
            <scheduler>wansched</scheduler>
          </egress>
        </qos>
      </physical>
  </vrf>
  <qos xmlns="urn:6wind:vrouter/qos">
    <class>
      <name>control</name>
      <cp>true</cp>
    </class>
    <shaper>
      <name>wanshaper</name>
      <burst>125000</burst>
      <bandwidth>100000000</bandwidth>
    </shaper>
    <scheduler>
      <name>wansched</name>
      <pq>
        <nb-queue>2</nb-queue>
        <queue>
          <id>1</id>
          <class>
            <name>control</name>
          </class>
        </queue>
      </pq>
    </scheduler>
  </qos>
</config>