Note

CG-NAT requires a CG-NAT Application License.

CG-NAT

CG-NAT, also known as Large Scale NAT (LSN), is an extension of NAT for large scale networks and ISPs.

The key advantages of CG-NAT compared to NAT are:

  • High Transparency: CG-NAT implements multiple advanced features like Endpoint-Independent Mapping, Endpoint-Independent Filtering, address pooling and port parity preservation. These features provide better experience to ‘nated’ users and allow scaling.

  • Fairness and Resource Sharing: CG-NAT provides options to limit the number of connections per user. This ensures that resources are equitably shared between the different users.

  • Optimized Logging system: CG-NAT can generate large amounts of logging data. CG-NAT implements a feature called port block allocation to limit the number of log entries by grouping per port range.

  • Transition between IPv6-only and IPv4-only networks thanks to NAT64, in conjunction to DNS64.

Caution

Stateful IP packet filtering and CG-NAT are exclusive. If CG-NAT is enabled, stateful IP packet filtering must be disabled on ports bound to the fast path.

See also

The CG-NAT command reference and the CG-NAT fast path limits command reference for details.

Configuration

Pool

A CG-NAT pool contains a list of IPv4 addresses used to change the IPv4 or IPv6 source address and port of a packet.

The CG-NAT implements a feature called port block allocation. Each time a new user sent a packet through the CG-NAT router, a block of ports (i.e. a port range) from one of the IPv4 addresses in the pool is allocated to this one. The number of ports given to a user is configurable via the block-size option.

Here is an example of a CG-NAT pool:

vsr running config# vrf main
vsr running vrf main# cg-nat
vsr running cg-nat!# pool mypool
vsr running mypool!# address 192.0.2.33-192.0.2.49
vsr running mypool!# address 192.0.1.0/24
vsr running mypool!# address 192.0.3.1
vsr running mypool!# allocation-mode dynamic-block block-size 512

A pool needs to be associated to a CG-NAT rule, see the next section.

Rule

We have three types of NAT rules:

  • dynamic: private addresses are dynamically mapped to public addresses: for each private address, a public address is dynamically allocated from a pool and associated to the private address.

  • deterministic: each private address is always mapped to the same public address and is assigned a dedicated port pool at the time of configuration

  • static: mapping between private addresses and public ones are static. Public addresses are configured directly in the rule.

static SNAT44

All packets routed through an output interface and matching the filtering criteria defined in a CG-NAT rule are source nated with an ip.

Here is an example of a CG-NAT rule:

vsr running mypool!# ..
vsr running cg-nat#! rule 1
vsr running rule 1#! static-snat44
vsr running static-snat44#! match
vsr running match#! source ipv4-address 100.64.0.0/32
vsr running match#! outbound-interface eth1
vsr running static-snat44#! translate-to
vsr running translate-to#! ipv4-address 192.0.2.1
vsr running translate-to# commit

To display the applied configuration:

vsr running config# show state vrf main cg-nat
cg-nat
    enabled true
    rule 1
        static-snat44
            match
                source
                    ipv4-address 100.64.0.1/32
                    ..
                outbound-interface eth1
                ..
            translate-to
                ipv4-address 192.0.2.1
                ..
            ..
        ..
    logging
        enabled false
        ..
    ..

static DNAT44

All packets received on an input interface and matching the filtering criteria defined in a CG-NAT rule are destination nated with an ip.

Here is an example of a CG-NAT rule:

vsr running mypool!# ..
vsr running cg-nat#! rule 1
vsr running rule 1#! static-dnat44
vsr running static-dnat44#! match
vsr running match#! destination ipv4-address 192.0.2.1/32
vsr running match#! inbound-interface eth1
vsr running static-dnat44#! translate-to
vsr running translate-to#! ipv4-address 100.64.0.1
vsr running translate-to# commit

To display the applied configuration:

vsr running config# show state vrf main cg-nat
cg-nat
    enabled true
    rule 1
        static-dnat44
            match
                destination
                    ipv4-address 192.0.2.1/32
                    ..
                inbound-interface eth1
                ..
            translate-to
                ipv4-address 100.64.0.1
                ..
            ..
        ..
    logging
        enabled false
        ..
    ..

dynamic SNAT44

All packets routed through an output interface and matching the filtering criteria defined in a CG-NAT rule are source nated with an ip dynamically assigned from one CG-NAT pool.

Here is an example of a CG-NAT rule:

vsr running mypool!# ..
vsr running cg-nat#! rule 1
vsr running rule 1#! dynamic-snat44
vsr running dynamic-snat44#! match
vsr running match#! source ipv4-address 100.64.0.0/10
vsr running match#! outbound-interface eth1
vsr running dynamic-snat44#! translate-to
vsr running translate-to#! pool-name mypool
vsr running translate-to# commit

To display the applied configuration:

vsr running config# show state vrf main cg-nat
cg-nat
    enabled true
    pool mypool
        address 192.0.1.1-192.0.1.254
        address 192.0.2.33-192.0.2.49
        address 192.0.3.1
        allocation-mode dynamic-block
            block-size 512
            ..
        port-range
            1024
            65535
            ..
        ..
    rule 1
        dynamic-snat44
            match
                source
                    ipv4-address 100.64.0.0/10
                    ..
                outbound-interface eth1
                ..
            translate-to
                pool-name mypool
                max-conntracks-per-user 0
                max-blocks-per-user 1
                active-block-timeout 0
                user-timeout 120
                port-algo parity
                endpoint-mapping independent
                endpoint-filtering independent
                hairpinning false
                address-pooling paired
                ..
            ..
        ..
    logging
        enabled false
        ..
    ..

The same configuration can be made using this NETCONF XML configuration:

vsr running config# show config xml absolute vrf main cg-nat
<config xmlns="urn:6wind:vrouter">
  <vrf>
    <name>main</name>
    <cg-nat xmlns="urn:6wind:vrouter/cgnat">
      <enabled>true</enabled>
      <logging/>
      <pool>
        <name>mypool</name>
        <port-range/>
        <address>192.0.2.33-192.0.2.49</address>
        <address>192.0.1.0/24</address>
        <address>192.0.3.1</address>
        <allocation-mode>
          <dynamic-block>
            <block-size>512</block-size>
          </dynamic-block>
        </allocation-mode>
      </pool>
      <rule>
        <id>1</id>
        <dynamic-snat44>
          <match>
            <source>
              <ipv4-address>100.64.0.0/10</address>
            </source>
            <outbound-interface>eth1</outbound-interface>
          </match>
          <translate-to>
            <pool-name>mypool</pool-name>
          </translate-to>
        </dynamic-snat44>
      </rule>
    </cg-nat>
  </vrf>
</config>

dynamic port SNAT44

The dynamic-port-snat44 works like dynamic-snat44, except that IP and ports from the CG-NAT pool are directly associated without any port block. It means that all the ports of a public IP is shared with several users.

See the Understanding Port Allocation/dynamic port section for more information about this point.

Here is an example of a dynamic port CG-NAT rule:

vsr running mypool!# allocation-mode dynamic-port
vsr running mypool!# ..
vsr running cg-nat#! rule 1
vsr running rule 1#! dynamic-port-snat44
vsr running dynamic-port-snat44#! match
vsr running match#! source ipv4-address 100.64.0.0/10
vsr running match#! outbound-interface eth1
vsr running dynamic-port-snat44#! translate-to
vsr running translate-to#! pool-name mypool
vsr running translate-to# commit

deterministic SNAT44

The deterministic-snat44 works like dynamic-snat44, except that IP and ports from the CG-NAT pool are associated in a deterministic manner to the user at the time of configuration: the CG-NAT pool is partitioned in chunks of port blocks, so that each user has its own port block. See the Understanding Port Block Allocation/deterministic for more information about this point.

Here is an example of a deterministic CG-NAT rule:

vsr running mypool!# allocation-mode deterministic-block
vsr running mypool!# ..
vsr running cg-nat#! rule 1
vsr running rule 1#! deterministic-snat44
vsr running deterministic-snat44#! match
vsr running match#! source ipv4-address 100.64.0.0/10
vsr running match#! outbound-interface eth1
vsr running deterministic-snat44#! translate-to
vsr running translate-to#! pool-name mypool
vsr running translate-to# commit

dynamic SNAT64

In the case of NAT64, translation between IPv6-only addresses on the private side and IPv4 addresses on the public side is taken care of by mapping the IPv4 public addresses to a special IPv6 prefix that is configured in the translate-to section of the rule.

DNS64 must be configured to translate IPv4 addresses DNS replies to IPv6 ones using this prefix.

The rest of the configuration is similar to dynamic NAT44.

vsr running config# vrf main
vsr running vrf main# cg-nat
vsr running cg-nat#! rule 1
vsr running rule 1#! dynamic-snat64
vsr running dynamic-snat64#! match
vsr running match#! source ipv6-address fd00:100::/64
vsr running match#! outbound-interface eth2
vsr running match#! ..
vsr running dynamic-snat64#! translate-to
vsr running translate-to#! pool-name mypool
vsr running translate-to#! max-blocks-per-user 4
vsr running translate-to#! destination-prefix 64:ff9b::/96
vsr running translate-to# .. .. .. ..
vsr running vrf main# dns
vsr running dns# proxy
vsr running proxy# dns64 64:ff9b::/96
vsr running dns64 64:ff9b::/96# client fd00:100::/64
vsr running dns64 64:ff9b::/96# exclude 64:ff9b::/96
vsr running dns64 64:ff9b::/96# mapped not 10.0.0.0/8 192.168.0.0/16 172.16.0.0/12

The client option can be used to select which client the DNS64 function will apply to. The CPE subnet in the CG-NAT rule should have the same value.

The exclude option is a safeguard to ensure that no IPv6 address matching 64:ff9b::/96 will be returned to IPv6 clients. Without it the CG-NAT might translate IPv6 packets going to this IPv6 server into IPv4 packet with an incorrect IPv4 address.

The mapped option can be used to avoid mapping specific IPv4 address to IPv6. For example, it is a good idea not to embed any RFC 1918 addresses that name servers on the Internet might inadvertently return.

dynamic port SNAT64

The dynamic-port-snat64 works like dynamic-snat64, except that IP and ports from the CG-NAT pool are directly associated without any port block. It means that all the ports of a public IP is shared with several users.

See the Understanding Port Allocation/dynamic port section for more information about this point.

Here is an example of a dynamic port CG-NAT rule:

vsr running mypool!# allocation-mode dynamic-port
vsr running mypool!# ..
vsr running cg-nat#! rule 1
vsr running rule 1#! dynamic-port-snat64
vsr running dynamic-port-snat44#! match
vsr running match#! source ipv6-address fd00:100::/120
vsr running match#! outbound-interface eth1
vsr running dynamic-port-snat64#! translate-to
vsr running translate-to#! pool-name mypool
vsr running translate-to# destination-prefix 64:ff9b::/96
vsr running translate-to# commit

deterministic SNAT64

The deterministic-snat64 works like dynamic-snat64, except that IP and ports from the CG-NAT pool are associated in a deterministic manner to the user at the time of configuration: the CG-NAT pool is partitioned in chunk of port block, so that each user has its own port block. See the Understanding Port Allocation/deterministic block section for more information about this point.

Here is an example of a deterministic CG-NAT rule:

vsr running mypool!# allocation-mode deterministic-block
vsr running mypool!# ..
vsr running cg-nat#! rule 1
vsr running rule 1#! deterministic-snat64
vsr running deterministic-snat64#! match
vsr running match#! source ipv6-address fd00:100::/120
vsr running match#! outbound-interface eth1
vsr running match#! ..
vsr running deterministic-snat64#! translate-to
vsr running translate-to#! pool-name mypool
vsr running translate-to# destination-prefix 64:ff9b::/96
vsr running translate-to# commit

static SNAT64

All IPv6 packets routed through an output interface and matching the filtering criteria defined in a CG-NAT rule are source nated with an IPv4 address.

Here is an example of a CG-NAT rule:

vsr running mypool!# ..
vsr running cg-nat#! rule 1
vsr running rule 1#! static-snat64
vsr running static-snat64#! match
vsr running match#! source ipv6-address fd00:100::1/128
vsr running match#! outbound-interface eth1
vsr running static-snat64#! translate-to
vsr running translate-to#! ipv4-address 192.0.2.1
vsr running translate-to# commit

static DNAT46

All IPv4 packets received on an input interface and matching the filtering criteria defined in a CG-NAT rule are destination nated with an IPv6 address.

Here is an example of a CG-NAT rule:

vsr running mypool!# ..
vsr running cg-nat#! rule 1
vsr running rule 1#! static-dnat64
vsr running static-dnat64#! match
vsr running match#! destination ipv6-address 192.0.2.1/32
vsr running match#! inbound-interface eth1
vsr running static-dnat64#! translate-to
vsr running translate-to#! ipv4-address fd00:100::1/128
vsr running translate-to# commit

Understanding Port Allocation

dynamic port

Each time a new user sends a packet through the Virtual Service Router, an IP address in the pool is assigned to him. The public IPs are selected using a round-robin algorithm.

Here is an example to configure a pool with dynamic-port allocation-mode:

vsr running config# vrf main
vsr running vrf main# cg-nat
vsr running cg-nat!# pool mypool
vsr running mypool!# address 192.0.2.33-192.0.2.49
vsr running mypool!# address 192.0.1.0/24
vsr running mypool!# address 192.0.3.1
vsr running mypool!# allocation-mode dynamic-port port-algo random

For legal requirement, all connections must be logged to be able to provide the mapping between a user and a public IP address at a given point in time.

To enable connection logging:

.. code-block:: nc-cli

vsr running config# vrf main cg-nat logging enabled true vsr running config# vrf main cg-nat logging local true vsr running config# vrf main cg-nat logging event connection vsr running config# commit

As a user can consume all the ports of a public IP address is highly recommended to limit the maximum numbers of connections per users by using max-conntracks-per-user parameters of the cg-nat rule.

vsr running config# vrf main cg-nat rule 1 dynamic-port-snat44 translate-to max-conntracks-per-user 512

dynamic block

It is a legal requirement for an ISP to be able to provide the mapping between a user and a public IP address at a given point in time. With classic NAT, it means that each new user session has to be logged. This is obviously not scalable. Additionally with classic NAT, a user can consume all the ports of the public IP.

To reduce the amount of logs and equitably share the ports of the different public IPs, CG-NAT provides the Port Block Allocation (PBA) feature that consists in allocating blocks of ports to each user. As logging is done per block of ports, the amount of logs is reduced. And as the number and size of the blocks can be configured, the user port consumption is controlled. Here is how PBA works.

Each time a new user sends a packet through the Virtual Service Router, a block of ports is allocated to him from one of the IP addresses in the pool. The public IPs are selected using a round-robin algorithm. Each public IP is divided into blocks of ports, whose size and range are defined in the pool configuration. This prevents a single user from consuming all ports.

Here is an example to change the number of ports per block:

vsr running cg-nat# / vrf main cg-nat pool mypool allocation-mode dynamic-block block-size 256
vsr running cg-nat# commit

For the next session of the same user, a port is allocated from its block of ports, until the block is exhausted. In that case, a new block can be allocated for this user and it becomes the active block. There is only one active block per user. The maximum number of blocks per user is defined in the rule configuration.

Here is an example to change the maximum number of blocks per user:

vsr running cg-nat# / vrf main cg-nat rule 1 translate-to
max-blocks-per-user 2
vsr running cg-nat# commit

Since ports are allocated from the same block (until this one is empty), port prediction can potentially happen. To randomize port allocation, it is possible to allocate a new active block if the current active block has been active for too long, even if there are still some ports available in the active block. This feature is called active block timeout. As it can increase the average numbers of blocks allocated per user, it is disabled by default.

Here is an example to configure an active block timeout of 60 seconds:

vsr running cg-nat# / vrf main cg-nat rule 1 translate-to
active-block-timeout 60
vsr running cg-nat# commit

When all the ports are released from a non-active block, this one is released immediately. Regarding the active block, the block is only released when the user subscription times out. The default user timeout is 120 seconds.

Here is an example to change the user timeout:

vsr running cg-nat# / vrf main cg-nat rule 1 translate-to user-timeout 180
vsr running cg-nat# commit

deterministic block

ISPs have a legal requirement to be able to map a subscriber’s inside address with the address and port used on the public Internet (e.g., for abuse response). With dynamic allocation, any new block of port allocated/released need to be logged.

The port allocation algorithm for deterministic is predictable and sequential (i.e. the first block goes to address 1, the second block to address 2, etc.).

Thanks to this allocation method, it is possible to retrieve at any time a mapping between a public IP and a private one without any log. The following commands can be used for this purpose:

vsr> show cg-nat deterministic-public-block user-address 10.100.0.59
10.175.0.3 15046-15300
vsr> show cg-nat deterministic-user-address public-address 10.175.0.3 public-port 17001
10.100.0.66
vsr> show cg-nat deterministic-mappings | pager
10.100.0.0: 10.175.0.3 1-255
10.100.0.1: 10.175.0.3 256-510
10.100.0.2: 10.175.0.3 511-765
...

A deterministic rule needs to be associated to a pool with allocation-mode set to deterministic-block. Public IPs cannot be shared between deterministic and dynamic rules.

For deterministic rules, the block-size option is not mandatory. It is even recommended to not use it because it is automatically computed to this maximal value with the following formula: (port_range_of_the_pool * number_pool_ips) / number_of_users.

The users always have one block per protocol. As a consequence, deterministic rules don’t support the ‘max-blocks-per-user’ option.

Address Pooling

The address-pooling option defines if a public IP address is paired to a user:

  • paired (default value): It means that the same public IP address is used for all sessions originating from the same user.

  • no-paired: It means that different public IP addresses can be used for different sessions originating from the same user.

It’s recommended to use paired address pooling for applications that require all sessions associated with one private IP address to be translated to the same public IP address for multiple sessions.

Port algorithm

The port-algo defines which method is used to allocate ports:

  • parity (default): This algorithm preserves the parity of the port, i.e. an even port will be mapped to an even port, and an odd port will be mapped to an odd port.

  • random: This algorithm chooses a port randomly.

Here is an example to configure random port allocation for dynamic or deterministic rules.

vsr running translate-to# port-algo random
vsr running translate-to# commit

The port-algo can be configured on rule when block of ports are provided by the IP pool to each user. As each user has its own chunk of ports, a different algorithm can be used.

When ports are provided by the IP pool to each user. The algorithm to allocate a port must be configured on the pool. As the ports of an IP address is common to different users, the port allocation algorithm must be the same.

Port overloading

Port overloading option allows concurrent use of a single NAT translated port for multiple transport sessions, which allows a NAT to successfully process packets in a network that has a limited number of IP addresses. When enabled, these options can be configured:

  • unique-destination: Overload a port only when the destination address is unique or destination address and port pair is unique.

  • protocol: Enable port overloading for protocol TCP, UDP or both.

  • factor: Specify how much times a port can be used. For example, with a port range of 64512 and a port factor of 2, the maximum port capacity will be 129024.

Here is an example to configure port overloading for dynamic port pool.

vsr running dynamic-port# port-overloading
vsr running port-overloading# unique-destination address-and-port protocol tcp port-factor 32
vsr running port-overloading# commit

Note

  • Ports will be overloaded only when all ports available are at least used 1 times.

  • Port overloading is not compatible with endpoint mapping, endpoint filtering and hairpinning, these options must be disabled to use port overloading. It is also not compatible with Application Level Gateways which will be skipped for rules with port overloading enabled.

  • Port overloading is currently only available for dynamic port allocation mode and is configured on the pool.

Active Block Timeout

The active-block-timeout defines how the active block changes:

  • 0 (default): The active block changes only on port allocation, if it is full.

  • any other value: The active block also changes every time the timeout expires.

Note

As this option can increase the average number of blocks allocated per user, it is disabled by default.

Here is an example to configure active block timeout.

vsr running translate-to# active-block-timeout 60
vsr running translate-to# commit

Endpoint mapping

There are two endpoint mapping behaviors:

  • independent (default): The Virtual Service Router reuses the same port mapping for subsequent packets sent from the same internal IP address and port to any external IP address and port.

../../../_images/mapping-indep.png

  • dependent: The Virtual Service Router reuses the same port mapping for subsequent packets sent from the same internal IP address and port to the same external IP address and port.

../../../_images/mapping-dep.png

Here is an example to configure dependent endpoint mapping.

vsr running translate-to# endpoint-mapping dependent
vsr running translate-to# commit

Endpoint filtering

There are two endpoint filtering behaviors:

  • independent (default): Inbound packets from external endpoints are only filtered out if their destination IP address and port don’t match an existing public IP address and port mapping.

../../../_images/filtering-indep.png

  • dependent: Inbound packets from external endpoints are filtered out if they don’t match an existing mapping (source and destination IPs and ports, protocol).

../../../_images/filtering-dep.png

Here is an example to configure dependent endpoint filtering.

vsr running translate-to# endpoint-filtering dependent
vsr running translate-to# commit

Hairpinning

The hairpinning feature allows two endpoints (user 1 and user 2) on the private network to communicate together using their public IP addresses and ports.

By default, the hairpinning feature is disabled. To enable it, use the following command.

vsr running translate-to# hairpinning true
vsr running translate-to# commit

Conntracks

It is possible to set conntrack timeouts for each protocol. UDP, ICMP and GRE protocols only handle basic conntrack states (new, established, closed), whereas TCP offers more granularity.

vsr running config# vrf main network-stack fast-path conntrack timeouts tcp
syn-sent        simsyn-sent     syn-received    established     fin-sent
fin-received    closed          close-wait      fin-wait        last-ack
time-wait

vsr running config# vrf main network-stack fast-path conntrack timeouts udp
new            established    closed

It is also possible to change TCP behavior for specific cases.

vsr running config# vrf main network-stack fast-path conntrack behavior
tcp-window-check        tcp-rst-strict-order

ALG

Application-level gateways allow to use specific applications through CG-NAT.

vsr running config# vrf main network-stack fast-path alg
ftp          h323-q931    h323-ras     pptp         rtsp         sip-tcp
sip-udp      tftp         dns-udp

Note

In the previous versions, dns-udp was the default value. It is not the case anymore. Therefore, you may need to configure it in its new path: / vrf main network-stack fast-path alg. It is also possible to configure all vrfs with activated CG-NAT by using: / system network-stack fast-path alg. Note that if ALG is already configured in one vrf, its value will override the one set by / system network-stack fast-path alg.

Summary

The following table summarizes the different parameters for CG-NAT configuration.

Warning

Changing some configuration parameters requires to flush users. This is automatically done when required (see the table below), and causes a service interruption.

Category

Parameter

Flush

Note

pool

address

no

New addresses can be added without any impact. If an address is removed, all users assigned to it are flushed.

block-size

yes

port-algo

yes

port-overloading

yes

port-range

yes

rule

match

no

Conntracks not matching the new criteria are not flushed. Conntracks matching the new criteria are flushed.

pool-name

yes

max-blocks-per-user

no

Extra blocks are not flushed for users having more blocks than the new max. They become inactive and will be flushed after all sessions are released.

active-block-timeout

no

The new timeout starts after the current one expires.

user-timeout

no

The new timeout starts after the current one expires.

port-algo

no

For new blocks only.

endpoint-mapping

no

For new mappings only.

endpoint-filtering

no

For new mappings only.

hairpinning

no

For new mappings only.

address-pooling

no

For new users only.

Logging

events

Different types of event can be logged for CG-NAT.

connection

You can enable logs for CG-NAT to track each connection allocation/deallocation.

vsr running config# vrf main cg-nat logging enabled true
vsr running config# vrf main cg-nat logging local true
vsr running config# vrf main cg-nat logging event connection
vsr running config# commit

The logs can be displayed with the following command:

vsr running config# show log service cg-nat
Jun 14 12:17:35 dut-vm fp-cgnat[2484]: NEW CONN: fwd proto 6 10.100.0.1:4241 -> 10.200.0.1:33815, back proto 6 10.200.0.1:33815 --> 10.175.0.3:10752
Jun 14 12:19:04 dut-vm fp-cgnat[2484]: DESTROY CONN: fwd proto 6 10.100.0.1:4241 -> 10.200.0.1:33815, back proto 6 10.200.0.1:33915 -> 10.175.0.3:10753

The following fields are displayed for each connection allocation/deallocation:

  • timestamp of the event

  • type of event: NEW CONN (a new connection is created) or DESTROY CONN (a connection is released)

  • the forward flow with the following fields: - l4 protocol (i.e. 6 for tcp, 17 for udp, 1 for icmp, 47 for gre) - ip source and port source - ip destination and port destination

  • the backward flow with the same fields as the forward one

port blocks

You can enable logs for CG-NAT to track each port block allocation/deallocation for a user :

vsr running config# vrf main cg-nat logging enabled true
vsr running config# vrf main cg-nat logging local true
vsr running config# vrf main cg-nat logging event port-block
vsr running config# commit

The logs can be displayed with the following command:

vsr running config# show log service cg-nat
Jun 11 08:02:46 vsr fp-cgnat[4269]: CGNAT Log listen on 5001
Jun 11 08:03:09 vsr fp-cgnat[4269]: USER 100.64.0.1 (matching rule 1): NEW BLOCK (pool "mypool", ip public 192.0.2.33, proto 6, port 1 - 512)
Jun 11 08:07:30 vsr fp-cgnat[4269]: USER 100.64.0.1 (matching rule 1): DESTROY BLOCK (pool "mypool", ip public 192.0.2.33, proto 6, port 1 - 512)

The following fields are displayed for each port block allocation/deallocation:

  • IP address of the user

  • name of the CG-NAT rule matched by the user

  • type of event: NEW BLOCK (a new port range is associated to this user) or DESTROY BLOCK (port range is released for this user).

  • pool and ip used to nat the user

  • port range of the block

  • l4 protocol (i.e. 6 for tcp, 17 for udp, 1 for icmp, 47 for gre)

deterministic conf

The deterministic algorithm is predictable, so block allocation/deallocations don’t need to be logged. Anyway, to retrieve the mapping between user and public address, the deterministic algorithm needs to know the deterministic CG-NAT configuration.

To log the deterministic CG-NAT configuration:

vsr running config# vrf main cg-nat logging enabled true
vsr running config# vrf main cg-nat logging local true
vsr running config# vrf main cg-nat logging event deterministic-conf
vsr running config# commit

In consequence, the deterministic CG-NAT configuration is logged.

vsr running config# show log service cg-nat
Mar 02 14:32:06 dut-vm fp-cgnat[3526]: CGNAT Log listen on 5001
Mar 02 14:32:06 dut-vm fp-cgnat[3526]: cgnat-deterministic-conf;none
Mar 02 14:35:42 dut-vm fp-cgnat[3526]: cgnat-deterministic-conf;mypool|sequential1.0|10.100.0.0/24|10.175.0.3|255|1-65535

The deterministic CG-NAT configuration can be used to retrieve a user address behind a public address/port with the fp-cgnat-deterministic application.

# fp-cgnat-deterministic get-cpe-ip 10.175.0.3 17001 \
  'cgnat-deterministic-conf;mypool|sequential1.0|10.100.0.0/24|10.175.0.3|255|1-65535'
10.100.0.66

The ‘show cg-nat deterministic-public-block/deterministic-user-address/deterministic-mappings’ commands can also be used to retrieve a mapping for a previous deterministic CG-NAT configuration by using the date-and-time parameter. This one is able to retrieve the configuration applied at this time and get the information requested with fp-cgnat-deterministic application.

vsr> show cg-nat deterministic-user-address public-address 10.175.0.3 public-port 17001 date-and-time '2021-03-03 11:21:00'
10.100.0.66

remote syslog servers

For performance reason and permanent storage, it is highly recommended to log to one or more remote syslog servers. Logging in local should be only used for debug purpose.

vsr running config# vrf main cg-nat logging enabled true
vsr running config# vrf main cg-nat logging local false
vsr running config# vrf main cg-nat logging event connection
vsr running config# vrf main cg-nat logging rsyslog-server 10.125.0.2
vsr running config# vrf main cg-nat logging rsyslog-server 10.125.0.3
vsr running config# commit

When several rsyslog servers are defined, the logs are dispatched in function of the user’s ip address.

Some statistics and connection status of rsyslog-server can be retrieved with the following command:

vsr> show state vrf main cg-nat logging
logging
    enabled true
    local false
    event conntrack
    rsyslog-server 10.125.0.2
        port 514
        vrf main
        statistics
            transmit 41
            transmit-error 0
            build-error 0
            ..
        connection-status established
        ..
    rsyslog-server 10.125.0.3
        port 514
        vrf main
        statistics
            transmit 41
            transmit-error 0
            build-error 0
            ..
        connection-status established
        ..
    ..

Troubleshooting

Invalid packet state statistics

To display the CG-NAT statistics, use the following command.

vsr> show cg-nat statistics
...
Invalid packet state cases:
...
   0 TCP case RST
...
   0 TCP case I
   0 TCP case II
   0 TCP case III
...

If the TCP case I, II or III statistics are incremented, you may disable TCP window checks as follows.

vsr> edit running
vsr running config# vrf main network-stack fast-path conntrack
vsr running conntrack# behavior tcp-window-check enabled false
vsr running conntrack# commit

If the TCP case RST statistic is incremented, you may disable TCP RST strict ordering as follows.

vsr> edit running
vsr running config# vrf main network-stack fast-path conntrack
vsr running conntrack# behavior tcp-rst-strict-order enabled false
vsr running conntrack# commit

Note

Disabling these features improves performance to the detriment of TCP robustness.

State/NAT/USER/Block Allocation Failures

vsr> show cg-nat statistics
...
State and NAT entries:
...
   0 state allocation failures
...
   0 NAT entry allocation failures
   0 NAT port allocation failures
CGNat entries:
...
   0 USER allocation failures
...
   0 Block allocation failures
...

If one of these statistics is incremented, it means that one of the memory pools of the Virtual Service Router is full. Memory pool usage can be dumped using the following command.

vsr> show cg-nat mempool-usage
cgnat_user_pool : 2000/10000 (20.00%)
cgnat_block_pool : 8000/80000 (10.00%)
table_pool : 0/1056 (0.00%)
conn_pool : 1056736/1056736 (100.00%)
nat_pool : 1056736/1056736 (100.00%)

Another way to dump the memory pool is to look into the state using the following command.

vsr> show state / system fast-path limits cg-nat
cg-nat
    max-nat 1056736
    max-users 10000
    max-blocks 80000
    max-block-size 8192
    cur-nat 1056736
    cur-users 2000
    cur-blocks 8000
    ..
vsr> show state / system fast-path limits fp-max-conntracks
fp-max-conntracks 1056736
vsr> show state / system fast-path limits fp-cur-conntracks
fp-cur-conntracks 1056736

In the example above, the connection and NAT memory pools are full. Their size must be increased as follows.

vsr running config# / system fast-path limits cg-nat max-nat 2000000
vsr running config# / system fast-path limits fp-max-conntracks 2000000
vsr running cg-nat# commit

Refer to the capability tuning section.

No IP Public errors

vsr> show cg-nat statistics
...
CGNat entries:
...
   0 No IP Public
...

If this statistic is incremented, it means there are no blocks available in any public IP. This can be checked using the following command.

vsr> show cg-nat pool-usage pool-name mypool
tcp block usage: 4095/4095 (100.0%)
udp block usage: 4095/4095 (100.0%)
icmp block usage: 4095/4095 (100.0%)
gre block usage: 4095/4095 (100.0%)

To solve this issue, add a new public IP to the pool using the following command.

vsr> edit running
vsr running config# vrf main cg-nat pool mypool
vsr running pool mypool#  address 32.96.120.0/24
vsr running pool mypool# commit

NAT port allocation failures

There are two main reasons for port allocation failures:

  • A user has consumed all its port blocks. The maximum number of blocks per user can be increased in the rule using the max-blocks-per-user command.

  • No blocks are available on the public IP allocated to the user. In this case, the Full IP Public statistic is also incremented.

To list users with allocation failures to understand how many users are impacted, use the following command.

vsr> show cg-nat user rule-id 1 threshold-errors 1
100.64.0.1 -> 32.96.119.108
   2/2 tcp blocks, 0/2 udp blocks, 0/2 icmp blocks, 0/2 gre blocks
   63 no port errors, 0 no block errors, 0 full public ip errors

To understand why a specific user has many connections, use the following command.

vsr> show cg-nat conntracks rule-id 1 user-address 100.64.0.1
CONN:
   state:time_wait alg:none inactive_since:47s timeout:120s
   origin: tcp 100.64.0.1:1024 -> 32.96.118.2:6001
   reply : tcp 32.96.118.2:6001 -> 32.96.119.108:1216
   NAT source: 100.64.0.1:1024 -> 32.96.119.108:1216
CONN:
   state:time_wait alg:none inactive_since:56s timeout:120s
   origin: tcp 100.64.0.1:65024 -> 32.96.118.2:6000
   reply : tcp 32.96.118.2:6000 -> 32.96.119.108:1024
   NAT source: 100.64.0.1:65024 -> 32.96.119.108:1024
...

Maximum number of blocks reached

If the maximum number of blocks is reached, it probably means that you have not allocated enough blocks per user. You can collect some statistics to get average/percentile block and port usage of all users with the following commands.

vsr> show cg-nat block-statistics  rule-id 1
block-usage:
   1 user (with > 1 block = 1, ratio 100.00%)
   blocks per user: min = 2, max = 2, average = 2.00
   1 user (100.00%) have 2 blocks
vsr> show cg-nat port-statistics  rule-id 1
port-usage:
   1 user (with > 1 port = 1, ratio 100.00%)
   ports per user: min = 128, max = 128, average = 128.00
   1 user (100.00%) have 128 ports

You can also check the block usage and port usage of a specific user to get more details with the following commands.

vsr> show cg-nat blocks rule-id 1 user-address 100.64.0.1
BLOCK: status active, proto 1 tports 1024 - 1031 parity=1, usage 1/8
vsr> show cg-nat port-usage rule-id 1 user-address 100.64.0.1
tcp session usage: 0/16
udp session usage: 0/16
icmp session usage: 1/16
gre session usage: 0/16

Then, you can decide to increase the number of blocks per user or the block size. Refer to the Changing parameters section.

Full IP Public errors

vsr> show cg-nat statistics
...
CGNat entries:
...
   0 Full IP Public
...

The paired address pooling feature ensures the assignment of the same public IP address to all sessions originating from the same internal user, as described in RFC 4787 Req 2.

It means that when a user has started to use one public IP address, all its port blocks will be allocated from this same IP. Adding a new public IP to the pool won’t solve the issue, as the user cannot allocate a block from a new public IP.

A possible way to recover such situation is to add new IP address to the pool, and then flush all the current connections of all users, as follows.

vsr running config# / vrf main cg-nat pool mypool
vsr running pool mypool#  address 32.96.120.0/24
vsr running pool mypool# commit
Configuration committed.

vsr running pool mypool# flush cg-nat user rule-id 1

Dimensioning

The maximum numbers for NAT entries, CPEs (users), conntracks (sessions), blocks and block sizes are defined in the configuration. These limits can be adjusted to adapt to the amount of memory available in the system.

vsr running config# system fast-path limits cg-nat
max-nat    max-users     max-blocks    max-block-size

The following table shows a list of different limit combinations and the corresponding memory requirement. This is empirical and may have to be tuned according to your use case.

Max conntracks

Max nat entries

Max cpe

Max blocks

Required memory

1M

1M

10K

80K

5 GB

2M

2M

20K

80K

6 GB

4M

4M

20K

80K

8 GB

8M

8M

20K

80K

12 GB

16M

16M

20K

80K

24 GB

30M

30M

20K

80K

32 GB

Warning

Changing these values triggers a restart of the fast-path (hence, a service interruption). Therefore you should carefully think about your dimensioning before launching your Virtual Service Router into production.

Modifying these limits will automatically restart the fast path and interrupt packet processing. To check that the fast path is back up and running, use the following command.

vsr running config# show state system fast-path
fast-path
   enabled stopping
   ..
vsr running config#  show state system fast-path
fast-path
   enabled starting
   ..
vsr running config# # show state system fast-path
fast-path
   enabled true
   ...