Note

CG-NAT requires a Turbo CG-NAT Application License.

CG-NAT

CG-NAT, also known as Large Scale NAT (LSN), is an extension of NAT for large scale networks and ISPs.

The key advantages of CG-NAT compared to NAT are:

  • High Transparency: CG-NAT implements multiple advanced features like Endpoint-Independent Mapping, Endpoint-Independent Filtering, address pooling and port parity preservation. These features provide better experience to ‘nated’ users and allow scaling.

  • Fairness and Resource Sharing: CG-NAT provides options to limit the number of connections per user. This ensures that resources are equitably shared between the different users.

  • Optimized Logging system: CG-NAT can generate large amounts of logging data. CG-NAT implements a feature called port block allocation to limit the number of log entries by grouping per port range.

  • Transition between IPv6-only and IPv4-only networks thanks to NAT64, in conjunction to DNS64.

Caution

Stateful IP packet filtering and CG-NAT are exclusive. If CG-NAT is enabled, stateful IP packet filtering must be disabled on ports bound to the fast path.

See also

The CG-NAT command reference and the CG-NAT fast path limits command reference for details.

Configuration

Pool

A CG-NAT pool contains a list of IPv4 addresses used to change the IPv4 or IPv6 source address and port of a packet.

The CG-NAT implements a feature called port block allocation. Each time a new user sent a packet through the CG-NAT router, a block of ports (i.e. a port range) from one of the IPv4 addresses in the pool is allocated to this one. The number of ports given to an user is configurable via the block-size option.

Here is an example of a CG-NAT pool:

vrouter running config# vrf main
vrouter running vrf main# cg-nat
vrouter running cg-nat!# pool mypool
vrouter running mypool!# address 192.0.2.33-192.0.2.49
vrouter running mypool!# address 192.0.1.0/24
vrouter running mypool!# address 192.0.3.1
vrouter running mypool!# block-size 512

A pool needs to be associated to a CG-NAT rule, see the next section.

Rule

We have three types of NAT rules:

  • dynamic: Private addresses are dynamically mapped to public addresses: for each private address, a public address is dynamically allocated from a pool and associated to the private address.

  • deterministic: each private address is always mapped to the same public address and is assigned a dedicated port pool at the time of configuration

  • static: Mapping between private addresses and public ones are static. Public addresses are configured direcly in the rule.

static SNAT44

All packets routed through an output interface and matching the filtering criteria defined in a CG-NAT rule are source nated with an ip.

Here is an example of a CG-NAT rule:

vrouter running mypool!# ..
vrouter running cg-nat#! rule 1
vrouter running rule 1#! static-snat44
vrouter running static-snat44#! match
vrouter running match#! source ipv4-address 100.64.0.0/32
vrouter running match#! outbound-interface eth1
vrouter running static-snat44#! translate-to
vrouter running translate-to#! ipv4-address 192.0.2.1
vrouter running translate-to# commit

To display the applied configuration:

vrouter running config# show state vrf main cg-nat
cg-nat
    enabled true
    rule 1
        static-snat44
            match
                source
                    ipv4-address 100.64.0.1/32
                    ..
                outbound-interface eth1
                ..
            translate-to
                ipv4-address 192.0.2.1
                ..
            ..
        ..
    logging
        enabled false
        ..
    ..

static DNAT44

All packets received on an input interface and matching the filtering criteria defined in a CG-NAT rule are destination nated with an ip.

Here is an example of a CG-NAT rule:

vrouter running mypool!# ..
vrouter running cg-nat#! rule 1
vrouter running rule 1#! static-dnat44
vrouter running static-dnat44#! match
vrouter running match#! destination ipv4-address 192.0.2.1/32
vrouter running match#! inbound-interface eth1
vrouter running static-dnat44#! translate-to
vrouter running translate-to#! ipv4-address 100.64.0.1
vrouter running translate-to# commit

To display the applied configuration:

vrouter running config# show state vrf main cg-nat
cg-nat
    enabled true
    rule 1
        static-dnat44
            match
                destination
                    ipv4-address 192.0.2.1/32
                    ..
                inbound-interface eth1
                ..
            translate-to
                ipv4-address 100.64.0.1
                ..
            ..
        ..
    logging
        enabled false
        ..
    ..

dynamic SNAT44

All packets routed through an output interface and matching the filtering criteria defined in a CG-NAT rule are source nated with an ip dynamically assigned from one CG-NAT pool.

Here is an example of a CG-NAT rule:

vrouter running mypool!# ..
vrouter running cg-nat#! rule 1
vrouter running rule 1#! dynamic-snat44
vrouter running dynamic-snat44#! match
vrouter running match#! source ipv4-address 100.64.0.0/10
vrouter running match#! outbound-interface eth1
vrouter running dynamic-snat44#! translate-to
vrouter running translate-to#! pool-name mypool
vrouter running translate-to# commit

To display the applied configuration:

vrouter running config# show state vrf main cg-nat
cg-nat
    enabled true
    pool mypool
        address 192.0.1.1-192.0.1.254
        address 192.0.2.33-192.0.2.49
        address 192.0.3.1
        block-allocation-mode dynamic
        block-size 512
        port-range
            1024
            65535
            ..
        ..
    rule 1
        dynamic-snat44
            match
                source
                    ipv4-address 100.64.0.0/10
                    ..
                outbound-interface eth1
                ..
            translate-to
                pool-name mypool
                max-conntracks-per-user 0
                max-blocks-per-user 1
                active-block-timeout 0
                user-timeout 120
                port-algo parity
                endpoint-mapping independent
                endpoint-filtering independent
                hairpinning false
                address-pooling paired
                ..
            ..
        ..
    logging
        enabled false
        ..
    ..

The same configuration can be made using this NETCONF XML configuration:

vrouter running config# show config xml absolute vrf main cg-nat
<config xmlns="urn:6wind:vrouter">
  <vrf>
    <name>main</name>
    <cg-nat xmlns="urn:6wind:vrouter/cgnat">
      <enabled>true</enabled>
      <logging/>
      <pool>
        <name>mypool</name>
        <port-range/>
        <address>192.0.2.33-192.0.2.49</address>
        <address>192.0.1.0/24</address>
        <address>192.0.3.1</address>
        <block-allocation-mode>dynamic<block-allocation-mode/>
        <block-size>512</block-size>
      </pool>
      <rule>
        <id>1</id>
        <dynamic-snat44>
          <match>
            <source>
              <ipv4-address>100.64.0.0/10</address>
            </source>
            <outbound-interface>eth1</outbound-interface>
          </match>
          <translate-to>
            <pool-name>mypool</pool-name>
          </translate-to>
        </dynamic-snat44>
      </rule>
    </cg-nat>
  </vrf>
</config>

deterministic SNAT44

The deterministic-snat44 works like dynamic-snat44, except that IP and ports from the CG-NAT pool are associated in a deterministic manner to the user at the time of configuration: the CG-NAT pool is partitioned in chunk of port block, so that each user has its own port block. See the Understanding Port Block Allocation/deterministic for more information about this point.

Here is an example of a deterministic CG-NAT rule:

vrouter running mypool!# del block-size
vrouter running mypool!# block-allocation-mode deterministic
vrouter running mypool!# ..
vrouter running cg-nat#! rule 1
vrouter running rule 1#! deterministic-snat44
vrouter running deterministic-snat44#! match
vrouter running match#! source ipv4-address 100.64.0.0/10
vrouter running match#! outbound-interface eth1
vrouter running deterministic-snat44#! translate-to
vrouter running translate-to#! pool-name mypool
vrouter running translate-to# commit

dynamic SNAT64

In the case of NAT64, translation between IPv6-only addresses on the private side and IPv4 addresses on the public side is taken care of by mapping the IPv4 public addresses to a special IPv6 prefix that is configured in the translate-to section of the rule.

DNS64 must be configured to translate IPv4 addresses DNS replies to IPv6 ones using this prefix.

The rest of the configuration is similar to dynamic NAT44.

vrouter running config# vrf main
vrouter running vrf main# cg-nat
vrouter running cg-nat#! rule 1
vrouter running rule 1#! dynamic-snat64
vrouter running dynamic-snat64#! match
vrouter running match#! source ipv6-address fd00:100::/64
vrouter running match#! outbound-interface eth2
vrouter running match#! ..
vrouter running dynamic-snat64#! translate-to
vrouter running translate-to#! pool-name mypool
vrouter running translate-to#! max-blocks-per-user 4
vrouter running translate-to#! destination-prefix 64:ff9b::/96
vrouter running translate-to# .. .. .. ..
vrouter running vrf main# dns
vrouter running dns# proxy
vrouter running proxy# dns64 64:ff9b::/96
vrouter running dns64 64:ff9b::/96# client fd00:100::/64
vrouter running dns64 64:ff9b::/96# exclude 64:ff9b::/96
vrouter running dns64 64:ff9b::/96# mapped not 10.0.0.0/8 192.168.0.0/16 172.16.0.0/12

The client option can be used to select which client the DNS64 function will apply to. The CPE subnet in the CG-NAT rule should have the same value.

The exclude option is a safeguard to ensure that no IPv6 address matching 64:ff9b::/96 will be returned to IPv6 clients. Without it the CG-NAT might translate IPv6 packets going to this IPv6 server into IPv4 packet with an incorrect IPv4 address.

The mapped option can be used to avoid mapping specific IPv4 address to IPv6. For example, it is a good idea not to embed any RFC 1918 addresses that name servers on the Internet might inadvertently return.

static SNAT64

All IPv6 packets routed through an output interface and matching the filtering criteria defined in a CG-NAT rule are source nated with an IPv4 address.

Here is an example of a CG-NAT rule:

vrouter running mypool!# ..
vrouter running cg-nat#! rule 1
vrouter running rule 1#! static-snat64
vrouter running static-snat64#! match
vrouter running match#! source ipv6-address fd00:100::1/128
vrouter running match#! outbound-interface eth1
vrouter running static-snat64#! translate-to
vrouter running translate-to#! ipv4-address 192.0.2.1
vrouter running translate-to# commit

static DNAT46

All IPv4 packets received on an input interface and matching the filtering criteria defined in a CG-NAT rule are destination nated with an IPv6 address.

Here is an example of a CG-NAT rule:

vrouter running mypool!# ..
vrouter running cg-nat#! rule 1
vrouter running rule 1#! static-dnat64
vrouter running static-dnat64#! match
vrouter running match#! destination ipv6-address 192.0.2.1/32
vrouter running match#! inbound-interface eth1
vrouter running static-dnat64#! translate-to
vrouter running translate-to#! ipv4-address fd00:100::1/128
vrouter running translate-to# commit

Understanding Port Block Allocation

dynamic

It is a legal requirement for an ISP to be able to provide the mapping between a user and a public IP address at a given point in time. With classic NAT, it means that each new user session has to be logged. This is obviously not scalable. Additionally with classic NAT, a user can consume all the ports of the public IP.

To reduce the amount of logs and equitably share the ports of the different public IPs, CG-NAT provides the Port Block Allocation (PBA) feature that consists in allocating blocks of ports to each user. As logging is done per block of ports, the amount of logs is reduced. And as the number and size of the blocks can be configured, the user port consumption is controlled. Here is how PBA works.

Each time a new user sends a packet through the vRouter, a block of ports is allocated to him from one of the IP addresses in the pool. The public IPs are selected using a round-robin algorithm. Each public IP is divided into blocks of ports, whose size and range is defined in the pool configuration. This prevents a single user from consuming all ports.

Here is an example to change the number of ports per block:

vrouter running cg-nat# / vrf main cg-nat pool mypool block-size 256
vrouter running cg-nat# commit

For the next session of the same user, a port is allocated from its block of ports, until the block is exhausted. In that case, a new block can be allocated for this user and it becomes the active block. There is only one active block per user. The maximum number of blocks per user is defined in the rule configuration.

Here is an example to change the maximum number of blocks per user:

vrouter running cg-nat# / vrf main cg-nat rule 1 translate-to
max-blocks-per-user 2
vrouter running cg-nat# commit

Since ports are allocated from the same block (until this one is empty), port prediction can potentially happen. To randomize port allocation, it is possible to allocate a new active block if the current active block has been active for too long, even if there are still some ports available in the active block. This feature is called active block timeout. As it can increase the average numbers of blocks allocated per user, it is disabled by default.

Here is an example to configure an active block timeout of 60 seconds:

vrouter running cg-nat# / vrf main cg-nat rule 1 translate-to
active-block-timeout 60
vrouter running cg-nat# commit

When all the ports are released from a non-active block, this one is released immediately. Regarding the active block, the block is only released when the user subscription times out. The default user timeout is 120 seconds.

Here is an example to change the user timeout:

vrouter running cg-nat# / vrf main cg-nat rule 1 translate-to user-timeout 180
vrouter running cg-nat# commit

deterministic

ISPs have a legal requirement to be able to map a subscriber’s inside address with the address and port used on the public Internet (e.g., for abuse response). With dynamic allocation, any new block of port allocated/released need to be logged.

The port allocation algorithm for deterministic is predictable and sequential (i.e. the first block goes to address 1, the second block to address 2, etc.).

Thanks to this allocation method, it is possible to retrieve at any time a mapping between a public IP and a private one without any log. The following commands can be used for this purpose:

vrouter> show cg-nat deterministic-public-block user-address 10.100.0.59
10.175.0.3 15046-15300
vrouter> show cg-nat deterministic-user-address public-address 10.175.0.3 public-port 17001
10.100.0.66
vrouter> show cg-nat deterministic-mappings | pager
10.100.0.0: 10.175.0.3 1-255
10.100.0.1: 10.175.0.3 256-510
10.100.0.2: 10.175.0.3 511-765
...

A deterministic rule needs to be associated to a pool with block-allocation-mode set to deterministic. Public IPs cannot be shared between deterministic and dynamic rules.

For deterministic rules, the block-size option is not mandatory. It is even recommended to not use it. Because it is automatically computed to this maximal value with the following formula: (port_range_of_the_pool * number_pool_ips) / number_of_user.

The users always have one block per protocol. As a consequence, deterministic rules don’t support the ‘max-blocks-per-user’ option.

Address Pooling

The address-pooling option defines if a public IP address is paired to a user:

  • paired (default value): It means that the same public IP address is used for all sessions originating from the same user.

  • no-paired: It means that different public IP addresses can be used for different sessions originating from the same user.

It’s recommended to use paired address pooling for applications that require all sessions associated with one private IP address to be translated to the same public IP address for multiple sessions.

Port algorithm

The port-algo defines which method is used to allocate ports:

  • parity (default): This algorithm preserves the parity of the port, i.e. an even port will be mapped to an even port, and an odd port will be mapped to an odd port.

  • random: This algorithm chooses a port randomly.

Here is an example to configure random port allocation.

vrouter running translate-to# port-algo random
vrouter running translate-to# commit

Active Block Timeout

The active-block-timeout defines how the active block changes:

  • 0 (default): The active block changes only on port allocation, if it is full.

  • any other value: The active block also changes everytime the timeout expires.

Note

As this option can increase the average number of blocks allocated per user, it is disabled by default.

Here is an example to configure active block timeout.

vrouter running translate-to# active-block-timeout 60
vrouter running translate-to# commit

Endpoint mapping

There are two endpoint mapping behaviors:

  • independent (default): The vRouter reuses the same port mapping for subsequent packets sent from the same internal IP address and port to any external IP address and port.

../../../_images/mapping-indep.png

  • dependent: The vRouter reuses the same port mapping for subsequent packets sent from the same internal IP address and port to the same external IP address and port.

../../../_images/mapping-dep.png

Here is an example to configure dependent endpoint mapping.

vrouter running translate-to# endpoint-mapping dependent
vrouter running translate-to# commit

Endpoint filtering

There are two endpoint filtering behaviors:

  • independent (default): Inbound packets from external endpoints are only filtered out if their destination IP address and port don’t match an existing public IP address and port mapping.

../../../_images/filtering-indep.png

  • dependent: Inbound packets from external endpoints are filtered out if they don’t match an existing mapping (source and destination IPs and ports, protocol).

../../../_images/filtering-dep.png

Here is an example to configure dependent endpoint filtering.

vrouter running translate-to# endpoint-filtering dependent
vrouter running translate-to# commit

Hairpinning

The hairpinning feature allows two endpoints (user 1 and user 2) on the private network to communicate together using their public IP addresses and ports.

By default, the hairpinning feature is disabled. To enable it, use the following command.

vrouter running translate-to# hairpinning true
vrouter running translate-to# commit

Conntracks

It is possible to set conntrack timeouts for each protocol. UDP, ICMP and GRE protocols only handle basic conntrack states (new, established, closed), whereas TCP offers more granularity.

vrouter running config# vrf main cg-nat conntrack timeouts tcp
syn-sent        simsyn-sent     syn-received    established     fin-sent
fin-received    closed          close-wait      fin-wait        last-ack
time-wait

vrouter running config# vrf main cg-nat conntrack timeouts udp
new            established    closed

It is also possible to change TCP behavior for specific cases.

vrouter running config# vrf main cg-nat conntrack behavior
tcp-window-check        tcp-rst-strict-order

ALG

Application-level gateways allow to use specific applications through CG-NAT.

vrouter running config# vrf main cg-nat alg
ftp          h323-q931    h323-ras     pptp         rtsp         sip-tcp
sip-udp      tftp         dns-udp

Summary

The following table summarizes the different parameters for CG-NAT configuration.

Warning

Changing some configuration parameters requires to flush users. This is automatically done when required (see the table below), and causes a service interruption.

Category

Parameter

Flush

Note

pool

address

no

New addresses can be added without any impact. If an address is removed, all users assigned to it are flushed.

block-size

yes

port-range

yes

rule

match

no

Conntracks not matching the new criteria are not flushed. Conntracks matching the new criteria are flushed.

pool-name

yes

max-blocks-per-user

no

Extra blocks are not flushed for users having more blocks than the new max. They become inactive and will be flushed after all sessions are released.

active-block-timeout

no

The new timeout starts after the current one expires.

user-timeout

no

The new timeout starts after the current one expires.

port-algo

no

For new blocks only.

endpoint-mapping

no

For new mappings only.

endpoint-filtering

no

For new mappings only.

hairpinning

no

For new mappings only.

address-pooling

no

For new users only.

Logging

dynamic

You can enable logs for CG-NAT to track each port block allocation/deallocation for a user :

vrouter running config# vrf main cg-nat logging enabled true
commit

The logs can be displayed with the following command:

vrouter running config# show log service cgnat
Jun 11 08:02:46 vrouter systemd[1]: Started Fast Path cgnat log daemon.
Jun 11 08:02:46 vrouter fp-cgnat-logd[4269]: CGNAT Log listen on 5001
Jun 11 08:03:09 vrouter fp-cgnat-logd[4269]: USER 100.64.0.1 (matching rule 1): NEW BLOCK (pool "mypool", ip public 192.0.2.33, proto 6, port 1 - 512) at Tue Jun 11 08:03:09 2019
Jun 11 08:07:30 vrouter fp-cgnat-logd[4269]: USER 100.64.0.1 (matching rule 1): DESTROY BLOCK (pool "mypool", ip public 192.0.2.33, proto 6, port 1 - 512) at Tue Jun 11 08:07:30 2019

The following fields are displayed for each port block allocation/deallocation:

  • IP address of the user

  • name of the CG-NAT rule matched by the user

  • type of event: NEW BLOCK (a new port range is associated to this user) or DESTROY BLOCK (port range is released for this user).

  • pool and ip used to nat the user

  • port range of the block

  • l4 protocol (i.e. 6 for tcp, 17 for udp, 1 for icmp, 47 for gre)

  • timestamp of the event

deterministic

The deterministic algorithm is predictable, so block allocation/deallocations don’t need to be logged. Anyway, to retrieve the mapping betwee user and public address, the deterministic algorithm needs to know the deterministic CG-NAT is configured.

In consequence, the deterministic CG-NAT configuration is logged.

vrouter running config# show log service cgnat
Mar 02 14:32:06 dut-vm systemd[1]: Started Fast Path cgnat log daemon.
Mar 02 14:32:06 dut-vm fp-cgnat-logd[3526]: CGNAT Log listen on 5001
Mar 02 14:32:06 dut-vm fp-cgnat-logd[3526]: cgnat-deterministic-conf;none at 2021-03-02T14:32:06Z
Mar 02 14:35:42 dut-vm fp-cgnat-logd[3526]: cgnat-deterministic-conf;mypool|sequential1.0|10.100.0.0/24|10.175.0.3|255|1-65535 at 2021-03-02T14:35:42Z

The deterministic CG-NAT configuration can be used to retrieve a user address behind a public address/port with the fp-cgnat-deterministic application.

# fp-cgnat-deterministic get-cpe-ip 10.175.0.3 17001 \
  'cgnat-deterministic-conf;mypool|sequential1.0|10.100.0.0/24|10.175.0.3|255|1-65535'
10.100.0.66

The ‘show cg-nat deterministic-public-block/deterministic-user-address/deterministic-mappings’ commands can also be used to retrieve a mapping for a previous deterministic CG-NAT configuration by using the date-and-time parameter. This one is able to retrieve the configuration applied at this time and get the information requested with fp-cgnat-deterministic application.

vrouter> show cg-nat deterministic-user-address public-address 10.175.0.3 public-port 17001 date-and-time '2021-03-03 11:21:00'
10.100.0.66

Troubleshooting

Invalid packet state statistics

To display the CG-NAT statistics, use the following command.

vrouter> show cg-nat statistics
...
Invalid packet state cases:
...
   0 TCP case RST
...
   0 TCP case I
   0 TCP case II
   0 TCP case III
...

If the TCP case I, II or III statistics are incremented, you may disable TCP window checks as follows.

vrouter> edit running
vrouter running config# vrf main cg-nat conntrack
vrouter running conntrack# behavior tcp-window-check enabled false
vrouter running conntrack# commit

If the TCP case RST statistic is incremented, you may disable TCP RST strict ordering as follows.

vrouter> edit running
vrouter running config# vrf main cg-nat conntrack
vrouter running conntrack# behavior tcp-rst-strict-order enabled false
vrouter running conntrack# commit

Note

Disabling these features improves performance to the detriment of TCP robustness.

State/NAT/USER/Block Allocation Failures

vrouter> show cg-nat statistics
...
State and NAT entries:
...
   0 state allocation failures
...
   0 NAT entry allocation failures
   0 NAT port allocation failures
CGNat entries:
...
   0 USER allocation failures
...
   0 Block allocation failures
...

If one of these statistics is incremented, it means that one of the memory pools of the vRouter is full. Memory pool usage can be dumped using the following command.

vrouter> show cg-nat mempool-usage
cgnat_user_pool : 2000/10000 (20.00%)
cgnat_block_pool : 8000/80000 (10.00%)
table_pool : 0/1056 (0.00%)
conn_pool : 1056736/1056736 (100.00%)
nat_pool : 1056736/1056736 (100.00%)

In the example above, the connection and NAT memory pools are full. Their size must be increased as follows.

vrouter running config# / system
fast-path limits cg-nat
vrouter running cg-nat# max-conntracks 2000000
vrouter running cg-nat# max-nat-entries 2000000
vrouter running cg-nat# commit

Refer to the capability tuning section.

No IP Public errors

vrouter> show cg-nat statistics
...
CGNat entries:
...
   0 No IP Public
...

If this statistic is incremented, it means there are no blocks available in any public IP. This can be checked using the following command.

vrouter> show cg-nat pool-usage pool-name mypool
tcp block usage: 4095/4095 (100.0%)
udp block usage: 4095/4095 (100.0%)
icmp block usage: 4095/4095 (100.0%)
gre block usage: 4095/4095 (100.0%)

To solve this issue, add a new public IP to the pool using the following command.

vrouter> edit running
vourter running config# vrf main cg-nat pool mypool
vrouter running pool mypool#  address 32.96.120.0/24
vrouter running pool mypool# commit

NAT port allocation failures

There are two main reasons for port allocation failures:

  • A user has consumed all its port blocks. The maximum number of blocks per user can be increased in the rule using the max-blocks-per-user command.

  • No blocks are available on the public IP allocated to the user. In this case, the Full IP Public statistic is also incremented.

To list users with allocation failures to understand how many users are impacted, use the following command.

vrouter> show cg-nat user rule-id 1 threshold-errors 1
100.64.0.1 -> 32.96.119.108
   2/2 tcp blocks, 0/2 udp blocks, 0/2 icmp blocks, 0/2 gre blocks
   63 no port errors, 0 no block errors, 0 full public ip errors

To understand why a specific user has many connections, use the following command.

vrouter> show cg-nat conntracks rule-id 1 user-address 100.64.0.1
CON:
   vrfid 0 flags 0x6 alg none tsdiff 47 timeout 120
   forw proto 6 100.64.0.1:1024-> 32.96.118.2:6001 hash:be3505a5
   back proto 6 32.96.118.2:6001-> 32.96.119.108:1216 hash:92e65736
   state 10:
      F { end 0 maxend 0 mwin 0 wscale 0 flags 1}
      T { end 0 maxend 0 mwin 0 wscale 0 flags 0}
   NAT: original address 100.64.0.1 proto 6 oport 1024 tport 1216
CON:
   vrfid 0 flags 0x6 alg none tsdiff 56 timeout 120
   forw proto 6 100.64.0.1:65024-> 32.96.118.2:6000 hash:913f8bf7
   back proto 6 32.96.118.2:6000-> 32.96.119.108:1024 hash:27051895
   state 10:
      F {end 0 maxend 0 mwin 0 wscale 0 flags 1}
      T {end 0 maxend 0 mwin 0 wscale 0 flags 0}
   NAT: original address 100.64.0.1 proto 6 oport 65024 tport 1024
...

Maximum number of blocks reached

If the maximum number of blocks is reached, it probably means that you have not allocated enough blocks per user. You can collect some statistics to get average/percentile block and port usage of all users with the following commands.

vrouter> show cg-nat block-statistics  rule-id 1
block-usage:
   1 user (with > 1 block = 1, ratio 100.00%)
   blocks per user: min = 2, max = 2, average = 2.00
   1 user (100.00%) have 2 blocks
vrouter> show cg-nat port-statistics  rule-id 1
port-usage:
   1 user (with > 1 port = 1, ratio 100.00%)
   ports per user: min = 128, max = 128, average = 128.00
   1 user (100.00%) have 128 ports

You can also check the block usage and port usage of a specific user to get more details with the following commands.

vrouter> show cg-nat blocks rule-id 1 user-address 100.64.0.1
BLOCK: status active, proto 1 tports 1024 - 1031 parity=1, usage 1/8
vrouter> show cg-nat port-usage rule-id 1 user-address 100.64.0.1
tcp session usage: 0/16
udp session usage: 0/16
icmp session usage: 1/16
gre session usage: 0/16

Then, you can decide to increase the number of blocks per user or the block size. Refer to the Changing parameters section.

Full IP Public errors

vrouter> show cg-nat statistics
...
CGNat entries:
...
   0 Full IP Public
...

The paired address pooling feature ensures the assignment of the same public IP address to all sessions originating from the same internal user, as described in RFC 4787 Req 2.

It means that when a user has started to use one public IP address, all its port blocks will be allocated from this same IP. Adding a new public IP to the pool won’t solve the issue, as the user cannot allocate a block from a new public IP.

A possible way to recover such situation is to add new IP address to the pool, and then flush all the current connections of all users, as follows.

vrouter running config# / vrf main cg-nat pool mypool
vrouter running pool mypool#  address 32.96.120.0/24
vrouter running pool mypool# commit
Configuration committed.

vrouter running pool mypool# flush cg-nat user rule-id 1

Dimensioning

The maximum numbers for NAT entries, CPEs (users), conntracks (sessions), blocks and block sizes are defined in the configuration. These limits can be adjusted to adapt to the amount of memory available in the system.

vrouter running config# system fast-path limits cg-nat
max-conntracks     max-nat-entries    max-users          max-blocks
max-block-size

The following table shows a list of different limit combinations and the corresponding memory requirement. This is empirical and may have to be tuned according to your use case.

Max conntracks

Max nat entries

Max cpe

Max blocks

Required memory

1M

1M

10K

80K

5 GB

2M

2M

20K

80K

6 GB

4M

4M

20K

80K

8 GB

8M

8M

20K

80K

12 GB

16M

16M

20K

80K

24 GB

30M

30M

20K

80K

32 GB

Warning

Changing these values triggers a restart of the fast-path (hence, a service interruption). Therefore you should carefully think about your dimensioning before launching your Turbo Router into production.

Modifying these limits will automatically restart the fast path and interrupt packet processing. To check that the fast path is back up and running, use the following command.

vrouter running config# show state system fast-path
fast-path
   enabled stopping
   ..
vrouter running config#  show state system fast-path
fast-path
   enabled starting
   ..
vrouter running config# # show state system fast-path
fast-path
   enabled true
   ...