CG-NAT¶

CG-NAT, also known as Large Scale NAT (LSN), is an extension of NAT for large scale networks and ISPs.

The key advantages of CG-NAT compared to NAT are:

High Transparency:CG-NAT implements multiple advanced features like Endpoint-Independent Mapping, Endpoint-Independent Filtering, address pooling and port parity preservation. These features provide better experience to ‘nated’ users and allow scaling.
Fairness and Resource Sharing:CG-NAT provides options to limit the number of connections per user. This ensures that resources are equitably shared between the different users.
Optimized Logging system: CG-NAT can generate large amounts of logging data. CG-NAT implements a feature called port block allocation to limit the number of log entries by grouping per port range.

Caution

IP packet filtering and CG-NAT are exclusive. If CG-NAT is enabled, IP packet filtering must be disabled on ports bound to the fast path.

Configuration ¶

Pool ¶

A CG-NAT pool contains a list of IPv4 addresses used to change the source IPv4 address and port of a packet.

The CG-NAT implements a feature called port block allocation. Each time a new user sent a packet through the CG-NAT router, a block of ports (i.e. a port range) from one of the IPv4 addresses in the pool is allocated to this one. The number of ports given to an user is configurable via the block-size option.

Here is an example of a CG-NAT pool:

vrouter running config# vrf main
vrouter running vrf main# cg-nat
vrouter running cg-nat!# pool mypool
vrouter running mypool!# address 192.0.2.33-192.0.2.49
vrouter running mypool!# address 192.0.1.0/24
vrouter running mypool!# address 192.0.3.1
vrouter running mypool!# block-size 512

A pool needs to be associated to a rule, see the next section.

Rule ¶

All packets routed through an output interface and matching the filtering criteria defined in a CG-NAT rule are source nated with an ip from one CG-NAT pool.

Here is an example of a CG-NAT rule:

vrouter running mypool!# ..
vrouter running cg-nat#! rule 1
vrouter running rule 1#! match
vrouter running match#! source address 100.64.0.0/10
vrouter running match#! outbound-interface eth1
vrouter running rule 1#! translate-to
vrouter running translate-to#! pool-name mypool
vrouter running translate-to# commit

To display the applied configuration:

vrouter running config# show state vrf main cg-nat
cg-nat
    enabled true
    pool mypool
        address 192.0.1.1-192.0.1.254
        address 192.0.2.33-192.0.2.49
        address 192.0.3.1
        block-size 512
        port-range
            1024
            65535
            ..
        ..
    rule 1
        match
            source
                address 100.64.0.0/10
                ..
            outbound-interface eth1
            ..
        translate-to
            pool-name mypool
            max-conntracks-per-user 0
            max-blocks-per-user 1
            active-block-timeout 0
            user-timeout 120
            port-algo parity
            endpoint-mapping independent
            endpoint-filtering independent
            hairpinning false
            ..
        ..
    logging
        enabled false
        ..
    ..

The same configuration can be made using this NETCONF XML configuration:

vrouter running config# show config xml absolute vrf main cg-nat
<config xmlns="urn:6wind:vrouter">
  <vrf>
    <name>main</name>
    <cg-nat xmlns="urn:6wind:vrouter/cgnat">
      <enabled>true</enabled>
      <logging/>
      <pool>
        <name>mypool</name>
        <port-range/>
        <address>192.0.2.33-192.0.2.49</address>
        <address>192.0.1.0/24</address>
        <address>192.0.3.1</address>
        <block-size>512</block-size>
      </pool>
      <rule>
        <id>1</id>
        <match>
          <source>
            <address>100.64.0.0/10</address>
          </source>
          <outbound-interface>eth1</outbound-interface>
        </match>
        <translate-to>
          <pool-name>mypool</pool-name>
        </translate-to>
      </rule>
    </cg-nat>
  </vrf>
</config>

Understanding Port Block Allocation ¶

It is a legal requirement for an ISP to be able to provide the mapping between a user and a public IP address at a given point in time. With classic NAT, it means that each new user session has to be logged. This is obviously not scalable. Additionally with classic NAT, a user can consume all the ports of the public IP.

To reduce the amount of logs and equitably share the ports of the different public IPs, CG-NAT provides the Port Block Allocation (PBA) feature that consists in allocating blocks of ports to each user. As logging is done per block of ports, the amount of logs is reduced. And as the number and size of the blocks can be configured, the user port consumption is controlled. Here is how PBA works.

Each time a new user sends a packet through the vRouter, a block of ports is allocated to him from one of the IP addresses in the pool. The public IPs are selected using a round-robin algorithm. Each public IP is divided into blocks of ports, whose size and range is defined in the pool configuration. This prevents a single user from consuming all ports.

Here is an example to change the number of ports per block:

vrouter running cg-nat# / vrf main cg-nat pool mypool block-size 256
vrouter running cg-nat# commit

For the next session of the same user, a port is allocated from its block of ports, until the block is exhausted. In that case, a new block can be allocated for this user and it becomes the active block. There is only one active block per user. The maximum number of blocks per user is defined in the rule configuration.

Here is an example to change the maximum number of blocks per user:

vrouter running cg-nat# / vrf main cg-nat rule 1 translate-to
max-blocks-per-user 2
vrouter running cg-nat# commit

Since ports are allocated from the same block (until this one is empty), port prediction can potentially happen. To randomize port allocation, it is possible to allocate a new active block if the current active block has been active for too long, even if there are still some ports available in the active block. This feature is called active block timeout. As it can increase the average numbers of blocks allocated per user, it is disabled by default.

Here is an example to configure an active block timeout of 60 seconds:

vrouter running cg-nat# / vrf main cg-nat rule 1 translate-to
active-block-timeout 60
vrouter running cg-nat# commit

When all the ports are released from a non-active block, this one is released immediately. Regarding the active block, the block is only released when the user subscription times out. The default user timeout is 120 seconds.

Here is an example to change the user timeout:

vrouter running cg-nat# / vrf main cg-nat rule 1 translate-to user-timeout 180
vrouter running cg-nat# commit

Port algorithm ¶

The port-algo defines which method is used to allocate ports:

parity (default): This algorithm preserves the parity of the port, i.e. an even port will be mapped to an even port, and an odd port will be mapped to an odd port.
random: This algorithm chooses a port randomly.

Here is an example to configure random port allocation.

vrouter running translate-to# port-algo random
vrouter running translate-to# commit

Active Block Timeout ¶

The active-block-timeout defines how the active block changes:

0 (default): The active block changes only on port allocation, if it is full.
any other value: The active block also changes everytime the timeout expires.

Note

As this option can increase the average number of blocks allocated per user, it is disabled by default.

Here is an example to configure active block timeout.

vrouter running translate-to# active-block-timeout 60
vrouter running translate-to# commit

Endpoint mapping ¶

There are two endpoint mapping behaviors:

independent (default): The vRouter reuses the same port mapping for subsequent packets sent from the same internal IP address and port to any external IP address and port.

dependent: The vRouter reuses the same port mapping for subsequent packets sent from the same internal IP address and port to the same external IP address and port.

Here is an example to configure dependent endpoint mapping.

vrouter running translate-to# endpoint-mapping dependent
vrouter running translate-to# commit

Endpoint filtering ¶

There are two endpoint filtering behaviors:

independent (default): Inbound packets from external endpoints are only filtered out if their destination IP address and port don’t match an existing public IP address and port mapping.

dependent: Inbound packets from external endpoints are filtered out if they don’t match an existing mapping (source and destination IPs and ports, protocol).

Here is an example to configure dependent endpoint filtering.

vrouter running translate-to# endpoint-filtering dependent
vrouter running translate-to# commit

Hairpinning ¶

The hairpinning feature allows two endpoints (user 1 and user 2) on the private network to communicate together using their public IP addresses and ports.

By default, the hairpinning feature is disabled. To enable it, use the following command.

vrouter running translate-to# hairpinning true
vrouter running translate-to# commit

Conntracks ¶

It is possible to set conntrack timeouts for each protocol. UDP, ICMP and GRE protocols only handle basic conntrack states (new, established, closed), whereas TCP offers more granularity.

vrouter running config# vrf main cg-nat conntrack timeouts tcp
syn-sent        simsyn-sent     syn-received    established     fin-sent
fin-received    closed          close-wait      fin-wait        last-ack
time-wait

vrouter running config# vrf main cg-nat conntrack timeouts udp
new            established    closed

It is also possible to change TCP behavior for specific cases.

vrouter running config# vrf main cg-nat conntrack behavior
tcp-window-check        tcp-rst-strict-order

ALG ¶

Application-level gateways allow to use specific applications through CG-NAT.

vrouter running config# vrf main cg-nat alg
ftp          h323-q931    h323-ras     pptp         rtsp         sip-tcp
sip-udp      tftp

Summary ¶

The following table summarizes the different parameters for CG-NAT configuration.

Warning

Changing some configuration parameters requires to flush users. This is automatically done when required (see the table below), and causes a service interruption.

Category	Parameter	Flush	Note
pool	address	no	New addresses can be added without any impact. If an address is removed, all users assigned to it are flushed.
	block-size	yes
	port-range	yes
rule	match	no	Conntracks not matching the new criteria are not flushed. Conntracks matching the new criteria are flushed.
	pool-name	yes
	max-blocks-per-user	no	Extra blocks are not flushed for users having more blocks than the new max. They become inactive and will be flushed after all sessions are released.
	active-block-timeout	no	The new timeout starts after the current one expires.
	user-timeout	no	The new timeout starts after the current one expires.
	port-algo	no	For new blocks only.
	endpoint-mapping	no	For new mappings only.
	endpoint-filtering	no	For new mappings only.
	hairpinning	no	For new mappings only.

Logging ¶

You can enable logs for CG-NAT to track each port block allocation/deallocation for a user :

vrouter running config# vrf main cg-nat logging enabled true
commit

The logs can be displayed with the following command:

vrouter running config# show log service cgnat
Jun 11 08:02:46 vrouter systemd[1]: Started Fast Path cgnat log daemon.
Jun 11 08:02:46 vrouter fp-cgnat-logd[4269]: CGNAT Log listen on 5001
Jun 11 08:03:09 vrouter fp-cgnat-logd[4269]: USER 100.64.0.1 (matching rule 1): NEW BLOCK (pool "mypool", ip public 192.0.2.33, proto 6, port 1 - 512) at Tue Jun 11 08:03:09 2019
Jun 11 08:07:30 vrouter fp-cgnat-logd[4269]: USER 100.64.0.1 (matching rule 1): DESTROY BLOCK (pool "mypool", ip public 192.0.2.33, proto 6, port 1 - 512) at Tue Jun 11 08:07:30 2019

The following fields are displayed for each port block allocation/deallocation:

IP address of the user
name of the CG-NAT rule matched by the user
type of event: NEW BLOCK (a new port range is associated to this user) or DESTROY BLOCK (port range is released for this user).
pool and ip used to nat the user
port range of the block
l4 protocol (i.e. 6 for tcp, 17 for udp, 1 for icmp, 47 for gre)
timestamp of the event

Troubleshooting ¶

Invalid packet state statistics ¶

To display the CG-NAT statistics, use the following command.

vrouter> show cg-nat statistics
...
Invalid packet state cases:
...
   0 TCP case RST
...
   0 TCP case I
   0 TCP case II
   0 TCP case III
...

If the TCP case I, II or III statistics are incremented, you may disable TCP window checks as follows.

vrouter> edit running
vrouter running config# vrf main cg-nat conntrack
vrouter running conntrack# behavior tcp-window-check enabled false
vrouter running conntrack# commit

If the TCP case RST statistic is incremented, you may disable TCP RST strict ordering as follows.

vrouter> edit running
vrouter running config# vrf main cg-nat conntrack
vrouter running conntrack# behavior tcp-rst-strict-order enabled false
vrouter running conntrack# commit

Note

Disabling these features improves performance to the detriment of TCP robustness.

State/NAT/USER/Block Allocation Failures ¶

vrouter> show cg-nat statistics
...
State and NAT entries:
...
   0 state allocation failures
...
   0 NAT entry allocation failures
   0 NAT port allocation failures
CGNat entries:
...
   0 USER allocation failures
...
   0 Block allocation failures
...

If one of these statistics is incremented, it means that one of the memory pools of the vRouter is full. Memory pool usage can be dumped using the following command.

vrouter> show cg-nat mempool-usage
cgnat_user_pool : 2000/10000 (20.00%)
cgnat_block_pool : 8000/80000 (10.00%)
table_pool : 0/1056 (0.00%)
conn_pool : 1056736/1056736 (100.00%)
nat_pool : 1056736/1056736 (100.00%)

In the example above, the connection and NAT memory pools are full. Their size must be increased as follows.

vrouter running config# / system
fast-path limits cg-nat
vrouter running cg-nat# max-conntracks 2000000
vrouter running cg-nat# max-nat-entries 2000000
vrouter running cg-nat# commit

Refer to the capability tuning section.

No IP Public errors ¶

vrouter> show cg-nat statistics
...
CGNat entries:
...
   0 No IP Public
...

If this statistic is incremented, it means there are no blocks available in any public IP. This can be checked using the following command.

vrouter> show cg-nat pool-usage pool-name mypool
tcp block usage: 4095/4095 (100.0%)
udp block usage: 4095/4095 (100.0%)
icmp block usage: 4095/4095 (100.0%)
gre block usage: 4095/4095 (100.0%)

To solve this issue, add a new public IP to the pool using the following command.

vrouter> edit running
vourter running config# vrf main cg-nat pool mypool
vrouter running pool mypool#  address 32.96.120.0/24
vrouter running pool mypool# commit

NAT port allocation failures ¶

There are two main reasons for port allocation failures:

A user has consumed all its port blocks. The maximum number of blocks per user can be increased in the rule using the max-blocks-per-user command.
No blocks are available on the public IP allocated to the user. In this case, the Full IP Public statistic is also incremented.

To list users with allocation failures to understand how many users are impacted, use the following command.

vrouter> show cg-nat user rule-id 1 threshold-errors 1
100.64.0.1 -> 32.96.119.108
   2/2 tcp blocks, 0/2 udp blocks, 0/2 icmp blocks, 0/2 gre blocks
   63 no port errors, 0 no block errors, 0 full public ip errors

To understand why a specific user has many connections, use the following command.

vrouter> show cg-nat conntracks rule-id 1 user-address 100.64.0.1
CON:
   vrfid 0 flags 0x6 alg none tsdiff 47 timeout 120
   forw proto 6 100.64.0.1:1024-> 32.96.118.2:6001 hash:be3505a5
   back proto 6 32.96.118.2:6001-> 32.96.119.108:1216 hash:92e65736
   state 10:
      F { end 0 maxend 0 mwin 0 wscale 0 flags 1}
      T { end 0 maxend 0 mwin 0 wscale 0 flags 0}
   NAT: original address 100.64.0.1 proto 6 oport 1024 tport 1216
CON:
   vrfid 0 flags 0x6 alg none tsdiff 56 timeout 120
   forw proto 6 100.64.0.1:65024-> 32.96.118.2:6000 hash:913f8bf7
   back proto 6 32.96.118.2:6000-> 32.96.119.108:1024 hash:27051895
   state 10:
      F {end 0 maxend 0 mwin 0 wscale 0 flags 1}
      T {end 0 maxend 0 mwin 0 wscale 0 flags 0}
   NAT: original address 100.64.0.1 proto 6 oport 65024 tport 1024
...

Maximum number of blocks reached ¶

If the maximum number of blocks is reached, it probably means that you have not allocated enough blocks per user. You can collect some statistics to get average/percentile block and port usage of all users with the following commands.

vrouter> show cg-nat block-statistics  rule-id 1
block-usage:
   1 user (with > 1 block = 1, ratio 100.00%)
   blocks per user: min = 2, max = 2, average = 2.00
   1 user (100.00%) have 2 blocks
vrouter> show cg-nat port-statistics  rule-id 1
port-usage:
   1 user (with > 1 port = 1, ratio 100.00%)
   ports per user: min = 128, max = 128, average = 128.00
   1 user (100.00%) have 128 ports

You can also check the block usage and port usage of a specific user to get more details with the following commands.

vrouter> show cg-nat blocks rule-id 1 user-address 100.64.0.1
BLOCK: status active, proto 1 tports 1024 - 1031 parity=1, usage 1/8
vrouter> show cg-nat port-usage rule-id 1 user-address 100.64.0.1
tcp session usage: 0/16
udp session usage: 0/16
icmp session usage: 1/16
gre session usage: 0/16

Then, you can decide to increase the number of blocks per user or the block size. Refer to the Changing parameters section.

Full IP Public errors ¶

vrouter> show cg-nat statistics
...
CGNat entries:
...
   0 Full IP Public
...

The paired address pooling feature ensures the assignment of the same public IP address to all sessions originating from the same internal user, as described in RFC 4787 Req 2.

It means that when a user has started to use one public IP address, all its port blocks will be allocated from this same IP. Adding a new public IP to the pool won’t solve the issue, as the user cannot allocate a block from a new public IP.

A possible way to recover such situation is to add new IP address to the pool, and then flush all the current connections of all users, as follows.

vrouter running config# / vrf main cg-nat pool mypool
vrouter running pool mypool#  address 32.96.120.0/24
vrouter running pool mypool# commit
Configuration committed.

vrouter running pool mypool# flush cg-nat user rule-id 1

Dimensioning ¶

The maximum numbers for NAT entries, CPEs (users), conntracks (sessions), blocks and block sizes are defined in the configuration. These limits can be adjusted to adapt to the amount of memory available in the system.

vrouter running config# system fast-path limits cg-nat
max-conntracks     max-nat-entries    max-users          max-blocks
max-block-size

The following table shows a list of different limit combinations and the corresponding memory requirement. This is empirical and may have to be tuned according to your use case.

Max conntracks	Max nat entries	Max cpe	Max blocks	Required memory
1M	1M	10K	80K	5 GB
2M	2M	20K	80K	6 GB
4M	4M	20K	80K	8 GB
8M	8M	20K	80K	12 GB
16M	16M	20K	80K	24 GB
30M	30M	20K	80K	32 GB

Warning

Changing these values triggers a restart of the fast-path (hence, a service interruption). Therefore you should carefully think about your dimensioning before launching your Turbo CG-NAT into production.

Modifying these limits will automatically restart the fast path and interrupt packet processing. To check that the fast path is back up and running, use the following command.

vrouter running config# show state system fast-path
fast-path
   enabled stopping
   ..
vrouter running config#  show state system fast-path
fast-path
   enabled starting
   ..
vrouter running config# # show state system fast-path
fast-path
   enabled true
   ...