6. Troubleshooting

6.1. Invalid packet state statistics

To display the CG-NAT statistics, use the following command.

vrouter> show cg-nat statistics
...
Invalid packet state cases:
...
   0 TCP case RST
...
   0 TCP case I
   0 TCP case II
   0 TCP case III
...

If the TCP case I, II or III statistics are incremented, you may disable TCP window checks as follows.

vrouter> edit running
vrouter running config# vrf main cg-nat conntrack
vrouter running conntrack# behavior tcp-window-check enabled false
vrouter running conntrack# commit

If the TCP case RST statistic is incremented, you may disable TCP RST strict ordering as follows.

vrouter> edit running
vrouter running config# vrf main cg-nat conntrack
vrouter running conntrack# behavior tcp-rst-strict-order enabled false
vrouter running conntrack# commit

Note

Disabling these features improves performance at the expense of TCP robustness.

6.2. State/NAT/USER/Block Allocation Failures

vrouter> show cg-nat statistics
...
State and NAT entries:
...
   0 state allocation failures
...
   0 NAT entry allocation failures
   0 NAT port allocation failures
CGNat entries:
...
   0 USER allocation failures
...
   0 Block allocation failures
...

If one of these statistics is incremented, it means that one of the memory pools of the vRouter is full. Memory pool usage can be dumped using the following command.

vrouter> show cg-nat mempool-usage
cgnat_user_pool : 2000/10000 (20.00%)
cgnat_block_pool : 8000/80000 (10.00%)
table_pool : 0/1056 (0.00%)
conn_pool : 1056736/1056736 (100.00%)
nat_pool : 1056736/1056736 (100.00%)

In the example above, the connection and NAT memory pools are full. Their size must be increased as follows.

vrouter running config# / system fast-path limits cg-nat
vrouter running cg-nat# max-conntracks 2000000
vrouter running cg-nat# max-nat-entries 2000000
vrouter running cg-nat# commit

Refer to the capability tuning section.

6.3. No IP Public errors

vrouter> show cg-nat statistics
...
CGNat entries:
...
   0 No IP Public
...

If this statistic is incremented, it means there are no blocks available in any public IP. This can be checked using the following command.

vrouter> show cg-nat pool-usage pool-name mypool
tcp block usage: 4095/4095 (100.0%)
udp block usage: 4095/4095 (100.0%)
icmp block usage: 4095/4095 (100.0%)
gre block usage: 4095/4095 (100.0%)

To solve this issue, add a new public IP to the pool using the following command.

vrouter> edit running
vourter running config# vrf main cg-nat pool mypool
vrouter running pool mypool#  address 32.96.120.0/24
vrouter running pool mypool# commit

6.4. NAT port allocation failures

There are two main reasons for port allocation failures:

• A user has consumed all its port blocks. The maximum number of blocks per user can be increased in the rule using the max-blocks-per-user command.

• No blocks are available on the public IP allocated to the user. In this case, the Full IP Public statistic is also incremented.

To list users with allocation failures to understand how many users are impacted, use the following command.

vrouter> show cg-nat user rule-id 1 threshold-errors 1
2001:db8:200::1 -> 32.96.119.108
   256 conntracks
   2/2 tcp blocks, 0/2 udp blocks, 0/2 icmp blocks, 0/2 gre blocks
   63 no port errors, 0 no block errors, 0 full public ip errors

To understand why a specific user has many connections, use the following command.

vrouter> show cg-nat conntracks rule-id 1 user-address 2001:db8:200::1
CONN:
        vrfid 0 flags 0x6 alg none tsdiff 87 timeout 120
        forw proto 6 2001:0db8:0200::0001.45781-> 0064:ff9b::2060:7603.80 hash:7f25564e
        back proto 6 32.96.118.3:80-> 32.96.119.0:1030 hash:9f8a5d03
        state time_wait:
                F { end 704685151 maxend 704750431 mwin 64896 wscale 7 flags 1}
                T { end 1624303497 maxend 1624368393 mwin 65280 wscale 7 flags 0}
        NAT: original address 2001:0db8:0200::0001 proto 6 oport 45781 tport 1030
CONN:
        vrfid 0 flags 0x6 alg none tsdiff 89 timeout 120
        forw proto 6 2001:0db8:0200::0001.54651-> 0064:ff9b::2060:760f.80 hash:9b0932a7
        back proto 6 32.96.118.15:80-> 32.96.119.0:1026 hash:f5dbf6b1
        state time_wait:
                F { end 3995675323 maxend 3995740603 mwin 64896 wscale 7 flags 1}
                T { end 1877109092 maxend 1877173988 mwin 65280 wscale 7 flags 0}
        NAT: original address 2001:0db8:0200::0001 proto 6 oport 54651 tport 1026
...

6.5. Maximum number of blocks reached

If the maximum number of blocks is reached, it probably means that you have not allocated enough blocks per user. You can collect some statistics to get average/percentile block and port usage of all users with the following commands.

vrouter> show cg-nat block-statistics  rule-id 1
block-usage:
   1 user (with > 1 block = 1, ratio 100.00%)
   blocks per user: min = 2, max = 2, average = 2.00
   1 user (100.00%) have 2 blocks
vrouter> show cg-nat port-statistics  rule-id 1
port-usage:
   1 user (with > 1 port = 1, ratio 100.00%)
   ports per user: min = 128, max = 128, average = 128.00
   1 user (100.00%) have 128 ports

Then, you can decide to increase the number of blocks per user or the block size. Refer to the Changing parameters section.

6.6. Full IP Public errors

vrouter> show cg-nat statistics
...
CGNat entries:
...
   0 Full IP Public
...

The paired address pooling feature ensures the assignment of the same public IP address to all sessions originating from the same internal user, as described in RFC 4787 Req 2.

It means that when a user has started to use one public IP address, all its port blocks will be allocated from this same IP. Adding a new public IP to the pool won’t solve the issue, as the user cannot allocate a block from a new public IP.

A possible way to recover such situation is to add new IP address to the pool, and then flush all the current connections of all users, as follows.

vrouter running config# / vrf main cg-nat pool mypool
vrouter running pool mypool#  address 32.96.120.0/24
vrouter running pool mypool# commit
Configuration committed.

vrouter running pool mypool# flush cg-nat user rule-id 1

See also

See the User’s Guide for more information regarding:

• CG-NAT troubleshooting

6.7. Udp null checksum packet drop

By default, IPv4 UDP packets that don’t contain a UDP checksum (i.e. the UDP checksum field is zero) are dropped when they are translated into IPv6.

The ‘udp null checksum packet drop’ statistic in NAT64 section can be checked to detect packets dropped in this case.

vrouter> show cg-nat statistics
...
NAT64 Stats:
 19373 udp null checksum packet drop

It’s possible to not drop these packets and translate them by disabling an option.

vrouter> edit running
vrouter running config# vrf main cg-nat nat64
vrouter running conntrack# option drop-udp-zero-checksum enabled false
vrouter running conntrack# commit

Disabling this option has a performance cost. As udp null checksum is not allowed in IPv6, a new checksum needs to be computed.