6. Troubleshooting¶
6.1. Invalid packet state statistics¶
To display the CG-NAT statistics, use the following command.
vrouter> show cg-nat statistics
...
Invalid packet state cases:
...
0 TCP case RST
...
0 TCP case I
0 TCP case II
0 TCP case III
...
If the TCP case I, II or III statistics are incremented, you may disable TCP window checks as follows.
vrouter> edit running
vrouter running config# vrf main cg-nat conntrack
vrouter running conntrack# behavior tcp-window-check enabled false
vrouter running conntrack# commit
If the TCP case RST statistic is incremented, you may disable TCP RST strict ordering as follows.
vrouter> edit running
vrouter running config# vrf main cg-nat conntrack
vrouter running conntrack# behavior tcp-rst-strict-order enabled false
vrouter running conntrack# commit
Note
Disabling these features improves performance at the expense of TCP robustness.
6.2. State/NAT/USER/Block Allocation Failures¶
vrouter> show cg-nat statistics
...
State and NAT entries:
...
0 state allocation failures
...
0 NAT entry allocation failures
0 NAT port allocation failures
CGNat entries:
...
0 USER allocation failures
...
0 Block allocation failures
...
If one of these statistics is incremented, it means that one of the memory pools of the vRouter is full. Memory pool usage can be dumped using the following command.
vrouter> show cg-nat mempool-usage
cgnat_user_pool : 2000/10000 (20.00%)
cgnat_block_pool : 8000/80000 (10.00%)
table_pool : 0/1056 (0.00%)
conn_pool : 1056736/1056736 (100.00%)
nat_pool : 1056736/1056736 (100.00%)
In the example above, the connection and NAT memory pools are full. Their size must be increased as follows.
vrouter running config# / system fast-path limits cg-nat
vrouter running cg-nat# max-conntracks 2000000
vrouter running cg-nat# max-nat-entries 2000000
vrouter running cg-nat# commit
Refer to the capability tuning section.
6.3. No IP Public errors¶
vrouter> show cg-nat statistics
...
CGNat entries:
...
0 No IP Public
...
If this statistic is incremented, it means there are no blocks available in any public IP. This can be checked using the following command.
vrouter> show cg-nat pool-usage pool-name mypool
tcp block usage: 4095/4095 (100.0%)
udp block usage: 4095/4095 (100.0%)
icmp block usage: 4095/4095 (100.0%)
gre block usage: 4095/4095 (100.0%)
To solve this issue, add a new public IP to the pool using the following command.
vrouter> edit running
vourter running config# vrf main cg-nat pool mypool
vrouter running pool mypool# address 32.96.120.0/24
vrouter running pool mypool# commit
6.4. NAT port allocation failures¶
There are two main reasons for port allocation failures:
A user has consumed all its port blocks. The maximum number of blocks per user can be increased in the rule using the max-blocks-per-user command.
No blocks are available on the public IP allocated to the user. In this case, the Full IP Public statistic is also incremented.
To list users with allocation failures to understand how many users are impacted, use the following command.
vrouter> show cg-nat user rule-id 1 threshold-errors 1
100.64.0.1 -> 32.96.119.108
256 conntracks
2/2 tcp blocks, 0/2 udp blocks, 0/2 icmp blocks, 0/2 gre blocks
63 no port errors, 0 no block errors, 0 full public ip errors
To understand why a specific user has many connections, use the following command.
vrouter> show cg-nat conntracks rule-id 1 user-address 100.64.0.1
CONN:
vrfid 0 flags 0x6 alg none tsdiff 6 timeout 7440
forw proto 6 100.64.0.1:35733-> 32.96.118.2:80 hash:dc99ddf3
back proto 6 32.96.118.2:80-> 32.96.119.0:1027 hash:83dbb102
state established:
F { end 2701615772 maxend 2701680924 mwin 64256 wscale 7 flags 0}
T { end 3527341170 maxend 3527405426 mwin 65160 wscale 7 flags 0}
NAT: original address 100.64.0.1 proto 6 oport 35733 tport 1027
CONN:
vrfid 0 flags 0x6 alg none tsdiff 12 timeout 120
forw proto 6 100.64.0.1:48299-> 32.96.118.10:80 hash:aad3919f
back proto 6 32.96.118.10:80-> 32.96.119.0:1025 hash:e1de481c
state time_wait:
F { end 4165678865 maxend 4165744145 mwin 64256 wscale 7 flags 1}
T { end 1338325493 maxend 1338389749 mwin 65280 wscale 7 flags 0}
NAT: original address 100.64.0.1 proto 6 oport 48299 tport 1025
...
6.5. Maximum number of blocks reached¶
If the maximum number of blocks is reached, it probably means that you have not allocated enough blocks per user. You can collect some statistics to get average/percentile block and port usage of all users with the following commands.
vrouter> show cg-nat block-statistics rule-id 1
block-usage:
1 user (with > 1 block = 1, ratio 100.00%)
blocks per user: min = 2, max = 2, average = 2.00
1 user (100.00%) have 2 blocks
vrouter> show cg-nat port-statistics rule-id 1
port-usage:
1 user (with > 1 port = 1, ratio 100.00%)
ports per user: min = 128, max = 128, average = 128.00
1 user (100.00%) have 128 ports
Then, you can decide to increase the number of blocks per user or the block size. Refer to the Changing parameters section.
6.6. Full IP Public errors¶
vrouter> show cg-nat statistics
...
CGNat entries:
...
0 Full IP Public
...
The paired address pooling feature ensures the assignment of the same public IP address to all sessions originating from the same internal user, as described in RFC 4787 Req 2.
It means that when a user has started to use one public IP address, all its port blocks will be allocated from this same IP. Adding a new public IP to the pool won’t solve the issue, as the user cannot allocate a block from a new public IP.
A possible way to recover such situation is to add new IP address to the pool, and then flush all the current connections of all users, as follows.
vrouter running config# / vrf main cg-nat pool mypool
vrouter running pool mypool# address 32.96.120.0/24
vrouter running pool mypool# commit
Configuration committed.
vrouter running pool mypool# flush cg-nat user rule-id 1