5. Troubleshooting¶
5.1. CLI show commands¶
The CLI incorporates a number of show commands of which a few are shown here.
Showing the current basic state of all interfaces (add a command qualifier for more details):
border1-vm running config# show interface
Name State IP Addresses
---- ----- ------------
lo UNKNOWN 127.0.0.1/8
::1/128
loopback0 UNKNOWN 172.16.200.1/32
fe80::f897:62ff:fea0:d6d8/64
ntfp1 UP 172.16.100.1/24
fe80::dced:1ff:fe43:f429/64
ntfp2 UP fe80::dced:1ff:fe9a:8f9a/64
ntfp3 UP fe80::dced:1ff:fe9b:913e/64
vrrp_internal@ntfp1 UP 172.16.100.5/24
fe80::200:5eff:fe00:1c8/64
vlan3@ntfp2 UP 3.3.3.2/24
fe80::dced:1ff:fe9a:8f9a/64
vrrp3@vlan3 UP 3.3.3.4/24
fe80::200:5eff:fe00:103/64
vlan2@ntfp3 UP 2.2.2.2/24
fe80::dced:1ff:fe9b:913e/64
vrrp2@vlan2 UP 2.2.2.4/24
fe80::200:5eff:fe00:102/64
vlan1@ntfp3 UP 1.1.1.2/24
fe80::dced:1ff:fe9b:913e/64
vrrp1@vlan1 UP 1.1.1.4/24
fe80::200:5eff:fe00:101/64
Showing the detailed state of one particular interface: ntfp1
border1-vm running config# show interface name ntfp1 details
10: ntfp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether de:ed:01:43:f4:29 brd ff:ff:ff:ff:ff:ff
inet 172.16.100.1/24 scope global ntfp1
valid_lft forever preferred_lft forever
inet6 fe80::dced:1ff:fe43:f429/64 scope link
valid_lft forever preferred_lft forever
Basic interface UDP traffic dump example:
border1> cmd traffic-capture ntfp1 filter udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ntfp1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:02:50.846874 de:ed:01:43:f4:29 > de:ed:01:ef:eb:01, ethertype IPv4 (0x0800), length 206: 172.16.100.1.55536 > 172.16.100.254.6343: sFlowv5, IPv4 agent 172.16.200.1, agent-id 100000, length 164
16:02:50.846890 de:ed:01:43:f4:29 > de:ed:01:94:12:c2, ethertype IPv4 (0x0800), length 206: 172.16.100.1.55536 > 172.16.100.253.6343: sFlowv5, IPv4 agent 172.16.200.1, agent-id 100000, length 164
16:02:53.245215 de:ed:01:43:f4:29 > de:ed:01:ef:eb:01, ethertype IPv4 (0x0800), length 206: 172.16.100.1.55536 > 172.16.100.254.6343: sFlowv5, IPv4 agent 172.16.200.1, agent-id 100000, length 164
16:02:53.245240 de:ed:01:43:f4:29 > de:ed:01:94:12:c2, ethertype IPv4 (0x0800), length 206: 172.16.100.1.55536 > 172.16.100.253.6343: sFlowv5, IPv4 agent 172.16.200.1, agent-id 100000, length 164
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
The first obvious choice to troubleshoot connectivity problems is to verify that all the routes are in the routing table using the following command:
border1> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
O 1.1.1.0/24 [110/20] via 172.16.100.2, ntfp1, 00:18:05
C * 1.1.1.0/24 is directly connected, vrrp1, 00:59:53
C>* 1.1.1.0/24 is directly connected, vlan1, 00:59:58
O 2.2.2.0/24 [110/20] via 172.16.100.2, ntfp1, 00:18:05
C * 2.2.2.0/24 is directly connected, vrrp2, 00:59:53
C>* 2.2.2.0/24 is directly connected, vlan2, 00:59:58
O 3.3.3.0/24 [110/20] via 172.16.100.2, ntfp1, 00:18:05
C * 3.3.3.0/24 is directly connected, vrrp3, 00:59:53
C>* 3.3.3.0/24 is directly connected, vlan3, 00:59:59
O 172.16.100.0/24 [110/10] is directly connected, ntfp1, 00:18:06
C * 172.16.100.0/24 is directly connected, vrrp_internal, 00:59:53
C>* 172.16.100.0/24 is directly connected, ntfp1, 00:59:59
C>* 172.16.200.1/32 is directly connected, loopback0, 01:00:06
O>* 172.16.200.2/32 [110/20] via 172.16.100.2, ntfp1, 00:18:05
O>* 172.16.200.3/32 [110/20] via 172.16.100.3, ntfp1, 00:18:05
O>* 172.16.200.4/32 [110/20] via 172.16.100.4, ntfp1, 00:18:05
B>* 200.200.210.0/24 [110/20] via 172.16.200.3 (recursive), 00:18:05
* via 172.16.100.3, ntfp1, 00:18:05
B>* 200.200.221.0/24 [110/20] via 172.16.200.4 (recursive), 00:18:05
* via 172.16.100.4, ntfp1, 00:18:05
B>* 217.151.210.0/24 [20/0] via 1.1.1.1, vlan1, 00:59:55
B>* 217.151.211.0/24 [20/0] via 2.2.2.1, vlan2, 00:59:55
B>* 217.151.212.0/24 [20/0] via 3.3.3.1, vlan3, 00:59:54
Refining the show command, we can first look at the OSPF routes:
border1> show ospf route
============ OSPF network routing table ============
N 172.16.100.0/24 [10] area: 0.0.0.0
directly attached to ntfp1
directly attached to vrrp_internal
============ OSPF router routing table =============
R 172.16.200.2 [10] area: 0.0.0.0, ASBR
via 172.16.100.2, ntfp1
R 172.16.200.3 [10] area: 0.0.0.0, ASBR
via 172.16.100.3, ntfp1
R 172.16.200.4 [10] area: 0.0.0.0, ASBR
via 172.16.100.4, ntfp1
============ OSPF external routing table ===========
N E2 1.1.1.0/24 [10/20] tag: 0
via 172.16.100.2, ntfp1
N E2 2.2.2.0/24 [10/20] tag: 0
via 172.16.100.2, ntfp1
N E2 3.3.3.0/24 [10/20] tag: 0
via 172.16.100.2, ntfp1
N E2 172.16.200.2/32 [10/20] tag: 0
via 172.16.100.2, ntfp1
N E2 172.16.200.3/32 [10/20] tag: 0
via 172.16.100.3, ntfp1
N E2 172.16.200.4/32 [10/20] tag: 0
via 172.16.100.4, ntfp1
If OSPF routes seem to be missing, try verifying that OSPF has formed the correct neighbor relationships:
border1> show ospf neighbor
Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL
172.16.200.2 1 2-Way/DROther 38.240s 172.16.100.2 ntfp1:172.16.100.1 0 0 0
172.16.200.3 1 Full/Backup 35.259s 172.16.100.3 ntfp1:172.16.100.1 0 0 0
172.16.200.4 1 Full/DR 39.753s 172.16.100.4 ntfp1:172.16.100.1 0 0 0
And we can also verify the OSPF topology database:
border1> show ospf database
OSPF Router with ID (172.16.200.1)
Router Link States (Area 0.0.0.0)
Link ID ADV Router Age Seq# CkSum Link count
172.16.200.1 172.16.200.1 681 0x80000197 0x091b 2
172.16.200.2 172.16.200.2 644 0x80000196 0x1e3f 1
172.16.200.3 172.16.200.3 914 0x80000194 0x203c 1
172.16.200.4 172.16.200.4 122 0x80000194 0x1e3b 1
Net Link States (Area 0.0.0.0)
Link ID ADV Router Age Seq# CkSum
172.16.100.4 172.16.200.4 1441 0x80000193 0xcd28
AS External Link States
Link ID ADV Router Age Seq# CkSum Route
1.1.1.0 172.16.200.1 297 0x80000192 0x69c7 E2 1.1.1.0/24 [0x0]
1.1.1.0 172.16.200.2 269 0x80000191 0x65cb E2 1.1.1.0/24 [0x0]
2.2.2.0 172.16.200.1 377 0x80000192 0x45e8 E2 2.2.2.0/24 [0x0]
2.2.2.0 172.16.200.2 319 0x80000191 0x41ec E2 2.2.2.0/24 [0x0]
3.3.3.0 172.16.200.1 377 0x80000192 0x210a E2 3.3.3.0/24 [0x0]
3.3.3.0 172.16.200.2 329 0x80000191 0x1d0e E2 3.3.3.0/24 [0x0]
172.16.200.1 172.16.200.1 1271 0x80000191 0x41de E2 172.16.200.1/32 [0x0]
172.16.200.2 172.16.200.2 884 0x80000191 0x31ec E2 172.16.200.2/32 [0x0]
172.16.200.3 172.16.200.3 1304 0x80000191 0x21fa E2 172.16.200.3/32 [0x0]
172.16.200.4 172.16.200.4 392 0x80000191 0x1109 E2 172.16.200.4/32 [0x0]
If 2-way and FULL states have not been established between the OSPF neighbors, check that all OSPF interface settings are correct. All usual OSPF neighborship requirements must be fulfilled.
The next step would be to enable OSPF logging as shown under the CLI log commands section.
Now, let’s check BGP.
Verify the BGP routes:
border1> show bgp ipv4
BGP table version is 13, local router ID is 172.16.200.1, vrf id 0
Default local pref 100, local AS 65200
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i200.200.210.0/24 172.16.200.3 0 100 0 i
*>i200.200.220.0/24 172.16.200.4 0 100 0 i
* i217.151.210.0/24 1.1.1.1 0 100 0 100 100 i
*> 1.1.1.1 0 0 100 100 i
* i217.151.211.0/24 2.2.2.1 0 100 0 200 200 200 i
*> 2.2.2.1 0 0 200 200 200 i
* i217.151.212.0/24 3.3.3.1 0 100 0 300 i
*> 3.3.3.1 0 0 300 i
Displayed 5 routes and 8 total paths
Let’s check BGP neighbors; in this example just the Transit_3 neighbor for brevity:
border1> show bgp neighbor 3.3.3.1
BGP neighbor is 3.3.3.1, remote AS 300, local AS 65200, external link
Description: Transit3-IPv4
Hostname: transit3-vm
BGP version 4, remote router ID 7.7.7.7, local router ID 172.16.200.1
BGP state = Established, up for 00:30:02
Last read 00:00:02, Last write 00:00:02
Hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
AddPath:
IPv4 Unicast: RX advertised IPv4 Unicast and received
Route refresh: advertised and received(old & new)
Address Family IPv4 Unicast: advertised and received
Address Family IPv6 Unicast: received
Hostname Capability: advertised (name: border1,domain name: n/a) received (name: transit3-vm,domain name: n/a)
Graceful Restart Capabilty: advertised and received
Remote Restart timer is 120 seconds
Address families by peer:
none
Graceful restart informations:
End-of-RIB send: IPv4 Unicast
End-of-RIB received: IPv4 Unicast
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 3 4
Keepalives: 31 31
Route Refresh: 0 0
Capability: 0 0
Total: 35 36
Minimum time between advertisement runs is 0 seconds
For address family: IPv4 Unicast
Update group 3, subgroup 3
Packet Queue length 0
Inbound soft reconfiguration allowed
Community attribute sent to this neighbor(all)
Inbound path policy configured
Outbound path policy configured
Incoming update prefix filter list is *any_except_bogons
Route map for outgoing advertisements is *set_nexthop_vip_transit3
1 accepted prefixes
Connections established 1; dropped 0
Last reset 01:47:02, Waiting for peer OPEN
Local host: 3.3.3.2, Local port: 40048
Foreign host: 3.3.3.1, Foreign port: 179
Nexthop: 3.3.3.2
Nexthop global: fe80::dced:1ff:fe9a:8f9a
Nexthop local: fe80::dced:1ff:fe9a:8f9a
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 120
Read thread: on Write thread: on FD used: 29
Verify BGP flowspec (so far in this case nothing to show):
border1> show bgp ipv4 flowspec
BGP table version is 1, local router ID is 172.16.200.1, vrf id 0
Default local pref 100, local AS 65200
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i to 8.8.8.8/32 from 200.200.208.0/20 1.1.1.1 100 0 i
Displayed 1 routes and 1 total paths
Many more show commands are available, please check in the User’s Guide as appropriate.
5.2. CLI log commands¶
To display the system log locally (kernel logs in this case):
border1> show log facility kernel
-- Logs begin at Tue 2020-07-09 14:37:46 UTC, end at Tue 2020-07-09 21:03:52 UTC. --
Jul 09 14:40:24 border1 kernel: Silicon Labs C2 port support v. 0.51.0 - (C) 2007 Rodolfo Giometti
Jul 09 14:40:31 border1 kernel: VFIO - User Level meta-driver version: 0.3
Jul 09 14:40:32 border1 kernel: iommu: Adding device 0000:00:04.0 to group 0
Jul 09 14:40:32 border1 kernel: vfio-pci 0000:00:04.0: Adding kernel taint for vfio-noiommu group on device
Jul 09 14:40:32 border1 kernel: iommu: Adding device 0000:00:05.0 to group 1
Jul 09 14:40:32 border1 kernel: vfio-pci 0000:00:05.0: Adding kernel taint for vfio-noiommu group on device
Jul 09 14:40:32 border1 kernel: iommu: Adding device 0000:00:06.0 to group 2
Jul 09 14:40:32 border1 kernel: vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
Jul 09 14:40:33 border1 kernel: dpvi: loading out-of-tree module taints kernel.
Jul 09 14:40:33 border1 kernel: dpvi: module verification failed: signature and/or required key missing - tainting kernel
Jul 09 14:40:33 border1 kernel: dpvi_shmem: dpvi_shmem module initialized 00000000bfa363e7
To specifically look at routing system (BGP, OSPF,..) events:
border1> show log service routing
-- Logs begin at Fri 2020-07-26 09:16:24 UTC, end at Fri 2020-07-26 09:47:01 UTC. --
Jul 26 09:18:54 border1 systemd[1]: Starting zebra...
Jul 26 09:18:54 border1 systemd[1]: Started zebra.
Jul 26 09:18:54 border1 systemd[1]: Starting staticd...
Jul 26 09:18:54 border1 systemd[1]: Started staticd.
Jul 26 09:18:54 border1 systemd[1]: Starting bgpd..
Jul 26 09:19:13 border1 systemd[1]: Started bgpd.
Jul 26 09:18:54 border1 systemd[1]: Starting ospfd...
Jul 26 09:19:13 border1 systemd[1]: Started ospfd.
Logging of BGP neighbor changes:
border1> edit running
border1 running config# / vrf main routing bgp
border1 running bgp# log-neighbor-changes true
A per VRF remote logging capability can be enabled for the system log:
border1> edit running
border1 running config# / vrf main logging syslog
border1 running syslog#! remote-server 172.16.100.253 protocol tcp port 514
border1 running syslog# commit