5. Troubleshooting¶
5.1. CLI show commands¶
The CLI incorporates a number of show commands of which a few are shown here.
Showing the current basic state of all interfaces (add a command qualifier for more details):
border1-vm running config# show interface
Name State IP Addresses
---- ----- ------------
lo UNKNOWN 127.0.0.1/8
::1/128
ens3 UP 10.0.2.15/24
fe80::dcad:deff:fe01:203/64
loopback0 UNKNOWN 172.16.200.1/32
fe80::a060:efff:fe07:1acc/64
ntfp1 UP 172.16.100.1/24
fe80::dced:1ff:fe5d:87a4/64
ntfp2 UP fe80::dced:1ff:fe03:de92/64
ntfp3 UP fe80::dced:1ff:fe98:20f7/64
vlan3@ntfp2 UP 3.3.3.2/24
fe80::dced:1ff:fe03:de92/64
vlan1@ntfp3 UP 1.1.1.2/24
fe80::dced:1ff:fe98:20f7/64
vlan2@ntfp3 UP 2.2.2.2/24
fe80::dced:1ff:fe98:20f7/64
vrrp1@vlan1 UP 1.1.1.4/24
vrrp2@vlan2 UP 2.2.2.4/24
vrrp3@vlan3 UP 3.3.3.4/24
vrrp_internal@ntfp1 UP 172.16.100.5/24
Showing the detailed state of one particular interface: ntfp1
border1-vm running config# show interface name ntfp1 details
10: ntfp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether de:ed:01:5d:87:a4 brd ff:ff:ff:ff:ff:ff
inet 172.16.100.1/24 scope global ntfp1
valid_lft forever preferred_lft forever
inet6 fe80::dced:1ff:fe5d:87a4/64 scope link
valid_lft forever preferred_lft forever
Basic interface UDP traffic dump example:
border1> cmd show-traffic ntfp1 filter udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ntfp1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:38:47.221472 de:ed:01:e3:55:78 > de:ed:01:07:da:e2, ethertype IPv4 (0x0800), length 746: 172.16.100.2.45791 > 172.16.100.254.6343: sFlowv5, IPv4 agent 172.16.200.2, agent-id 100000, length 704
18:38:47.221482 de:ed:01:e3:55:78 > de:ed:01:07:da:e2, ethertype IPv4 (0x0800), length 746: 172.16.100.2.45791 > 172.16.100.254.6343: sFlowv5, IPv4 agent 172.16.200.2, agent-id 100000, length 704
18:38:47.221484 de:ed:01:e3:55:78 > de:ed:01:1b:a5:56, ethertype IPv4 (0x0800), length 746: 172.16.100.2.45791 > 172.16.100.253.6343: sFlowv5, IPv4 agent 172.16.200.2, agent-id 100000, length 704
18:38:47.221485 de:ed:01:e3:55:78 > de:ed:01:1b:a5:56, ethertype IPv4 (0x0800), length 746: 172.16.100.2.45791 > 172.16.100.253.6343: sFlowv5, IPv4 agent 172.16.200.2, agent-id 100000, length 704
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
The first obvious choice to troubleshoot connectivity problems is to verify that all the routes are in the routing table using the following command:
border1> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route
VRF main:
K>* 0.0.0.0/0 [0/0] via 10.0.2.2, ens3, 06:22:10
C * 1.1.1.0/24 is directly connected, vrrp1, 06:21:53
C>* 1.1.1.0/24 is directly connected, vlan1, 06:21:58
C * 2.2.2.0/24 is directly connected, vrrp2, 06:21:53
C>* 2.2.2.0/24 is directly connected, vlan2, 06:21:58
C * 3.3.3.0/24 is directly connected, vrrp3, 06:21:53
C>* 3.3.3.0/24 is directly connected, vlan3, 06:21:58
C>* 10.0.2.0/24 is directly connected, ens3, 06:22:10
O 172.16.100.0/24 [110/100] is directly connected, ntfp1, 06:21:11
is directly connected, vrrp_internal, 06:21:11
C * 172.16.100.0/24 is directly connected, vrrp_internal, 06:21:53
C>* 172.16.100.0/24 is directly connected, ntfp1, 06:21:58
C>* 172.16.200.1/32 is directly connected, loopback0, 06:22:08
B 172.16.200.2/32 [200/0] via 172.16.200.2, 06:21:04
O>* 172.16.200.2/32 [110/20] via 172.16.100.2, ntfp1, 06:21:10
* via 172.16.100.2, vrrp_internal, 06:21:10
B 172.16.200.3/32 [200/0] via 172.16.200.3, 06:21:04
O>* 172.16.200.3/32 [110/20] via 172.16.100.3, ntfp1, 06:21:05
* via 172.16.100.3, vrrp_internal, 06:21:05
B 172.16.200.4/32 [200/0] via 172.16.200.4, 06:21:09
O>* 172.16.200.4/32 [110/20] via 172.16.100.4, ntfp1, 06:21:10
* via 172.16.100.4, vrrp_internal, 06:21:10
B> 200.200.210.0/24 [200/0] via 172.16.200.3 (recursive), 06:21:04
* via 172.16.100.3, ntfp1, 06:21:04
* via 172.16.100.3, vrrp_internal, 06:21:04
B> 200.200.220.0/24 [200/0] via 172.16.200.4 (recursive), 06:21:09
* via 172.16.100.4, ntfp1, 06:21:09
* via 172.16.100.4, vrrp_internal, 06:21:09
B>* 217.151.210.0/24 [20/0] via 1.1.1.1, vlan1, 06:21:54
B>* 217.151.211.0/24 [20/0] via 2.2.2.1, vlan2, 06:21:54
B>* 217.151.212.0/24 [20/0] via 3.3.3.1, vlan3, 06:21:54
Refining the show command, we can first look at the OSPF routes:
border1> show ospf route
VRF Name: default
============ OSPF network routing table ============
N 172.16.100.0/24 [100] area: 0.0.0.0
directly attached to ntfp1
directly attached to vrrp_internal
============ OSPF router routing table =============
R 172.16.200.2 [100] area: 0.0.0.0, ASBR
via 172.16.100.2, ntfp1
via 172.16.100.2, vrrp_internal
R 172.16.200.3 [100] area: 0.0.0.0, ASBR
via 172.16.100.3, ntfp1
via 172.16.100.3, vrrp_internal
R 172.16.200.4 [100] area: 0.0.0.0, ASBR
via 172.16.100.4, ntfp1
via 172.16.100.4, vrrp_internal
============ OSPF external routing table ===========
N E2 172.16.200.2/32 [100/20] tag: 0
via 172.16.100.2, ntfp1
via 172.16.100.2, vrrp_internal
N E2 172.16.200.3/32 [100/20] tag: 0
via 172.16.100.3, ntfp1
via 172.16.100.3, vrrp_internal
N E2 172.16.200.4/32 [100/20] tag: 0
via 172.16.100.4, ntfp1
via 172.16.100.4, vrrp_internal
If OSPF routes seem to be missing, try verifying that OSPF has formed the correct neighbor relationships:
border1> show ospf neighbor
VRF Name: default
Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL
172.16.200.2 1 2-Way/DROther 36.233s 172.16.100.2 ntfp1:172.16.100.1 0 0 0
172.16.200.3 1 Full/Backup 34.142s 172.16.100.3 ntfp1:172.16.100.1 0 0 0
172.16.200.4 1 Full/DR 33.873s 172.16.100.4 ntfp1:172.16.100.1 0 0 0
172.16.200.2 1 ExStart/DR 32.820s 172.16.100.2 vrrp_internal:172.16.100.5 0 0 0
172.16.200.3 1 2-Way/DROther 31.615s 172.16.100.3 vrrp_internal:172.16.100.5 0 0 0
172.16.200.4 1 Full/Backup 33.979s 172.16.100.4 vrrp_internal:172.16.100.5 0 0 0
And we can also verify the OSPF topology database:
border1> show ospf database
VRF Name: default
OSPF Router with ID (172.16.200.1)
Router Link States (Area 0.0.0.0)
Link ID ADV Router Age Seq# CkSum Link count
172.16.200.1 172.16.200.1 716 0x80000011 0xba10 2
172.16.200.2 172.16.200.2 723 0x80000018 0x96e8 1
172.16.200.3 172.16.200.3 717 0x8000000f 0x4c93 1
172.16.200.4 172.16.200.4 717 0x80000011 0x4694 1
Net Link States (Area 0.0.0.0)
Link ID ADV Router Age Seq# CkSum
172.16.100.4 172.16.200.4 717 0x8000000f 0x6c7e
AS External Link States
Link ID ADV Router Age Seq# CkSum Route
172.16.200.1 172.16.200.1 716 0x8000000b 0x5156 E2 172.16.200.1/32 [0x0]
172.16.200.2 172.16.200.2 977 0x80000008 0x4761 E2 172.16.200.2/32 [0x0]
172.16.200.3 172.16.200.3 717 0x8000000a 0x3371 E2 172.16.200.3/32 [0x0]
172.16.200.4 172.16.200.4 717 0x8000000b 0x2180 E2 172.16.200.4/32 [0x0]
If 2-way and FULL states have not been established between the OSPF neighbors, check that all OSPF interface settings are correct. All usual OSPF neighborship requirements must be fulfilled.
The next step would be to enable OSPF logging as shown under the CLI log commands section.
Now, let’s check BGP.
Verify the BGP routes:
border1> show bgp ipv4
BGP table version is 13, local router ID is 172.16.200.1, vrf id 0
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* i1.1.1.0/24 172.16.200.2 0 100 0 ?
*> 0.0.0.0 0 32768 ?
* i2.2.2.0/24 172.16.200.2 0 100 0 ?
*> 0.0.0.0 0 32768 ?
* i3.3.3.0/24 172.16.200.2 0 100 0 ?
*> 0.0.0.0 0 32768 ?
* i10.0.2.0/24 172.16.200.3 0 100 0 ?
* i 172.16.200.2 0 100 0 ?
* i 172.16.200.4 0 100 0 ?
*> 0.0.0.0 0 32768 ?
* i172.16.100.0/24 172.16.200.3 0 100 0 ?
* i 172.16.200.2 0 100 0 ?
* i 172.16.200.4 0 100 0 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.200.1/32 0.0.0.0 0 32768 ?
*>i172.16.200.2/32 172.16.200.2 0 100 0 ?
*>i172.16.200.3/32 172.16.200.3 0 100 0 ?
*>i172.16.200.4/32 172.16.200.4 0 100 0 ?
*>i200.200.210.0/24 172.16.200.3 0 100 0 ?
*>i200.200.220.0/24 172.16.200.4 0 100 0 ?
*> 217.151.210.0/24 1.1.1.1 0 0 100 100 i
* i 1.1.1.1 0 100 0 100 100 i
*> 217.151.211.0/24 2.2.2.1 0 0 200 200 200 i
* i 2.2.2.1 0 100 0 200 200 200 i
*> 217.151.212.0/24 3.3.3.1 0 0 300 i
* i 3.3.3.1 0 100 0 300 i
Displayed 14 routes and 26 total paths
Let’s check BGP neighbors; in this example just the Transit_3 neighbor for brevity:
border1> show bgp neighbor 3.3.3.1
BGP neighbor is 3.3.3.1, remote AS 300, local AS 65200, external link
Description: Transit3-IPv4
Hostname: transit3-vm
BGP version 4, remote router ID 7.7.7.7
BGP state = Established, up for 00:30:02
Last read 00:00:02, Last write 00:00:02
Hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
AddPath:
IPv4 Unicast: RX advertised IPv4 Unicast and received
Route refresh: advertised and received(old & new)
Address Family IPv4 Unicast: advertised and received
Address Family IPv6 Unicast: received
Hostname Capability: advertised (name: border1,domain name: n/a) received (name: transit3-vm,domain name: n/a)
Graceful Restart Capabilty: advertised and received
Remote Restart timer is 120 seconds
Address families by peer:
none
Graceful restart informations:
End-of-RIB send: IPv4 Unicast
End-of-RIB received: IPv4 Unicast
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 3 4
Keepalives: 31 31
Route Refresh: 0 0
Capability: 0 0
Total: 35 36
Minimum time between advertisement runs is 0 seconds
For address family: IPv4 Unicast
Update group 1, subgroup 1
Packet Queue length 0
Inbound soft reconfiguration allowed
Community attribute sent to this neighbor(all)
Inbound path policy configured
Outbound path policy configured
Incoming update prefix filter list is *filter-bogons
Route map for outgoing advertisements is *TRANSIT-OUT
1 accepted prefixes
Connections established 1; dropped 0
Last reset never
Local host: 3.3.3.2, Local port: 40048
Foreign host: 3.3.3.1, Foreign port: 179
Nexthop: 3.3.3.2
Nexthop global: fe80::dced:1ff:fed8:6d1c
Nexthop local: fe80::dced:1ff:fed8:6d1c
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 120
Read thread: on Write thread: on
Verify BGP flowspec (so far in this case nothing to show):
border1> show bgp ipv4 flowspec
No BGP prefixes displayed, 0 exist
Many more show commands are available, please check in the User’s Guide as appropriate.
5.2. CLI log commands¶
To display the system log locally (kernel logs in this case):
border1> show log facility kernel
-- Logs begin at Tue 2019-07-09 14:37:46 UTC, end at Tue 2019-07-09 21:03:52 UTC. --
Jul 09 14:40:24 border1 kernel: Silicon Labs C2 port support v. 0.51.0 - (C) 2007 Rodolfo Giometti
Jul 09 14:40:31 border1 kernel: VFIO - User Level meta-driver version: 0.3
Jul 09 14:40:32 border1 kernel: iommu: Adding device 0000:00:04.0 to group 0
Jul 09 14:40:32 border1 kernel: vfio-pci 0000:00:04.0: Adding kernel taint for vfio-noiommu group on device
Jul 09 14:40:32 border1 kernel: iommu: Adding device 0000:00:05.0 to group 1
Jul 09 14:40:32 border1 kernel: vfio-pci 0000:00:05.0: Adding kernel taint for vfio-noiommu group on device
Jul 09 14:40:32 border1 kernel: iommu: Adding device 0000:00:06.0 to group 2
Jul 09 14:40:32 border1 kernel: vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
Jul 09 14:40:33 border1 kernel: dpvi: loading out-of-tree module taints kernel.
Jul 09 14:40:33 border1 kernel: dpvi: module verification failed: signature and/or required key missing - tainting kernel
Jul 09 14:40:33 border1 kernel: dpvi_shmem: dpvi_shmem module initialized 00000000bfa363e7
To specifically look at routing system (BGP, OSPF,..) events:
border1> show log service routing
-- Logs begin at Fri 2019-07-26 09:16:24 UTC, end at Fri 2019-07-26 09:47:01 UTC. --
Jul 26 09:18:54 border1 systemd[1]: Started zebra.
Jul 26 09:19:13 border1 systemd[1]: Started bgpd.
Jul 26 09:19:13 border1 systemd[1]: Started ospfd.
Logging of BGP neighbor changes:
border1> edit running
border1 running config# / vrf main routing bgp
border1 running bgp# log-neighbor-changes true
A per VRF remote logging capability can be enabled for the system log:
border1> edit running
border1 running config# / vrf main logging syslog
border1 running syslog#! remote-server 172.16.100.253 protocol tcp port 514
border1 running syslog# commit