5. Troubleshooting

5.1. CLI show commands

The CLI incorporates a number of show commands of which a few are shown here.

Showing the current basic state of all interfaces (add a command qualifier for more details):

border1-vm running config# show interface
Name                State          IP Addresses
----                -----          ------------
lo                  UNKNOWN        127.0.0.1/8
                                   ::1/128
ens3                UP             10.0.2.15/24
                                   fe80::dcad:deff:fe01:203/64
loopback0           UNKNOWN        172.16.200.1/32
                                   fe80::a060:efff:fe07:1acc/64
ntfp1               UP             172.16.100.1/24
                                   fe80::dced:1ff:fe5d:87a4/64
ntfp2               UP             fe80::dced:1ff:fe03:de92/64
ntfp3               UP             fe80::dced:1ff:fe98:20f7/64
vlan3@ntfp2         UP             3.3.3.2/24
                                   fe80::dced:1ff:fe03:de92/64
vlan1@ntfp3         UP             1.1.1.2/24
                                   fe80::dced:1ff:fe98:20f7/64
vlan2@ntfp3         UP             2.2.2.2/24
                                   fe80::dced:1ff:fe98:20f7/64
vrrp1@vlan1         UP             1.1.1.4/24
vrrp2@vlan2         UP             2.2.2.4/24
vrrp3@vlan3         UP             3.3.3.4/24
vrrp_internal@ntfp1 UP             172.16.100.5/24

Showing the detailed state of one particular interface: ntfp1

border1-vm running config# show interface name ntfp1 details
10: ntfp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether de:ed:01:5d:87:a4 brd ff:ff:ff:ff:ff:ff
    inet 172.16.100.1/24 scope global ntfp1
       valid_lft forever preferred_lft forever
    inet6 fe80::dced:1ff:fe5d:87a4/64 scope link
       valid_lft forever preferred_lft forever

Basic interface UDP traffic dump example:

border1> cmd show-traffic ntfp1 filter udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ntfp1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:38:47.221472 de:ed:01:e3:55:78 > de:ed:01:07:da:e2, ethertype IPv4 (0x0800), length 746: 172.16.100.2.45791 > 172.16.100.254.6343: sFlowv5, IPv4 agent 172.16.200.2, agent-id 100000, length 704
18:38:47.221482 de:ed:01:e3:55:78 > de:ed:01:07:da:e2, ethertype IPv4 (0x0800), length 746: 172.16.100.2.45791 > 172.16.100.254.6343: sFlowv5, IPv4 agent 172.16.200.2, agent-id 100000, length 704
18:38:47.221484 de:ed:01:e3:55:78 > de:ed:01:1b:a5:56, ethertype IPv4 (0x0800), length 746: 172.16.100.2.45791 > 172.16.100.253.6343: sFlowv5, IPv4 agent 172.16.200.2, agent-id 100000, length 704
18:38:47.221485 de:ed:01:e3:55:78 > de:ed:01:1b:a5:56, ethertype IPv4 (0x0800), length 746: 172.16.100.2.45791 > 172.16.100.253.6343: sFlowv5, IPv4 agent 172.16.200.2, agent-id 100000, length 704
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

See also

See the User’s Guide for more information regarding:

The first obvious choice to troubleshoot connectivity problems is to verify that all the routes are in the routing table using the following command:

border1> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route

VRF main:
K>* 0.0.0.0/0 [0/0] via 10.0.2.2, ens3, 06:22:10
C * 1.1.1.0/24 is directly connected, vrrp1, 06:21:53
C>* 1.1.1.0/24 is directly connected, vlan1, 06:21:58
C * 2.2.2.0/24 is directly connected, vrrp2, 06:21:53
C>* 2.2.2.0/24 is directly connected, vlan2, 06:21:58
C * 3.3.3.0/24 is directly connected, vrrp3, 06:21:53
C>* 3.3.3.0/24 is directly connected, vlan3, 06:21:58
C>* 10.0.2.0/24 is directly connected, ens3, 06:22:10
O   172.16.100.0/24 [110/100] is directly connected, ntfp1, 06:21:11
                              is directly connected, vrrp_internal, 06:21:11
C * 172.16.100.0/24 is directly connected, vrrp_internal, 06:21:53
C>* 172.16.100.0/24 is directly connected, ntfp1, 06:21:58
C>* 172.16.200.1/32 is directly connected, loopback0, 06:22:08
B   172.16.200.2/32 [200/0] via 172.16.200.2, 06:21:04
O>* 172.16.200.2/32 [110/20] via 172.16.100.2, ntfp1, 06:21:10
  *                          via 172.16.100.2, vrrp_internal, 06:21:10
B   172.16.200.3/32 [200/0] via 172.16.200.3, 06:21:04
O>* 172.16.200.3/32 [110/20] via 172.16.100.3, ntfp1, 06:21:05
  *                          via 172.16.100.3, vrrp_internal, 06:21:05
B   172.16.200.4/32 [200/0] via 172.16.200.4, 06:21:09
O>* 172.16.200.4/32 [110/20] via 172.16.100.4, ntfp1, 06:21:10
  *                          via 172.16.100.4, vrrp_internal, 06:21:10
B>  200.200.210.0/24 [200/0] via 172.16.200.3 (recursive), 06:21:04
  *                            via 172.16.100.3, ntfp1, 06:21:04
  *                            via 172.16.100.3, vrrp_internal, 06:21:04
B>  200.200.220.0/24 [200/0] via 172.16.200.4 (recursive), 06:21:09
  *                            via 172.16.100.4, ntfp1, 06:21:09
  *                            via 172.16.100.4, vrrp_internal, 06:21:09
B>* 217.151.210.0/24 [20/0] via 1.1.1.1, vlan1, 06:21:54
B>* 217.151.211.0/24 [20/0] via 2.2.2.1, vlan2, 06:21:54
B>* 217.151.212.0/24 [20/0] via 3.3.3.1, vlan3, 06:21:54

Refining the show command, we can first look at the OSPF routes:

border1> show ospf route
VRF Name: default
============ OSPF network routing table ============
N    172.16.100.0/24       [100] area: 0.0.0.0
                           directly attached to ntfp1
                           directly attached to vrrp_internal

============ OSPF router routing table =============
R    172.16.200.2          [100] area: 0.0.0.0, ASBR
                           via 172.16.100.2, ntfp1
                           via 172.16.100.2, vrrp_internal
R    172.16.200.3          [100] area: 0.0.0.0, ASBR
                           via 172.16.100.3, ntfp1
                           via 172.16.100.3, vrrp_internal
R    172.16.200.4          [100] area: 0.0.0.0, ASBR
                           via 172.16.100.4, ntfp1
                           via 172.16.100.4, vrrp_internal

============ OSPF external routing table ===========
N E2 172.16.200.2/32       [100/20] tag: 0
                           via 172.16.100.2, ntfp1
                           via 172.16.100.2, vrrp_internal
N E2 172.16.200.3/32       [100/20] tag: 0
                           via 172.16.100.3, ntfp1
                           via 172.16.100.3, vrrp_internal
N E2 172.16.200.4/32       [100/20] tag: 0
                           via 172.16.100.4, ntfp1
                           via 172.16.100.4, vrrp_internal

If OSPF routes seem to be missing, try verifying that OSPF has formed the correct neighbor relationships:

border1> show ospf neighbor
VRF Name: default

Neighbor ID     Pri State           Dead Time Address         Interface                  RXmtL RqstL DBsmL
172.16.200.2      1 2-Way/DROther     36.233s 172.16.100.2    ntfp1:172.16.100.1             0     0     0
172.16.200.3      1 Full/Backup       34.142s 172.16.100.3    ntfp1:172.16.100.1             0     0     0
172.16.200.4      1 Full/DR           33.873s 172.16.100.4    ntfp1:172.16.100.1             0     0     0
172.16.200.2      1 ExStart/DR        32.820s 172.16.100.2    vrrp_internal:172.16.100.5     0     0     0
172.16.200.3      1 2-Way/DROther     31.615s 172.16.100.3    vrrp_internal:172.16.100.5     0     0     0
172.16.200.4      1 Full/Backup       33.979s 172.16.100.4    vrrp_internal:172.16.100.5     0     0     0

And we can also verify the OSPF topology database:

border1> show ospf database
VRF Name: default

       OSPF Router with ID (172.16.200.1)

                Router Link States (Area 0.0.0.0)

Link ID         ADV Router      Age  Seq#       CkSum  Link count
172.16.200.1    172.16.200.1     716 0x80000011 0xba10 2
172.16.200.2    172.16.200.2     723 0x80000018 0x96e8 1
172.16.200.3    172.16.200.3     717 0x8000000f 0x4c93 1
172.16.200.4    172.16.200.4     717 0x80000011 0x4694 1

                Net Link States (Area 0.0.0.0)

Link ID         ADV Router      Age  Seq#       CkSum
172.16.100.4    172.16.200.4     717 0x8000000f 0x6c7e

                AS External Link States

Link ID         ADV Router      Age  Seq#       CkSum  Route
172.16.200.1    172.16.200.1     716 0x8000000b 0x5156 E2 172.16.200.1/32 [0x0]
172.16.200.2    172.16.200.2     977 0x80000008 0x4761 E2 172.16.200.2/32 [0x0]
172.16.200.3    172.16.200.3     717 0x8000000a 0x3371 E2 172.16.200.3/32 [0x0]
172.16.200.4    172.16.200.4     717 0x8000000b 0x2180 E2 172.16.200.4/32 [0x0]

If 2-way and FULL states have not been established between the OSPF neighbors, check that all OSPF interface settings are correct. All usual OSPF neighborship requirements must be fulfilled.

The next step would be to enable OSPF logging as shown under the CLI log commands section.

Now, let’s check BGP.

Verify the BGP routes:

border1> show bgp ipv4
BGP table version is 13, local router ID is 172.16.200.1, vrf id 0
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
* i1.1.1.0/24       172.16.200.2             0    100      0 ?
*>                  0.0.0.0                  0         32768 ?
* i2.2.2.0/24       172.16.200.2             0    100      0 ?
*>                  0.0.0.0                  0         32768 ?
* i3.3.3.0/24       172.16.200.2             0    100      0 ?
*>                  0.0.0.0                  0         32768 ?
* i10.0.2.0/24      172.16.200.3             0    100      0 ?
* i                 172.16.200.2             0    100      0 ?
* i                 172.16.200.4             0    100      0 ?
*>                  0.0.0.0                  0         32768 ?
* i172.16.100.0/24  172.16.200.3             0    100      0 ?
* i                 172.16.200.2             0    100      0 ?
* i                 172.16.200.4             0    100      0 ?
*>                  0.0.0.0                  0         32768 ?
*> 172.16.200.1/32  0.0.0.0                  0         32768 ?
*>i172.16.200.2/32  172.16.200.2             0    100      0 ?
*>i172.16.200.3/32  172.16.200.3             0    100      0 ?
*>i172.16.200.4/32  172.16.200.4             0    100      0 ?
*>i200.200.210.0/24 172.16.200.3             0    100      0 ?
*>i200.200.220.0/24 172.16.200.4             0    100      0 ?
*> 217.151.210.0/24 1.1.1.1                  0             0 100 100 i
* i                 1.1.1.1                  0    100      0 100 100 i
*> 217.151.211.0/24 2.2.2.1                  0             0 200 200 200 i
* i                 2.2.2.1                  0    100      0 200 200 200 i
*> 217.151.212.0/24 3.3.3.1                  0             0 300 i
* i                 3.3.3.1                  0    100      0 300 i

Displayed  14 routes and 26 total paths

Let’s check BGP neighbors; in this example just the Transit_3 neighbor for brevity:

border1> show bgp neighbor 3.3.3.1
 BGP neighbor is 3.3.3.1, remote AS 300, local AS 65200, external link
 Description: Transit3-IPv4
Hostname: transit3-vm
  BGP version 4, remote router ID 7.7.7.7
  BGP state = Established, up for 00:30:02
  Last read 00:00:02, Last write 00:00:02
  Hold time is 180, keepalive interval is 60 seconds
  Neighbor capabilities:
        4 Byte AS: advertised and received
        AddPath:
        IPv4 Unicast: RX advertised IPv4 Unicast and received
        Route refresh: advertised and received(old & new)
        Address Family IPv4 Unicast: advertised and received
        Address Family IPv6 Unicast: received
        Hostname Capability: advertised (name: border1,domain name: n/a) received (name: transit3-vm,domain name: n/a)
        Graceful Restart Capabilty: advertised and received
        Remote Restart timer is 120 seconds
        Address families by peer:
        none
  Graceful restart informations:
        End-of-RIB send: IPv4 Unicast
        End-of-RIB received: IPv4 Unicast
  Message statistics:
        Inq depth is 0
        Outq depth is 0
                        Sent       Rcvd
        Opens:                  1          1
        Notifications:          0          0
        Updates:                3          4
        Keepalives:            31         31
        Route Refresh:          0          0
        Capability:             0          0
        Total:                 35         36
  Minimum time between advertisement runs is 0 seconds

For address family: IPv4 Unicast
 Update group 1, subgroup 1
 Packet Queue length 0
 Inbound soft reconfiguration allowed
 Community attribute sent to this neighbor(all)
 Inbound path policy configured
 Outbound path policy configured
 Incoming update prefix filter list is *filter-bogons
 Route map for outgoing advertisements is *TRANSIT-OUT
 1 accepted prefixes

 Connections established 1; dropped 0
 Last reset never
Local host: 3.3.3.2, Local port: 40048
Foreign host: 3.3.3.1, Foreign port: 179
Nexthop: 3.3.3.2
Nexthop global: fe80::dced:1ff:fed8:6d1c
Nexthop local: fe80::dced:1ff:fed8:6d1c
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 120
Read thread: on  Write thread: on

Verify BGP flowspec (so far in this case nothing to show):

border1> show bgp ipv4 flowspec
No BGP prefixes displayed, 0 exist

Many more show commands are available, please check in the User’s Guide as appropriate.

5.2. CLI log commands

To display the system log locally (kernel logs in this case):

border1> show log facility kernel
-- Logs begin at Tue 2019-07-09 14:37:46 UTC, end at Tue 2019-07-09 21:03:52 UTC. --
Jul 09 14:40:24 border1 kernel: Silicon Labs C2 port support v. 0.51.0 - (C) 2007 Rodolfo Giometti
Jul 09 14:40:31 border1 kernel: VFIO - User Level meta-driver version: 0.3
Jul 09 14:40:32 border1 kernel: iommu: Adding device 0000:00:04.0 to group 0
Jul 09 14:40:32 border1 kernel: vfio-pci 0000:00:04.0: Adding kernel taint for vfio-noiommu group on device
Jul 09 14:40:32 border1 kernel: iommu: Adding device 0000:00:05.0 to group 1
Jul 09 14:40:32 border1 kernel: vfio-pci 0000:00:05.0: Adding kernel taint for vfio-noiommu group on device
Jul 09 14:40:32 border1 kernel: iommu: Adding device 0000:00:06.0 to group 2
Jul 09 14:40:32 border1 kernel: vfio-pci 0000:00:06.0: Adding kernel taint for vfio-noiommu group on device
Jul 09 14:40:33 border1 kernel: dpvi: loading out-of-tree module taints kernel.
Jul 09 14:40:33 border1 kernel: dpvi: module verification failed: signature and/or required key missing - tainting kernel
Jul 09 14:40:33 border1 kernel: dpvi_shmem: dpvi_shmem module initialized 00000000bfa363e7

To specifically look at routing system (BGP, OSPF,..) events:

border1> show log service routing
-- Logs begin at Fri 2019-07-26 09:16:24 UTC, end at Fri 2019-07-26 09:47:01 UTC. --
Jul 26 09:18:54 border1 systemd[1]: Started zebra.
Jul 26 09:19:13 border1 systemd[1]: Started bgpd.
Jul 26 09:19:13 border1 systemd[1]: Started ospfd.

Logging of BGP neighbor changes:

border1> edit running
border1 running config# / vrf main routing bgp
border1 running bgp# log-neighbor-changes true

A per VRF remote logging capability can be enabled for the system log:

border1> edit running
border1 running config# / vrf main logging syslog
border1 running syslog#! remote-server 172.16.100.253 protocol tcp port 514
border1 running syslog# commit

See also

For more details, please refer to: