DMVPN and NHRP Configuration¶
NHRP use case example¶
The following configuration examples apply on the topology described in the
above picture: 3 spokes connected to a hub.
The first two chapters describe how to use GRE interfaces and NHRP to
establish connections between the spokes via the hub, either statically or
dynamically.
The last chapter introduces the redirect feature that enables dynamic tunnel
establishment between spokes.
For all those examples, NHRP configuration requires that the IP address of GRE interface be configured with 32 bitmask, as follows. Configuring an other bitmask will lead to a misconfiguration problem.
spoke1 running config# / vrf main interface gre gre1 ipv4 address 10.255.255.1/32
Static configuration¶
We can define static NHRP entries for remote endpoints on each node. From a protocol point of view, hub and spokes don’t exchange messages then. The hub declares the protocol address and the NBMA address of the spokes. The spokes only declare a NHRP entry for the hub. In that case, spoke-to-spoke communications flow through the hub. Therefore the hub declares a multipoint GRE interface facing all the spokes and the spokes declare a GRE interface facing the hub only. In addition, each node runs a BGP instance to advertise its private network and learn the route towards the private subnets of the other nodes.
First we configure the three spokes with a static NHRP entry for the hub:
spoke1 running config# / vrf main interface physical lan port pci-b0s5
spoke1 running config# / vrf main interface physical lan ipv4 address 192.168.1.1/24
spoke1 running config# / vrf main interface physical wan port pci-b0s4
spoke1 running config# / vrf main interface physical wan ipv4 address 11.11.11.11/24
spoke1 running config# / vrf main interface gre gre1 ipv4 address 10.255.255.1/32
spoke1 running config#! / vrf main interface gre gre1 link-interface wan
spoke1 running config#! / vrf main interface gre gre1 local 11.11.11.11
spoke1 running config# / vrf main interface gre gre1 remote 44.44.44.44
spoke1 running config# / vrf main routing static ipv4-route 0.0.0.0/0 next-hop 11.11.11.1
spoke1 running config# / vrf main routing nhrp enabled true
spoke1 running config# / vrf main routing interface gre1 ip nhrp registration-no-unique true
spoke1 running config# / vrf main routing interface gre1 ip nhrp network-id 123
spoke1 running config# / vrf main routing interface gre1 ip nhrp holdtime 1200
spoke1 running config# / vrf main routing interface gre1 ip nhrp nhrp-map 10.255.255.4 nbma 44.44.44.44
spoke1 running config# / vrf main routing bgp as 65000
spoke1 running config# / vrf main routing bgp router-id 10.255.255.1
spoke1 running config# / vrf main routing bgp address-family ipv4-unicast network 192.168.1.0/24
spoke1 running network 192.168.1.0/24# / vrf main routing bgp address-family ipv4-unicast network 10.255.255.1/32
spoke1 running network 10.255.255.1/32# / vrf main routing bgp neighbor 10.255.255.4 remote-as 65000
spoke2 running config# / vrf main interface physical lan port pci-b0s5
spoke2 running config# / vrf main interface physical lan ipv4 address 192.168.2.1/24
spoke2 running config# / vrf main interface physical wan port pci-b0s4
spoke2 running config# / vrf main interface physical wan ipv4 address 22.22.22.22/24
spoke2 running config# / vrf main interface gre gre2 ipv4 address 10.255.255.2/32
spoke2 running config#! / vrf main interface gre gre2 link-interface wan
spoke2 running config#! / vrf main interface gre gre2 local 22.22.22.22
spoke2 running config# / vrf main interface gre gre2 remote 44.44.44.44
spoke2 running config# / vrf main routing static ipv4-route 0.0.0.0/0 next-hop 22.22.22.1
spoke2 running config# / vrf main routing nhrp enabled true
spoke2 running config# / vrf main routing interface gre2 ip nhrp registration-no-unique true
spoke2 running config# / vrf main routing interface gre2 ip nhrp network-id 123
spoke2 running config# / vrf main routing interface gre2 ip nhrp holdtime 1200
spoke2 running config# / vrf main routing interface gre2 ip nhrp nhrp-map 10.255.255.4 nbma 44.44.44.44
spoke2 running config# / vrf main routing bgp as 65000
spoke2 running config# / vrf main routing bgp router-id 10.255.255.2
spoke2 running config# / vrf main routing bgp address-family ipv4-unicast network 192.168.2.0/24
spoke2 running network 192.168.2.0/24# / vrf main routing bgp address-family ipv4-unicast network 10.255.255.2/32
spoke2 running network 10.255.255.2/32# / vrf main routing bgp neighbor 10.255.255.4 remote-as 65000
spoke3 running config# / vrf main interface physical lan port pci-b0s5
spoke3 running config# / vrf main interface physical lan ipv4 address 192.168.3.1/24
spoke3 running config# / vrf main interface physical wan port pci-b0s4
spoke3 running config# / vrf main interface physical wan ipv4 address 33.33.33.33/24
spoke3 running config# / vrf main interface gre gre3 ipv4 address 10.255.255.3/32
spoke3 running config#! / vrf main interface gre gre3 link-interface wan
spoke3 running config#! / vrf main interface gre gre3 local 33.33.33.33
spoke3 running config# / vrf main interface gre gre3 remote 44.44.44.44
spoke3 running config# / vrf main routing static ipv4-route 0.0.0.0/0 next-hop 33.33.33.1
spoke3 running config# / vrf main routing nhrp enabled true
spoke3 running config# / vrf main routing interface gre3 ip nhrp registration-no-unique true
spoke3 running config# / vrf main routing interface gre3 ip nhrp network-id 123
spoke3 running config# / vrf main routing interface gre3 ip nhrp holdtime 1200
spoke3 running config# / vrf main routing interface gre3 ip nhrp nhrp-map 10.255.255.4 nbma 44.44.44.44
spoke3 running config# / vrf main routing bgp as 65000
spoke3 running config# / vrf main routing bgp router-id 10.255.255.3
spoke3 running config# / vrf main routing bgp address-family ipv4-unicast network 192.168.3.0/24
spoke3 running network 192.168.3.0/24# / vrf main routing bgp address-family ipv4-unicast network 10.255.255.3/32
spoke3 running network 10.255.255.3/32# / vrf main routing bgp neighbor 10.255.255.4 remote-as 65000
Similarly, we configure static NHRP entries for each spoke on the hub. The main difference is that we configure a multipoint GRE on this node, by omitting a remote address, to be able to reach each spoke:
hub running config# / vrf main interface physical wan port pci-b0s4
hub running config# / vrf main interface physical wan ipv4 address 44.44.44.44/24
hub running config# / vrf main interface physical lan port pci-b0s5
hub running config# / vrf main interface physical lan ipv4 address 192.168.4.1/24
hub running config# / vrf main interface gre gre4 ipv4 address 10.255.255.4/32
hub running config#! / vrf main interface gre gre4 link-interface wan
hub running config#! / vrf main interface gre gre4 local 44.44.44.44
hub running config# / vrf main routing static ipv4-route 0.0.0.0/0 next-hop 44.44.44.1
hub running config# / vrf main routing nhrp enabled true
hub running config# / vrf main routing interface gre4 ip nhrp registration-no-unique true
hub running config# / vrf main routing interface gre4 ip nhrp network-id 123
hub running config# / vrf main routing interface gre4 ip nhrp holdtime 1200
hub running config# / vrf main routing interface gre4 ip nhrp nhrp-map 10.255.255.1 nbma 11.11.11.11
hub running config# / vrf main routing interface gre4 ip nhrp nhrp-map 10.255.255.2 nbma 22.22.22.22
hub running config# / vrf main routing interface gre4 ip nhrp nhrp-map 10.255.255.3 nbma 33.33.33.33
hub running config# / vrf main routing bgp as 65000
hub running config# / vrf main routing bgp router-id 10.255.255.4
hub running config# / vrf main routing bgp address-family ipv4-unicast network 192.168.4.0/24
hub running network 192.168.4.0/24# / vrf main routing bgp listen neighbor-range 10.255.255.0/24 neighbor-group GROUP
hub running network 192.168.4.0/24#! / vrf main routing bgp neighbor-group GROUP remote-as 65000
hub running network 192.168.4.0/24# / vrf main routing bgp neighbor-group GROUP address-family ipv4-unicast nexthop-self force true
hub running network 192.168.4.0/24# / vrf main routing bgp neighbor-group GROUP address-family ipv4-unicast route-reflector-client true
A router is configured to represent the network between the spokes and the hub with the following configuration:
router-vm running config# / vrf main interface physical eth1 port pci-b0s4
router-vm running config# / vrf main interface physical eth1 ipv4 address 11.11.11.1/24
router-vm running config# / vrf main interface physical eth2 port pci-b0s5
router-vm running config# / vrf main interface physical eth2 ipv4 address 22.22.22.1/24
router-vm running config# / vrf main interface physical eth3 port pci-b0s6
router-vm running config# / vrf main interface physical eth3 ipv4 address 33.33.33.1/24
router-vm running config# / vrf main interface physical eth4 port pci-b0s7
router-vm running config# / vrf main interface physical eth4 ipv4 address 44.44.44.1/24
We can check the NHRP cache entries on each node:
hub> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre4 local 10.255.255.4 44.44.44.44 44.44.44.44 -
gre4 static 10.255.255.3 33.33.33.33 - U
gre4 static 10.255.255.2 22.22.22.22 - U
gre4 static 10.255.255.1 11.11.11.11 - U
spoke1> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre1 static 10.255.255.4 44.44.44.44 -
gre1 local 10.255.255.1 11.11.11.11 11.11.11.11 -
spoke2> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre2 static 10.255.255.4 44.44.44.44 -
gre2 local 10.255.255.2 22.22.22.22 22.22.22.22 -
spoke3> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre3 static 10.255.255.4 44.44.44.44 -
gre3 local 10.255.255.3 33.33.33.33 33.33.33.33 -
We can see that there is a NHRP connection between the hub and each device:
hub> show nhrp-connection
Src Dst Flags SAs Identity
44.44.44.44 22.22.22.22 n 0
44.44.44.44 33.33.33.33 n 0
44.44.44.44 11.11.11.11 n 0
spoke1> show nhrp-connection
Src Dst Flags SAs Identity
11.11.11.11 44.44.44.44 n 0
spoke2> show nhrp-connection
Src Dst Flags SAs Identity
22.22.22.22 44.44.44.44 n 0
spoke3> show nhrp-connection
Src Dst Flags SAs Identity
33.33.33.33 44.44.44.44 n 0
As a consequence, a NHRP route towards the protocol address of the hub is installed on each spoke. This way, they can establish a BGP peering and learn the routes to other nodes:
spoke1> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 11.11.11.1, wan, weight 1, 00:00:18
C>* 10.255.255.1/32 is directly connected, gre1, 00:00:19
B> 10.255.255.2/32 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:11
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:11
B> 10.255.255.3/32 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:06
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:06
N>* 10.255.255.4/32 [10/0] is directly connected, gre1, weight 1, 00:00:18
C>* 11.11.11.0/24 is directly connected, wan, 00:00:20
C>* 192.168.1.0/24 is directly connected, lan, 00:00:20
B> 192.168.2.0/24 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:11
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:11
B> 192.168.3.0/24 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:06
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:06
B> 192.168.4.0/24 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:15
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:15
10 routes displayed.
spoke2> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 22.22.22.1, wan, weight 1, 00:00:14
B> 10.255.255.1/32 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:12
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:12
C>* 10.255.255.2/32 is directly connected, gre2, 00:00:16
B> 10.255.255.3/32 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:07
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:07
N>* 10.255.255.4/32 [10/0] is directly connected, gre2, weight 1, 00:00:14
C>* 22.22.22.0/24 is directly connected, wan, 00:00:16
B> 192.168.1.0/24 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:12
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:12
C>* 192.168.2.0/24 is directly connected, lan, 00:00:16
B> 192.168.3.0/24 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:07
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:07
B> 192.168.4.0/24 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:12
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:12
10 routes displayed.
spoke3> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 33.33.33.1, wan, weight 1, 00:00:11
B> 10.255.255.1/32 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:09
* via 10.255.255.4, gre3 onlink, weight 1, 00:00:09
B> 10.255.255.2/32 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:09
* via 10.255.255.4, gre3 onlink, weight 1, 00:00:09
C>* 10.255.255.3/32 is directly connected, gre3, 00:00:13
N>* 10.255.255.4/32 [10/0] is directly connected, gre3, weight 1, 00:00:11
C>* 33.33.33.0/24 is directly connected, wan, 00:00:13
B> 192.168.1.0/24 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:09
* via 10.255.255.4, gre3 onlink, weight 1, 00:00:09
B> 192.168.2.0/24 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:09
* via 10.255.255.4, gre3 onlink, weight 1, 00:00:09
C>* 192.168.3.0/24 is directly connected, lan, 00:00:13
B> 192.168.4.0/24 [200/0] via 10.255.255.4 (recursive), weight 1, 00:00:09
* via 10.255.255.4, gre3 onlink, weight 1, 00:00:09
10 routes displayed.
The hub installs a NHRP route for each spoke and learns the remote private networks through BGP:
hub> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 44.44.44.1, wan, weight 1, 00:00:21
B 10.255.255.1/32 [200/0] via 10.255.255.1 inactive, weight 1, 00:00:14
N>* 10.255.255.1/32 [10/0] is directly connected, gre4, weight 1, 00:00:21
B 10.255.255.2/32 [200/0] via 10.255.255.2 inactive, weight 1, 00:00:09
N>* 10.255.255.2/32 [10/0] is directly connected, gre4, weight 1, 00:00:21
B 10.255.255.3/32 [200/0] via 10.255.255.3 inactive, weight 1, 00:00:05
N>* 10.255.255.3/32 [10/0] is directly connected, gre4, weight 1, 00:00:21
C>* 10.255.255.4/32 is directly connected, gre4, 00:00:23
C>* 44.44.44.0/24 is directly connected, wan, 00:00:23
B> 192.168.1.0/24 [200/0] via 10.255.255.1 (recursive), weight 1, 00:00:14
* via 10.255.255.1, gre4 onlink, weight 1, 00:00:14
B> 192.168.2.0/24 [200/0] via 10.255.255.2 (recursive), weight 1, 00:00:09
* via 10.255.255.2, gre4 onlink, weight 1, 00:00:09
B> 192.168.3.0/24 [200/0] via 10.255.255.3 (recursive), weight 1, 00:00:05
* via 10.255.255.3, gre4 onlink, weight 1, 00:00:05
C>* 192.168.4.0/24 is directly connected, lan, 00:00:23
10 routes displayed.
Dynamic configuration¶
Static configuration doesn’t scale well as you have to declare each spoke individually on the hub. We can fully leverage NHRP and use dynamic configuration instead. In this configuration, the NHRP server on the hub listens for registration requests. Each spoke declares the hub as a NHRP server and thus registers upon startup using NHRP control packets. With the basic setup below, spoke-to-spoke communication still runs through the hub but can then be improved to allow Direct spoke-to-spoke communication. That’s why we configure point to multipoint GRE interface on both spoke and hub side.
BGP configuration¶
In addition, each node runs a BGP instance to advertise its private network. As the chapter deals with spoke to hub communication, only hub is interested in getting the private networks located behind spokes.
iBGP configuration¶
It is possible to keep using iBGP sessions between spokes and hub, like it has been done on previous experiment. In that case, spokes will install iBGP routes to reach private networks from other spokes, but by using hub as nexthop.
eBGP configuration¶
It is also possible to avoid transmitting the private networks information to all spokes. Two reasons for that:
In that setup where data is always passing through the hub, there is no point giving this information to spokes that always point to hub.
This routing information is redundant with NHRP when direct spoke to spoke communication will be put in place. Even if iBGP routes were used, NHRP routes to those private networks would override those BGP routes, and NHRP routes would directly have the nexthop set to the appropriate spoke.
For the rest of this chapter, We choose to use eBGP configuration. The hub will
transmit two aggregated networks, 192.168.0.0/16 and 10.255.255.0/24. This stands
for the subnetworks to all spokes. You can note that eBGP requires peers to be
directly connected. As the route to reach the hub IP on the spoke is a NHRP route
and not a connected route, an additional configuration is put in place, in order to
inject incoming BGP network. This is done via ebgp-connected-route-check BGP
configuration command:
Note
Using eBGP avoids spokes to learn BGP route entries from spokes on same AS coming from an external AS. This specificity works because the spokes BGP configurations use the same AS and are not fully meshed.
Note
An other alternative to defining aggregated networks on hub side is to define a default-route at hub side, like below BGP command shows it:
hub running bgp# / vrf main routing bgp neighour-group GROUP address-family ipv4-unicast default-originate
For that, there must not be any conflicts with learnt default route of the ISPs of local spokes.
Configuration¶
The following configuration applies on the same topology with one hub and three spokes.
spoke1 running config# / vrf main interface physical lan port pci-b0s5
spoke1 running config# / vrf main interface physical lan ipv4 address 192.168.1.1/24
spoke1 running config# / vrf main interface physical wan port pci-b0s4
spoke1 running config# / vrf main interface physical wan ipv4 address 11.11.11.11/24
spoke1 running config# / vrf main interface gre gre1 ipv4 address 10.255.255.1/32
spoke1 running config#! / vrf main interface gre gre1 link-interface wan
spoke1 running config#! / vrf main interface gre gre1 local 11.11.11.11
spoke1 running config# / vrf main interface gre gre1 ttl 64
spoke1 running config# / vrf main routing static ipv4-route 0.0.0.0/0 next-hop 11.11.11.1
spoke1 running config# / vrf main routing nhrp enabled true
spoke1 running config# / vrf main routing interface gre1 ip nhrp registration-no-unique true
spoke1 running config# / vrf main routing interface gre1 ip nhrp network-id 123
spoke1 running config# / vrf main routing interface gre1 ip nhrp holdtime 1200
spoke1 running config# / vrf main routing interface gre1 ip nhrp nhrp-nhs dynamic nbma 44.44.44.44
spoke1 running config# / vrf main routing bgp as 65099
spoke1 running config# / vrf main routing bgp ebgp-connected-route-check false
spoke1 running config# / vrf main routing bgp ebgp-requires-policy false
spoke1 running config# / vrf main routing bgp router-id 10.255.255.1
spoke1 running config# / vrf main routing bgp address-family ipv4-unicast network 192.168.1.0/24
spoke1 running network 192.168.1.0/24# / vrf main routing bgp neighbor 10.255.255.4 remote-as 65000
spoke2 running config# / vrf main interface physical lan port pci-b0s5
spoke2 running config# / vrf main interface physical lan ipv4 address 192.168.2.1/24
spoke2 running config# / vrf main interface physical wan port pci-b0s4
spoke2 running config# / vrf main interface physical wan ipv4 address 22.22.22.22/24
spoke2 running config# / vrf main interface gre gre2 ipv4 address 10.255.255.2/32
spoke2 running config#! / vrf main interface gre gre2 link-interface wan
spoke2 running config#! / vrf main interface gre gre2 local 22.22.22.22
spoke2 running config# / vrf main interface gre gre2 ttl 64
spoke2 running config# / vrf main routing static ipv4-route 0.0.0.0/0 next-hop 22.22.22.1
spoke2 running config# / vrf main routing nhrp enabled true
spoke2 running config# / vrf main routing interface gre2 ip nhrp registration-no-unique true
spoke2 running config# / vrf main routing interface gre2 ip nhrp network-id 123
spoke2 running config# / vrf main routing interface gre2 ip nhrp holdtime 1200
spoke2 running config# / vrf main routing interface gre2 ip nhrp nhrp-nhs dynamic nbma 44.44.44.44
spoke2 running config# / vrf main routing bgp as 65099
spoke2 running config# / vrf main routing bgp ebgp-connected-route-check false
spoke2 running config# / vrf main routing bgp ebgp-requires-policy false
spoke2 running config# / vrf main routing bgp router-id 10.255.255.2
spoke2 running config# / vrf main routing bgp address-family ipv4-unicast network 192.168.2.0/24
spoke2 running network 192.168.2.0/24# / vrf main routing bgp neighbor 10.255.255.4 remote-as 65000
spoke3 running config# / vrf main interface physical lan port pci-b0s5
spoke3 running config# / vrf main interface physical lan ipv4 address 192.168.3.1/24
spoke3 running config# / vrf main interface physical wan port pci-b0s4
spoke3 running config# / vrf main interface physical wan ipv4 address 33.33.33.33/24
spoke3 running config# / vrf main interface gre gre3 ipv4 address 10.255.255.3/32
spoke3 running config#! / vrf main interface gre gre3 link-interface wan
spoke3 running config#! / vrf main interface gre gre3 local 33.33.33.33
spoke3 running config# / vrf main interface gre gre3 ttl 64
spoke3 running config# / vrf main routing static ipv4-route 0.0.0.0/0 next-hop 33.33.33.1
spoke3 running config# / vrf main routing nhrp enabled true
spoke3 running config# / vrf main routing interface gre3 ip nhrp registration-no-unique true
spoke3 running config# / vrf main routing interface gre3 ip nhrp network-id 123
spoke3 running config# / vrf main routing interface gre3 ip nhrp holdtime 1200
spoke3 running config# / vrf main routing interface gre3 ip nhrp nhrp-nhs dynamic nbma 44.44.44.44
spoke3 running config# / vrf main routing bgp as 65099
spoke3 running config# / vrf main routing bgp ebgp-connected-route-check false
spoke3 running config# / vrf main routing bgp ebgp-requires-policy false
spoke3 running config# / vrf main routing bgp router-id 10.255.255.3
spoke3 running config# / vrf main routing bgp address-family ipv4-unicast network 192.168.3.0/24
spoke3 running network 192.168.3.0/24# / vrf main routing bgp neighbor 10.255.255.4 remote-as 65000
Similarly, we do not declare any spoke entry on the hub. This node only listens for NHRP registration requests and then for BGP peerings coming from the spokes.
hub running config# / vrf main interface physical wan port pci-b0s4
hub running config# / vrf main interface physical wan ipv4 address 44.44.44.44/24
hub running config# / vrf main interface physical lan port pci-b0s5
hub running config# / vrf main interface physical lan ipv4 address 192.168.4.1/24
hub running config# / vrf main interface gre gre4 ipv4 address 10.255.255.4/32
hub running config#! / vrf main interface gre gre4 link-interface wan
hub running config#! / vrf main interface gre gre4 local 44.44.44.44
hub running config# / vrf main interface gre gre4 ttl 64
hub running config# / vrf main routing static ipv4-route 0.0.0.0/0 next-hop 44.44.44.1
hub running config# / vrf main routing nhrp enabled true
hub running config# / vrf main routing interface gre4 ip nhrp registration-no-unique true
hub running config# / vrf main routing interface gre4 ip nhrp network-id 123
hub running config# / vrf main routing interface gre4 ip nhrp holdtime 1200
hub running config# / vrf main routing bgp as 65000
hub running config# / vrf main routing bgp router-id 10.255.255.4
hub running config# / vrf main routing bgp ebgp-connected-route-check false
hub running config# / vrf main routing bgp ebgp-requires-policy false
hub running config# / vrf main routing bgp network-import-check false
hub running config# / vrf main routing bgp address-family ipv4-unicast network 192.168.4.0/24
hub running network 192.168.4.0/24# / vrf main routing bgp listen neighbor-range 10.255.255.0/24 neighbor-group GROUP
hub running network 192.168.4.0/24#! / vrf main routing bgp neighbor-group GROUP remote-as 65099
hub running network 192.168.4.0/24# / vrf main routing bgp neighbor-group GROUP address-family ipv4-unicast nexthop-self force true
hub running network 192.168.4.0/24# / vrf main routing bgp neighbor-group GROUP address-family ipv4-unicast route-reflector-client true
hub running network 192.168.4.0/24# / vrf main routing bgp address-family ipv4-unicast network 192.168.0.0/16
hub running network 192.168.0.0/16# / vrf main routing bgp address-family ipv4-unicast network 10.255.255.0/24
A router is configured to represent the network between the spokes and the hub with the following configuration:
router-vm running config# / vrf main interface physical eth1 port pci-b0s4
router-vm running config# / vrf main interface physical eth1 ipv4 address 11.11.11.1/24
router-vm running config# / vrf main interface physical eth2 port pci-b0s5
router-vm running config# / vrf main interface physical eth2 ipv4 address 22.22.22.1/24
router-vm running config# / vrf main interface physical eth3 port pci-b0s6
router-vm running config# / vrf main interface physical eth3 ipv4 address 33.33.33.1/24
router-vm running config# / vrf main interface physical eth4 port pci-b0s7
router-vm running config# / vrf main interface physical eth4 ipv4 address 44.44.44.1/24
We can check the NHRP cache entries on each node:
hub> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre4 local 10.255.255.4 44.44.44.44 44.44.44.44 -
gre4 dynamic 10.255.255.3 33.33.33.33 33.33.33.33 UT
gre4 dynamic 10.255.255.2 22.22.22.22 22.22.22.22 UT
gre4 dynamic 10.255.255.1 11.11.11.11 11.11.11.11 UT
spoke1> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre1 nhs 10.255.255.4 44.44.44.44 44.44.44.44 UT
gre1 local 10.255.255.1 11.11.11.11 11.11.11.11 -
spoke2> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre2 nhs 10.255.255.4 44.44.44.44 44.44.44.44 UT
gre2 local 10.255.255.2 22.22.22.22 22.22.22.22 -
spoke3> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre3 nhs 10.255.255.4 44.44.44.44 44.44.44.44 UT
gre3 local 10.255.255.3 33.33.33.33 33.33.33.33 -
And the NHRP connections: each spoke has an NHRP connection with the hub:
hub> show nhrp-connection
Src Dst Flags SAs Identity
44.44.44.44 22.22.22.22 n 0
44.44.44.44 33.33.33.33 n 0
44.44.44.44 11.11.11.11 n 0
spoke1> show nhrp-connection
Src Dst Flags SAs Identity
11.11.11.11 44.44.44.44 n 0
spoke2> show nhrp-connection
Src Dst Flags SAs Identity
22.22.22.22 44.44.44.44 n 0
spoke3> show nhrp-connection
Src Dst Flags SAs Identity
33.33.33.33 44.44.44.44 n 0
We can also dump the link layer contexts of gre4 interface on the hub, to
check the status of connected spokes, by using following command:
hub> show state / vrf main interface gre gre4 ipv4
ipv4
address 10.255.255.4/32
enabled true
neighbor 10.255.255.1 link-layer-address 11.11.11.11 state reachable
neighbor 10.255.255.2 link-layer-address 22.22.22.22 state reachable
neighbor 10.255.255.3 link-layer-address 33.33.33.33 state reachable
..
As a consequence, a NHRP route towards the protocol address of the hub is installed on each spoke. This way, each spoke can establish a BGP peering session and learn the routes to the hub:
spoke1> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 11.11.11.1, wan, weight 1, 00:00:30
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:28
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:28
C>* 10.255.255.1/32 is directly connected, gre1, 00:00:32
N>* 10.255.255.4/32 [10/0] is directly connected, gre1, weight 1, 00:00:29
C>* 11.11.11.0/24 is directly connected, wan, 00:00:32
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:28
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:28
C>* 192.168.1.0/24 is directly connected, lan, 00:00:32
B> 192.168.4.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:28
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:28
8 routes displayed.
spoke2> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 22.22.22.1, wan, weight 1, 00:00:28
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:26
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:26
C>* 10.255.255.2/32 is directly connected, gre2, 00:00:30
N>* 10.255.255.4/32 [10/0] is directly connected, gre2, weight 1, 00:00:27
C>* 22.22.22.0/24 is directly connected, wan, 00:00:30
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:26
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:26
C>* 192.168.2.0/24 is directly connected, lan, 00:00:30
B> 192.168.4.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:26
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:26
8 routes displayed.
spoke3> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 33.33.33.1, wan, weight 1, 00:00:24
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:22
* via 10.255.255.4, gre3 onlink, weight 1, 00:00:22
C>* 10.255.255.3/32 is directly connected, gre3, 00:00:26
N>* 10.255.255.4/32 [10/0] is directly connected, gre3, weight 1, 00:00:23
C>* 33.33.33.0/24 is directly connected, wan, 00:00:26
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:22
* via 10.255.255.4, gre3 onlink, weight 1, 00:00:22
C>* 192.168.3.0/24 is directly connected, lan, 00:00:26
B> 192.168.4.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:22
* via 10.255.255.4, gre3 onlink, weight 1, 00:00:22
8 routes displayed.
The hub installs a NHRP route for each spoke and learns the remote private networks through BGP:
hub> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 44.44.44.1, wan, weight 1, 00:00:34
N>* 10.255.255.1/32 [10/0] is directly connected, gre4, weight 1, 00:00:29
N>* 10.255.255.2/32 [10/0] is directly connected, gre4, weight 1, 00:00:24
N>* 10.255.255.3/32 [10/0] is directly connected, gre4, weight 1, 00:00:20
C>* 10.255.255.4/32 is directly connected, gre4, 00:00:36
C>* 44.44.44.0/24 is directly connected, wan, 00:00:36
B> 192.168.1.0/24 [20/0] via 10.255.255.1 (recursive), weight 1, 00:00:28
* via 10.255.255.1, gre4 onlink, weight 1, 00:00:28
B> 192.168.2.0/24 [20/0] via 10.255.255.2 (recursive), weight 1, 00:00:23
* via 10.255.255.2, gre4 onlink, weight 1, 00:00:23
B> 192.168.3.0/24 [20/0] via 10.255.255.3 (recursive), weight 1, 00:00:19
* via 10.255.255.3, gre4 onlink, weight 1, 00:00:19
C>* 192.168.4.0/24 is directly connected, lan, 00:00:36
10 routes displayed.
As you can see, the remote networks are learnt via BGP server from hub, and NHRP routes are set up with a higher priority (10) compared with BGP (20).
All traffic goes through the hub device. This is also the main drawback of this configuration, since it is not possible to establish spoke-to-spoke traffic.
Direct spoke-to-spoke communication¶
This enhancement enables direct spoke-to-spoke traffic, without routing through the hub. It uses the NHRP redirect feature and must be applied on top of a Dynamic configuration.
This feature, once enabled on the hub, detects a traffic flow that enters and
exits the hub through the same GRE interface. This traffic flow can be
redirected as the hub is only routing the flow between two spokes. Then the
NHRP daemon of the hub issues a NHRP traffic indication message containing the
detected packet to the emitter. A spoke receiving such a message performs a
NHRP resolution request to find the NBMA address of the destination and then
continues the communication directly with the destination spoke.
We first enable the redirect feature on the hub:
hub running config# / vrf main routing interface gre4 ip nhrp redirect true
hub running config#! / vrf main routing nhrp hub-mode true
Note
The traffic flow detection mechanism is implemented by the fast path. Therefore, the fast path must be enabled on the hub to benefit from the spoke-to-spoke direct communication.
Once set, the
redirectfunctionality can not be configured on GRE interfaces from other VRs.
This enables the sampling of the overlay traffic of the gre4 interface. Only the
packets that come from this interface and are forwarded through the same
interface are eligible for capture. In order to avoid exhausting the device
CPU, the NHRP daemon applies a default sampling rate of 1 packet out of
400. If the destination address matches one of the registered spokes, the hub
attaches the packet to a NHRP traffic indication packet, and sends it to the
sender spoke.
Then we enable the shortcut feature on the spokes so that these messages get
processed:
spoke1 running config# / vrf main routing interface gre1 ip nhrp shortcut true
spoke2 running config# / vrf main routing interface gre2 ip nhrp shortcut true
spoke3 running config# / vrf main routing interface gre3 ip nhrp shortcut true
As the message contains the original destination IP, upon reception, the spoke sends a NHRP resolution request, routed by the hub and replied by the destination spoke. Then both spokes continue the communication bypassing the hub.
On direct spoke to spoke communication, one can distinguish two kinds of traffic. Each traffic will result in NHRP messages exchange involving spokes and hub, as depicted above.
The
protocol addresstoprotocol addresscommunication between spokes. This is done by issuing traffic between those IP addresses. It will result in the creation of a dynamic cache entry.The traffic between private networks located behind GRE interfaces. In addition to the creation of the above mentioned dynamic cache entry, it will result in the creation of a shortcut entry.
Dynamic cache entry¶
Let’s perform traffic between protocol address of each spoke. To achieve this
communication, NHRP protocol will create a dynamic cache entry identifying the
two spokes that want to communicate together. From dataplane perspective, a
NHRP connected route will be setup on the GRE interface, after the mutual
resolution. Example is given below by issuing following cmd ping command:
spoke1> cmd ping 10.255.255.2 source 10.255.255.1 rate 100 count 1000 | match packets
1000 packets transmitted, 1000 received, 0% packet loss, time 9985ms
The cache entries will look like below:
spoke1> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre1 nhs 10.255.255.4 44.44.44.44 44.44.44.44 UT
gre1 dynamic 10.255.255.2 22.22.22.22 22.22.22.22 UT
gre1 local 10.255.255.1 11.11.11.11 11.11.11.11 -
spoke2> show nhrp cache
Iface Type Protocol NBMA Claimed NBMA Flags Identity
gre2 nhs 10.255.255.4 44.44.44.44 44.44.44.44 UT
gre2 local 10.255.255.2 22.22.22.22 22.22.22.22 -
gre2 dynamic 10.255.255.1 11.11.11.11 11.11.11.11 UT
It is worth to be noted that in the last spoke2 dump, NBMA address can be
the hub NBMA address, because the resolution request coming from spoke1
has been forwarded by hub, and no direct resolution request has been received
by spoke2 from spoke1. To be sure that the association between spoke1
protocol address and its NBMA address has been correctly done, the attribute
NBMA-NAT-OA-Address in show nhrp command indicates the real NBMA address
used.
spoke2> show nhrp
Type: nhs
Flags: up used
Protocol-Address: 10.255.255.4/32
NBMA-Address: 44.44.44.44
Type: local
Flags:
Protocol-Address: 10.255.255.2/32
Type: dynamic
Flags: up used
Protocol-Address: 10.255.255.1/32
NBMA-Address: 11.11.11.11
NBMA-NAT-OA-Address: 11.11.11.11
We can see that there is now an additional NHRP connection from spoke1 to
spoke2:
hub> show nhrp-connection
Src Dst Flags SAs Identity
44.44.44.44 22.22.22.22 n 0
44.44.44.44 33.33.33.33 n 0
44.44.44.44 11.11.11.11 n 0
spoke1> show nhrp-connection
Src Dst Flags SAs Identity
11.11.11.11 22.22.22.22 n 0
11.11.11.11 44.44.44.44 n 0
Like it has been noted for spoke2 with show nhrp cache dump, the remote entry
to 11.11.11.11 may not be present, while no direct resolution has been identified
by spoke2.
spoke2> show nhrp-connection
Src Dst Flags SAs Identity
22.22.22.22 44.44.44.44 n 0
22.22.22.22 11.11.11.11 n 0
As indicated below, a new NHRP connected route entry has been set up. Note that up to
now, only traffic between Protocol Address of spokes has been initiated; and that
traffic between private networks (192.168.x.0/24) still uses the hub.
spoke1> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 11.11.11.1, wan, weight 1, 00:00:57
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:21
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:21
C>* 10.255.255.1/32 is directly connected, gre1, 00:00:59
N>* 10.255.255.2/32 [10/0] is directly connected, gre1, weight 1, 00:00:16
N>* 10.255.255.4/32 [10/0] is directly connected, gre1, weight 1, 00:00:21
C>* 11.11.11.0/24 is directly connected, wan, 00:00:59
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:21
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:21
C>* 192.168.1.0/24 is directly connected, lan, 00:00:59
B> 192.168.4.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:21
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:21
9 routes displayed.
spoke2> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 22.22.22.1, wan, weight 1, 00:00:54
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:21
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:21
N>* 10.255.255.1/32 [10/0] is directly connected, gre2, weight 1, 00:00:17
C>* 10.255.255.2/32 is directly connected, gre2, 00:00:56
N>* 10.255.255.4/32 [10/0] is directly connected, gre2, weight 1, 00:00:21
C>* 22.22.22.0/24 is directly connected, wan, 00:00:56
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:21
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:21
C>* 192.168.2.0/24 is directly connected, lan, 00:00:56
B> 192.168.4.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:21
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:21
9 routes displayed.
Shortcut entry¶
Now, let us perform a cmd ping command between private networks. This command will
result in the capture of this traffic on the hub and will lead to the creation of
a shortcut entry on spokes.
spoke1> cmd ping 192.168.2.1 source 192.168.1.1 rate 100 count 1000 | match packets
1000 packets transmitted, 1000 received, 0% packet loss, time 9995ms
The NHRP entries help in creating route entries linked to private networks behind
each GRE device. For instance, sub-network 192.168.2.0/24 is discovered by NHRP,
on spoke1, when traffic is issued from spoke1 to that destination network. On hub,
the traffic is intercepted and sent to spoke2 for NHRP resolution. spoke2 parses
the available networks hidden behind GRE interface, and if present, transmits a
resolution response to spoke1 which creates a nexthop route entry linked to spoke2
dynamic entry, as follows:
spoke1> show nhrp shortcut
Type Prefix Via Identity
dynamic 192.168.2.0/24 10.255.255.2
spoke1> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 11.11.11.1, wan, weight 1, 00:01:14
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:38
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:38
C>* 10.255.255.1/32 is directly connected, gre1, 00:01:16
N>* 10.255.255.4/32 [10/0] is directly connected, gre1, weight 1, 00:00:38
C>* 11.11.11.0/24 is directly connected, wan, 00:01:16
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:38
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:38
C>* 192.168.1.0/24 is directly connected, lan, 00:01:16
N>* 192.168.2.0/24 [10/0] via 10.255.255.2, gre1 onlink, weight 1, 00:00:13
N>* 192.168.2.1/32 [10/0] is directly connected, gre1, weight 1, 00:00:13
B> 192.168.4.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:38
* via 10.255.255.4, gre1 onlink, weight 1, 00:00:38
10 routes displayed.
spoke2> show nhrp shortcut
Type Prefix Via Identity
dynamic 192.168.1.0/24 10.255.255.1
spoke2> show ipv4-routes
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, N - NHRP, T - Table
> - selected route, * - FIB route, r - rejected, b - backup
L3VRF default:
S>* 0.0.0.0/0 [1/0] via 22.22.22.1, wan, weight 1, 00:01:10
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:37
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:37
N>* 10.255.255.1/32 [10/0] is directly connected, gre2, weight 1, 00:00:33
C>* 10.255.255.2/32 is directly connected, gre2, 00:01:12
N>* 10.255.255.4/32 [10/0] is directly connected, gre2, weight 1, 00:00:37
C>* 22.22.22.0/24 is directly connected, wan, 00:01:12
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:37
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:37
N>* 192.168.1.0/24 [10/0] via 10.255.255.1, gre2 onlink, weight 1, 00:00:13
N>* 192.168.1.1/32 [10/0] is directly connected, gre2, weight 1, 00:00:13
C>* 192.168.2.0/24 is directly connected, lan, 00:01:12
B> 192.168.4.0/24 [20/0] via 10.255.255.4 (recursive), weight 1, 00:00:37
* via 10.255.255.4, gre2 onlink, weight 1, 00:00:37
11 routes displayed.
As can be seen, a NHRP onlink route has been created, and reflects the NHRP shortcut entry created. This entry relies on the NHRP dynamic cache entry previously created when initiating traffic between Protocol Address of spokes.
Note
To bring more clarity, the chapter presented how the cache entry is created, then how the shortcut entry is created, according to each incoming traffic. This said, in reality, launching private network traffic will result in the creation of the two NHRP routes necessary to make spoke-to-spoke communication.
Nexthop route to 192.168.2.0/24 is introduced, and partially overrides
192.168.0.0/16 defined by BGP. As illustrated, the routing decision in NHRP is
made thanks to routing information the protocol has when receiving a resolution
request. The traffic indication was about a specific IP whereas the resulting route
reflects the route of the remote spoke. In our case, the network 192.168.2.0/24
network information is sent from spoke2 to spoke1. All routing information
contained in the vrf where GRE interface sits can be used.
Spoke to spoke expiration¶
Spoke to spoke cache entries are not maintained based on the traffic presence between two spokes. Instead, before the cache expiration, NHRP resolution requests and reply are exchanged between both spokes to renew the cache lifetime.