DMVPN and NHRP Configuration¶
NHRP use case example¶
The following configuration examples apply on the topology described in the
above picture: 3 spokes connected to a hub.
The first two chapters describe how to use GRE interfaces and NHRP to
establish connections between the spokes via the hub, either statically or
dynamically.
The last chapter introduces the redirect feature that enables dynamic tunnel
establishment between spokes.
For all those examples, NHRP configuration requires that the IP address of GRE interface be configured with 32 bitmask, as follows. Configuring an other bitmask will lead to a misconfiguration problem.
spoke1
spoke1 running config# vrf main
spoke1 running vrf main# interface gre gre1
spoke1 running gre gre1# ipv4 address 10.255.255.1/32
Static configuration¶
We can define static NHRP entries for remote endpoints on each node. From a protocol point of view, hub and spokes don’t exchange messages then. The hub declares the protocol address and the NBMA address of the spokes. The spokes only declare a NHRP entry for the hub. In that case, spoke-to-spoke communications flow through the hub. Therefore the hub declares a multipoint GRE interface facing all the spokes and the spokes declare a GRE interface facing the hub only. In addition, each node runs a BGP instance to advertise its private network and learn the route towards the private subnets of the other nodes.
First we configure the three spokes with a static NHRP entry for the hub:
spoke1
spoke1 running config# vrf main
spoke1 running vrf main# interface physical wan
spoke1 running physical wan#! port pci-b0s4
spoke1 running physical wan# ipv4 address 11.11.11.11/24
spoke1 running physical wan# ..
spoke1 running interface# physical lan
spoke1 running physical lan#! port pci-b0s5
spoke1 running physical lan# ipv4 address 192.168.1.1/24
spoke1 running physical lan# ..
spoke1 running interface# gre gre1
spoke1 running gre gre1#! ipv4 address 10.255.255.1/32
spoke1 running gre gre1#! link-interface wan
spoke1 running gre gre1#! local 11.11.11.11
spoke1 running gre gre1# remote 44.44.44.44
spoke1 running gre gre1# .. ..
spoke1 running vrf main# routing
spoke1 running routing# static ipv4-route 0.0.0.0/0 next-hop 11.11.11.1
spoke1 running routing# nhrp enabled true
spoke1 running routing# interface gre1 ip nhrp
spoke1 running nhrp# registration-no-unique true
spoke1 running nhrp# network-id 123
spoke1 running nhrp# holdtime 1200
spoke1 running nhrp# nhrp-map 10.255.255.4 nbma 44.44.44.44
spoke1 running nhrp# .. .. ..
spoke1 running routing# bgp
spoke1 running bgp#! as 65000
spoke1 running bgp# router-id 10.255.255.1
spoke1 running bgp# address-family ipv4-unicast network 192.168.1.0/24
spoke1 running network 192.168.1.0/24# ..
spoke1 running ipv4-unicast# network 10.255.255.1/32
spoke1 running network 10.255.255.255.1/32# .. .. ..
spoke1 running bgp# neighbor 10.255.255.4
spoke1 running neighbor 10.255.255.4#! remote-as 65000
spoke1 running neighbor 10.255.255.4# commit
spoke2
spoke2 running config# vrf main
spoke2 running vrf main# interface physical wan
spoke2 running physical wan#! port pci-b0s4
spoke2 running physical wan# ipv4 address 22.22.22.22/24
spoke2 running physical wan# ..
spoke2 running interface# physical lan
spoke2 running physical lan#! port pci-b0s5
spoke2 running physical lan# ipv4 address 192.168.2.1/24
spoke2 running physical lan# ..
spoke2 running interface# gre gre2
spoke2 running gre gre2#! ipv4 address 10.255.255.2/32
spoke2 running gre gre2#! link-interface wan
spoke2 running gre gre2#! local 22.22.22.22
spoke2 running gre gre2# remote 44.44.44.44
spoke2 running gre gre2# .. ..
spoke2 running vrf main# routing
spoke2 running routing# static ipv4-route 0.0.0.0/0 next-hop 22.22.22.1
spoke2 running routing# nhrp enabled true
spoke2 running routing# interface gre2 ip nhrp
spoke2 running nhrp# registration-no-unique true
spoke2 running nhrp# network-id 123
spoke2 running nhrp# holdtime 1200
spoke2 running nhrp# nhrp-map 10.255.255.4 nbma 44.44.44.44
spoke2 running nhrp# .. .. ..
spoke2 running routing# bgp
spoke2 running bgp#! as 65000
spoke2 running bgp# router-id 10.255.255.2
spoke2 running bgp# address-family ipv4-unicast network 192.168.2.0/24
spoke1 running network 192.168.2.0/24# ..
spoke1 running ipv4-unicast# network 10.255.255.2/32
spoke1 running network 10.255.255.255.2/32# .. .. ..
spoke2 running bgp# neighbor 10.255.255.4
spoke2 running neighbor 10.255.255.4#! remote-as 65000
spoke2 running neighbor 10.255.255.4# commit
spoke3
spoke3 running config# vrf main
spoke3 running vrf main# interface physical wan
spoke3 running physical wan#! port pci-b0s4
spoke3 running physical wan# ipv4 address 33.33.33.33/24
spoke3 running physical wan# ..
spoke3 running interface# physical lan
spoke3 running physical lan#! port pci-b0s5
spoke3 running physical lan# ipv4 address 192.168.3.1/24
spoke3 running physical lan# ..
spoke3 running interface# gre gre3
spoke3 running gre gre3# ipv4 address 10.255.255.3/32
spoke3 running gre gre3# link-interface wan
spoke3 running gre gre3# local 33.33.33.33
spoke3 running gre gre3# remote 44.44.44.44
spoke3 running gre gre3# .. ..
spoke3 running vrf main# routing
spoke3 running routing# static ipv4-route 0.0.0.0/0 next-hop 33.33.33.1
spoke3 running routing# nhrp enabled true
spoke3 running routing# interface gre3 ip nhrp
spoke3 running nhrp# registration-no-unique true
spoke3 running nhrp# network-id 123
spoke3 running nhrp# holdtime 1200
spoke3 running nhrp# nhrp-map 10.255.255.4 nbma 44.44.44.44
spoke3 running nhrp# .. .. ..
spoke3 running routing# bgp
spoke3 running bgp#! as 65000
spoke3 running bgp# router-id 10.255.255.3
spoke3 running bgp# address-family ipv4-unicast network 192.168.3.0/24
spoke1 running network 192.168.3.0/24# ..
spoke1 running ipv4-unicast# network 10.255.255.3/32
spoke1 running network 10.255.255.255.3/32# .. .. ..
spoke3 running bgp# neighbor 10.255.255.4
spoke3 running neighbor 10.255.255.4#! remote-as 65000
spoke3 running neighbor 10.255.255.4# commit
Similarly, we configure static NHRP entries for each spoke on the hub. The main difference is that we configure a multipoint GRE on this node, by omitting a remote address, to be able to reach each spoke:
hub
hub running config# vrf main
hub running vrf main# interface physical wan
hub running physical wan#! port pci-b0s4
hub running physical wan# ipv4 address 44.44.44.44/24
hub running physical wan# ..
hub running interface# physical lan
hub running physical lan#! port pci-b0s5
hub running physical lan# ipv4 address 192.168.4.1/24
hub running physical lan# ..
hub running interface# gre gre4
hub running gre gre4#! ipv4 address 10.255.255.4/32
hub running gre gre4#! link-interface wan
hub running gre gre4#! local 44.44.44.44
hub running gre gre4# .. ..
hub running vrf main# routing
hub running routing# static ipv4-route 0.0.0.0/0 next-hop 44.44.44.1
hub running routing# nhrp enabled true
hub running routing# interface gre4 ip nhrp
hub running nhrp# registration-no-unique true
hub running nhrp# network-id 123
hub running nhrp# holdtime 1200
hub running nhrp# nhrp-map 10.255.255.1 nbma 11.11.11.11
hub running nhrp# nhrp-map 10.255.255.2 nbma 22.22.22.22
hub running nhrp# nhrp-map 10.255.255.3 nbma 33.33.33.33
hub running nhrp# .. .. ..
hub running routing# bgp
hub running bgp#! as 65000
hub running bgp# router-id 10.255.255.4
hub running bgp# address-family ipv4-unicast
hub running ipv4-unicast# network 192.168.4.0/24
hub running network 192.168.4.0/24# .. .. ..
hub running bgp# listen neighbor-range 10.255.255.0/24 neighbor-group nhrp_group
hub running bgp#! neighbor-group nhrp_group
hub running neighbor-group nhrp_group#! remote-as 65000
hub running neighbor-group nhrp_group# address-family ipv4-unicast
hub running ipv4-unicast# nexthop-self force true
hub running ipv4-unicast# route-reflector-client true
hub running ipv4-unicast# commit
We can check the NHRP cache entries on each node:
hub> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre4 local 10.255.255.4 - -
gre4 static 10.255.255.3 33.33.33.33
gre4 static 10.255.255.2 22.22.22.22
gre4 static 10.255.255.1 11.11.11.11
spoke1> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre1 static 10.255.255.4 44.44.44.44
gre1 local 10.255.255.1 - -
spoke2> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre2 static 10.255.255.4 44.44.44.44
gre2 local 10.255.255.2 - -
spoke3> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre3 static 10.255.255.4 44.44.44.44
gre3 local 10.255.255.3 - -
We can see that there is a NHRP connection between the hub and each device:
hub> show nhrp-connection
Src Dst Flags SAs Identity
44.44.44.44 22.22.22.22 n 0
44.44.44.44 33.33.33.33 n 0
44.44.44.44 11.11.11.11 n 0
spoke1> show nhrp-connection
Src Dst Flags SAs Identity
11.11.11.11 44.44.44.44 n 0
spoke2> show nhrp-connection
Src Dst Flags SAs Identity
22.22.22.22 44.44.44.44 n 0
spoke3> show nhrp-connection
Src Dst Flags SAs Identity
33.33.33.33 44.44.44.44 n 0
As a consequence, a NHRP route towards the protocol address of the hub is installed on each spoke. This way, they can establish a BGP peering and learn the routes to other nodes:
spoke1> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 11.11.11.1, wan, 00:01:24
C>* 10.255.255.1/32 is directly connected, gre1, 00:01:24
B> 10.255.255.2/32 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
B> 10.255.255.3/32 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
N>* 10.255.255.4/32 [10/0] is directly connected, gre1, 00:01:24
C>* 11.11.11.0/24 is directly connected, wan, 00:01:24
C>* 192.168.1.0/24 is directly connected, lan, 00:01:24
B> 192.168.2.0/24 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
B> 192.168.3.0/24 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
B> 192.168.4.0/24 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
spoke2> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 22.22.22.1, wan, 00:07:21
B> 10.255.255.1/32 [200/0] via 10.255.255.4 (recursive), 00:07:20
* via 10.255.255.4, gre2 onlink, 00:07:20
C>* 10.255.255.2/32 is directly connected, gre2, 00:07:21
B> 10.255.255.3/32 [200/0] via 10.255.255.4 (recursive), 00:07:20
* via 10.255.255.4, gre2 onlink, 00:07:20
N>* 10.255.255.4/32 [10/0] is directly connected, gre2, 00:07:21
C>* 22.22.22.0/24 is directly connected, wan, 00:07:21
B> 192.168.1.0/24 [200/0] via 10.255.255.4 (recursive), 00:07:20
* via 10.255.255.4, gre2 onlink, 00:07:20
C>* 192.168.2.0/24 is directly connected, lan, 00:07:21
B> 192.168.3.0/24 [200/0] via 10.255.255.4 (recursive), 00:07:20
* via 10.255.255.4, gre2 onlink, 00:07:20
B> 192.168.4.0/24 [200/0] via 10.255.255.4 (recursive), 00:07:20
* via 10.255.255.4, gre2 onlink, 00:07:20
spoke3> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 33.33.33.1, wan, 00:08:24
B> 10.255.255.1/32 [200/0] via 10.255.255.4 (recursive), 00:08:22
* via 10.255.255.4, gre3 onlink, 00:08:22
B> 10.255.255.2/32 [200/0] via 10.255.255.4 (recursive), 00:08:23
* via 10.255.255.4, gre3 onlink, 00:08:23
C>* 10.255.255.3/32 is directly connected, gre3, 00:08:24
N>* 10.255.255.4/32 [10/0] is directly connected, gre3, 00:08:24
C>* 33.33.33.0/24 is directly connected, wan, 00:08:24
B> 192.168.1.0/24 [200/0] via 10.255.255.4 (recursive), 00:08:22
* via 10.255.255.4, gre3 onlink, 00:08:22
B> 192.168.2.0/24 [200/0] via 10.255.255.4 (recursive), 00:08:23
* via 10.255.255.4, gre3 onlink, 00:08:23
C>* 192.168.3.0/24 is directly connected, lan, 00:08:24
B> 192.168.4.0/24 [200/0] via 10.255.255.4 (recursive), 00:08:23
* via 10.255.255.4, gre3 onlink, 00:08:23
The hub installs a NHRP route for each spoke and learns the remote private networks through BGP:
hub> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 44.44.44.1, wan, 00:09:05
B 10.255.255.1/32 [200/0] via 10.255.255.1 inactive, 00:09:04
N>* 10.255.255.1/32 [10/0] is directly connected, gre4, 00:09:05
B 10.255.255.2/32 [200/0] via 10.255.255.2 inactive, 00:09:04
N>* 10.255.255.2/32 [10/0] is directly connected, gre4, 00:09:05
B 10.255.255.3/32 [200/0] via 10.255.255.3 inactive, 00:09:04
N>* 10.255.255.3/32 [10/0] is directly connected, gre4, 00:09:05
C>* 10.255.255.4/32 is directly connected, gre4, 00:09:05
C>* 44.44.44.0/24 is directly connected, wan, 00:09:05
B> 192.168.1.0/24 [200/0] via 10.255.255.1 (recursive), 00:09:04
* via 10.255.255.1, gre4 onlink, 00:09:04
B> 192.168.2.0/24 [200/0] via 10.255.255.2 (recursive), 00:09:04
* via 10.255.255.2, gre4 onlink, 00:09:04
B> 192.168.3.0/24 [200/0] via 10.255.255.3 (recursive), 00:09:04
* via 10.255.255.3, gre4 onlink, 00:09:04
C>* 192.168.4.0/24 is directly connected, lan, 00:09:05
Dynamic configuration¶
Static configuration doesn’t scale well as you have to declare each spoke individually on the hub. We can fully leverage NHRP and use dynamic configuration instead. In this configuration, the NHRP server on the hub listens for registration requests. Each spoke declares the hub as a NHRP server and thus registers upon startup using NHRP control packets. With the basic setup below, spoke-to-spoke communication still runs through the hub but can then be improved to allow Direct spoke-to-spoke communication. That’s why we configure point to multipoint GRE interface on both spoke and hub side.
BGP configuration¶
In addition, each node runs a BGP instance to advertise its private network. As the chapter deals with spoke to hub communication, only hub is interested in getting the private networks located behind spokes.
iBGP configuration¶
It is possible to keep using iBGP sessions between spokes and hub, like it
has been done on previous experiment.
In that case, spokes will install iBGP routes to reach private networks from
other spokes, but by using hub as nexthop. A snapshot of configuration can be
illustrated below with spoke1, spoke2 and hub devices:
Below dump gives the initial ipv4 routing table on spoke1. One can see routes
installed to reach spoke2, spoke3 and hub.
spoke1> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 11.11.11.1, wan, 00:01:24
C>* 10.255.255.1/32 is directly connected, gre1, 00:01:24
B> 10.255.255.2/32 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
B> 10.255.255.3/32 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
N>* 10.255.255.4/32 [10/0] is directly connected, gre1, 00:01:24
C>* 11.11.11.0/24 is directly connected, wan, 00:01:24
C>* 192.168.1.0/24 is directly connected, lan, 00:01:24
B> 192.168.2.0/24 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
B> 192.168.3.0/24 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
B> 192.168.4.0/24 [200/0] via 10.255.255.4 (recursive), 00:01:23
* via 10.255.255.4, gre1 onlink, 00:01:23
eBGP configuration¶
It is also possible to avoid transmitting the private networks information to all spokes. Two reasons for that:
In that setup where data is always passing through the hub, there is no point giving this information to spokes that always point to hub.
This routing information is redundant with NHRP when direct spoke to spoke communication will be put in place. Even if iBGP routes were used, NHRP routes to those private networks would override those BGP routes, and NHRP routes would directly have the nexthop set to the appropriate spoke.
For the rest of this chapter, We choose to use eBGP configuration. The hub will
transmit two aggregated networks, 192.168.0.0/16 and 10.255.255.0/24. This stands
for the subnetworks to all spokes. You can note that eBGP requires peers to be
directly connected. As the route to reach the hub IP on the spoke is a NHRP route
and not a connected route, an additional configuration is put in place, in order to
inject incoming BGP network. This is done via ebgp-connected-route-check BGP
configuration command:
Note
Using eBGP avoids spokes to learn BGP route entries from spokes on same AS coming from an external AS. This specificity works because the spokes BGP configurations use the same AS and are not fully meshed.
Note
An other alternative to defining aggregated networks on hub side is to define a default-route at hub side, like below BGP command shows it:
hub running bgp# neighour-group nhrp_group address-family ipv4-unicast
hub running ipv4-unicast# default-originate
hub running default-originate# ..
hub running ipv4-unicast# commit
For that, there must not be any conflicts with learnt default route
of the ISPs of local spokes. A good practice would be to build GRE
interfaces on separate VRF from wan interface. Below example
illustrates how to move NHRP contexts and lan interface to a VRF
nhrp. In that case, NHRP routes will be visible from the VRF
nhrp.
spoke1 running config# vrf nhrp
spoke1 running vrf nhrp# interface physical lan
spoke1 running physical lan#! port pci-b0s5
spoke1 running physical lan# ipv4 address 192.168.1.1/24
spoke1 running physical lan# ..
spoke1 running interface# gre gre1
spoke1 running gre gre1#! ipv4 address 10.255.255.1/32
spoke1 running gre gre1#! link-interface wan
spoke1 running gre gre1#! link-vrf main
spoke1 running gre gre1#! local 11.11.11.11
spoke1 running gre gre1# ttl 64
spoke1 running gre gre1# .. ..
spoke1 running vrf nhrp# routing
spoke1 running routing# nhrp enabled true
spoke1 running routing# interface gre1 ip nhrp
spoke1 running nhrp# registration-no-unique true
spoke1 running nhrp# network-id 123
spoke1 running nhrp# holdtime 1200
spoke1 running nhrp# nhrp-nhs dynamic nbma 44.44.44.44
spoke1 running nhrp# .. .. ..
spoke1 running routing# bgp
spoke1 running bgp#! as 65099
spoke1 running bgp# ebgp-connected-route-check false
spoke1 running bgp# router-id 10.255.255.1
spoke1 running bgp# address-family ipv4-unicast network 192.168.1.0/24
spoke1 running network 192.168.1.0/24# .. .. ..
spoke1 running bgp# neighbor 10.255.255.4
spoke1 running neighbor 10.255.255.4#! remote-as 65000
spoke1 running neighbor 10.255.255.4# commit
Configuration¶
The following configuration applies on the same topology with one hub and three spokes.
spoke1
spoke1 running config# vrf main
spoke1 running vrf main# interface physical wan
spoke1 running physical wan#! port pci-b0s4
spoke1 running physical wan# ipv4 address 11.11.11.11/24
spoke1 running physical wan# ..
spoke1 running interface# physical lan
spoke1 running physical lan#! port pci-b0s5
spoke1 running physical lan# ipv4 address 192.168.1.1/24
spoke1 running physical lan# ..
spoke1 running interface# gre gre1
spoke1 running gre gre1#! ipv4 address 10.255.255.1/32
spoke1 running gre gre1#! link-interface wan
spoke1 running gre gre1#! local 11.11.11.11
spoke1 running gre gre1#! ttl 64
spoke1 running gre gre1# .. ..
spoke1 running vrf main# routing
spoke1 running routing# static ipv4-route 0.0.0.0/0 next-hop 11.11.11.1
spoke1 running routing# nhrp enabled true
spoke1 running routing# interface gre1 ip nhrp
spoke1 running nhrp# registration-no-unique true
spoke1 running nhrp# network-id 123
spoke1 running nhrp# holdtime 1200
spoke1 running nhrp# nhrp-nhs dynamic nbma 44.44.44.44
spoke1 running nhrp# .. .. ..
spoke1 running routing# bgp
spoke1 running bgp#! as 65099
spoke1 running bgp# ebgp-connected-route-check false
spoke1 running bgp# router-id 10.255.255.1
spoke1 running bgp# address-family ipv4-unicast network 192.168.1.0/24
spoke1 running network 192.168.1.0/24# .. .. ..
spoke1 running bgp# neighbor 10.255.255.4
spoke1 running neighbor 10.255.255.4#! remote-as 65000
spoke1 running neighbor 10.255.255.4# commit
spoke2
spoke2 running config# vrf main
spoke2 running vrf main# interface physical wan
spoke2 running physical wan#! port pci-b0s4
spoke2 running physical wan# ipv4 address 22.22.22.22/24
spoke2 running physical wan# ..
spoke2 running interface# physical lan
spoke2 running physical lan#! port pci-b0s5
spoke2 running physical lan# ipv4 address 192.168.2.1/24
spoke2 running physical lan# ..
spoke2 running interface# gre gre2
spoke2 running gre gre2#! ipv4 address 10.255.255.2/32
spoke2 running gre gre2#! link-interface wan
spoke2 running gre gre2#! local 22.22.22.22
spoke2 running gre gre2# ttl 64
spoke2 running gre gre2# .. ..
spoke2 running vrf main# routing
spoke2 running routing# static ipv4-route 0.0.0.0/0 next-hop 22.22.22.1
spoke2 running routing# nhrp enabled true
spoke2 running routing# interface gre2 ip nhrp
spoke2 running nhrp# registration-no-unique true
spoke2 running nhrp# network-id 123
spoke2 running nhrp# holdtime 1200
spoke2 running nhrp# nhrp-nhs dynamic nbma 44.44.44.44
spoke2 running nhrp# .. .. ..
spoke2 running routing# bgp
spoke2 running bgp#! as 65099
spoke2 running bgp# ebgp-connected-route-check false
spoke2 running bgp# router-id 10.255.255.2
spoke2 running bgp# address-family ipv4-unicast network 192.168.2.0/24
spoke2 running network 192.168.2.0/24# .. .. ..
spoke2 running bgp# neighbor 10.255.255.4
spoke2 running neighbor 10.255.255.4#! remote-as 65000
spoke2 running neighbor 10.255.255.4# commit
spoke3
spoke3 running config# vrf main
spoke3 running vrf main# interface physical wan
spoke3 running physical wan#! port pci-b0s4
spoke3 running physical wan# ipv4 address 33.33.33.33/24
spoke3 running physical wan# ..
spoke3 running interface# physical lan
spoke3 running physical lan#! port pci-b0s5
spoke3 running physical lan# ipv4 address 192.168.3.1/24
spoke3 running physical lan# ..
spoke3 running interface# gre gre3
spoke3 running gre gre3#! ipv4 address 10.255.255.3/32
spoke3 running gre gre3#! link-interface wan
spoke3 running gre gre3#! local 33.33.33.33
spoke3 running gre gre3# ttl 64
spoke3 running gre gre3# .. ..
spoke3 running vrf main# routing
spoke3 running routing# static ipv4-route 0.0.0.0/0 next-hop 33.33.33.1
spoke3 running routing# nhrp enabled true
spoke3 running routing# interface gre3 ip nhrp
spoke3 running nhrp# registration-no-unique true
spoke3 running nhrp# network-id 123
spoke3 running nhrp# holdtime 1200
spoke3 running nhrp# nhrp-nhs dynamic nbma 44.44.44.44
spoke3 running nhrp# .. .. ..
spoke3 running routing# bgp
spoke3 running bgp#! as 65099
spoke3 running bgp# ebgp-connected-route-check false
spoke3 running bgp# router-id 10.255.255.3
spoke3 running bgp# address-family ipv4-unicast network 192.168.3.0/24
spoke3 running network 192.168.3.0/24# .. .. ..
spoke3 running bgp# neighbor 10.255.255.4
spoke3 running neighbor 10.255.255.4#! remote-as 65000
spoke3 running neighbor 10.255.255.4# commit
Similarly, we do not declare any spoke entry on the hub. This node only listens for NHRP registration requests and then for BGP peerings coming from the spokes.
hub
hub running config# vrf main
hub running vrf main# interface physical wan
hub running physical wan#! port pci-b0s4
hub running physical wan# ipv4 address 44.44.44.44/24
hub running physical wan# ..
hub running interface# physical lan
hub running physical lan#! port pci-b0s5
hub running physical lan# ipv4 address 192.168.4.1/24
hub running physical lan# ..
hub running interface# gre gre4
hub running gre gre4#! ipv4 address 10.255.255.4/32
hub running gre gre4#! link-interface wan
hub running gre gre4#! local 44.44.44.44
hub running gre gre1# ttl 64
hub running gre gre4# .. ..
hub running vrf main# routing
hub running routing# static ipv4-route 0.0.0.0/0 next-hop 44.44.44.1
hub running routing# nhrp enabled true
hub running routing# interface gre4 ip nhrp
hub running nhrp# registration-no-unique true
hub running nhrp# network-id 123
hub running nhrp# holdtime 1200
hub running nhrp# .. .. ..
hub running routing# bgp
hub running bgp#! as 65000
hub running bgp# router-id 10.255.255.4
hub running bgp# ebgp-connected-route-check false
hub running bgp# listen neighbor-range 10.255.255.0/24 neighbor-group nhrp_group
hub running bgp#! neighbor-group nhrp_group
hub running neighbor-group nhrp_group#! remote-as 65099
hub running neighbor-group nhrp_group# ..
hub running bgp# address-family ipv4-unicast
hub running ipv4-unicast# network 10.255.255.0/24
hub running network 10.255.255.255.0/24# ..
hub running ipv4-unicast# network 192.168.0.0/16
hub running network 192.168.0.0/16# commit
We can check the NHRP cache entries on each node:
hub> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre4 local 10.255.255.4 - -
gre4 dynamic 10.255.255.3 33.33.33.33 T
gre4 dynamic 10.255.255.2 22.22.22.22 T
gre4 dynamic 10.255.255.1 11.11.11.11 T
spoke1> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre1 nhs 10.255.255.4 44.44.44.44 T
gre1 local 10.255.255.1 - -
spoke2> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre2 nhs 10.255.255.4 44.44.44.44 T
gre2 local 10.255.255.2 - -
spoke3> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre3 nhs 10.255.255.4 44.44.44.44 T
gre3 local 10.255.255.3 - -
And the NHRP connections: each spoke has an NHRP connection with the hub:
hub> show nhrp-connection
Src Dst Flags SAs Identity
44.44.44.44 22.22.22.22 n 0
44.44.44.44 33.33.33.33 n 0
44.44.44.44 11.11.11.11 n 0
spoke1> show nhrp-connection
Src Dst Flags SAs Identity
11.11.11.11 44.44.44.44 n 0
spoke2 show nhrp-connection
Src Dst Flags SAs Identity
22.22.22.22 44.44.44.44 n 0
spoke3> show nhrp-connection
Src Dst Flags SAs Identity
33.33.33.33 44.44.44.44 n 0
We can also dump the link layer contexts of gre4 interface on the hub, to
check the status of connected spokes, by using following command:
hub> show state / vrf main interface gre gre4 ipv4
ipv4
address 10.255.255.4/32
neighbor 10.255.255.2 link-layer-address 22.22.22.22 state reachable
neighbor 10.255.255.3 link-layer-address 33.33.33.33 state reachable
neighbor 10.255.255.1 link-layer-address 11.11.11.11 state reachable
As a consequence, a NHRP route towards the protocol address of the hub is installed on each spoke. This way, each spoke can establish a BGP peering session and learn the routes to the hub:
spoke1> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 11.11.11.1, wan, 00:01:59
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), 00:01:58
* via 10.255.255.4, gre1 onlink, 00:01:58
C>* 10.255.255.1/32 is directly connected, gre1, 00:01:59
N>* 10.255.255.4/32 [10/0] is directly connected, gre1, 00:01:59
C>* 11.11.11.0/24 is directly connected, wan, 00:01:59
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), 00:01:58
* via 10.255.255.4, gre1 onlink, 00:01:58
C>* 192.168.1.0/24 is directly connected, lan, 00:01:59
spoke2> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 22.22.22.1, wan, 00:03:28
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), 00:03:27
* via 10.255.255.4, gre2 onlink, 00:03:27
C>* 10.255.255.2/32 is directly connected, gre2, 00:03:28
N>* 10.255.255.4/32 [10/0] is directly connected, gre2, 00:03:28
C>* 22.22.22.0/24 is directly connected, wan, 00:03:28
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), 00:03:27
* via 10.255.255.4, gre2 onlink, 00:03:27
C>* 192.168.2.0/24 is directly connected, lan, 00:03:28
spoke3> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 33.33.33.1, wan, 00:04:25
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), 00:04:24
* via 10.255.255.4, gre3 onlink, 00:04:24
C>* 10.255.255.3/32 is directly connected, gre3, 00:04:25
N>* 10.255.255.4/32 [10/0] is directly connected, gre3, 00:04:25
C>* 33.33.33.0/24 is directly connected, wan, 00:04:25
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), 00:04:24
* via 10.255.255.4, gre3 onlink, 00:04:24
C>* 192.168.3.0/24 is directly connected, lan, 00:04:25
The hub installs a NHRP route for each spoke and learns the remote private networks through BGP:
hub> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 44.44.44.1, wan, 00:06:28
N>* 10.255.255.1/32 [10/0] is directly connected, gre4, 00:06:27
N>* 10.255.255.2/32 [10/0] is directly connected, gre4, 00:06:27
N>* 10.255.255.3/32 [10/0] is directly connected, gre4, 00:06:27
C>* 10.255.255.4/32 is directly connected, gre4, 00:06:28
C>* 44.44.44.0/24 is directly connected, wan, 00:06:28
B> 192.168.1.0/24 [20/0] via 10.255.255.1 (recursive), 00:06:26
* via 10.255.255.1, gre4 onlink, 00:06:26
B> 192.168.2.0/24 [20/0] via 10.255.255.2 (recursive), 00:06:26
* via 10.255.255.2, gre4 onlink, 00:06:26
B> 192.168.3.0/24 [20/0] via 10.255.255.3 (recursive), 00:06:26
* via 10.255.255.3, gre4 onlink, 00:06:26
C>* 192.168.4.0/24 is directly connected, lan, 00:06:28
As you can see, the remote networks are learnt via BGP server from hub, and NHRP routes are set up with a higher priority (10) compared with BGP (20).
All traffic goes through the hub device. This is also the main drawback of this configuration, since it is not possible to establish spoke-to-spoke traffic.
Direct spoke-to-spoke communication¶
This enhancement enables direct spoke-to-spoke traffic, without routing through the hub. It uses the NHRP redirect feature and must be applied on top of a Dynamic configuration.
This feature, once enabled on the hub, detects a traffic flow that enters and
exits the hub through the same GRE interface. This traffic flow can be
redirected as the hub is only routing the flow between two spokes. Then the
NHRP daemon of the hub issues a NHRP traffic indication message containing the
detected packet to the emitter. A spoke receiving such a message performs a
NHRP resolution request to find the NBMA address of the destination and then
continues the communication directly with the destination spoke.
Note
The traffic flow detection mechanism is implemented by the fast path. Therefore, the fast path must be enabled on the hub to benefit from the spoke-to-spoke direct communication.
We first enable the redirect feature on the hub:
hub> edit running
hub running config# / vrf main routing interface gre4 ip nhrp redirect true
hub running config#! / vrf main routing nhrp hub-mode true
hub running config# commit
This enables the sampling of the overlay traffic of the gre4 interface. Only the
packets that come from this interface and are forwarded through the same
interface are eligible for capture. In order to avoid exhausting the device
CPU, the NHRP daemon applies a default sampling rate of 1 packet out of
400. If the destination address matches one of the registered spokes, the hub
attaches the packet to a NHRP traffic indication packet, and sends it to the
sender spoke.
Then we enable the shortcut feature on the spokes so that these messages get
processed:
spoke1> edit running
spoke1 running config# vrf main routing interface gre1 ip nhrp shortcut true
spoke1 running config# commit
spoke2> edit running
spoke2 running config# vrf main routing interface gre2 ip nhrp shortcut true
spoke2 running config# commit
spoke3> edit running
spoke3 running config# vrf main routing interface gre3 ip nhrp shortcut true
spoke3 running config# commit
As the message contains the original destination IP, upon reception, the spoke sends a NHRP resolution request, routed by the hub and replied by the destination spoke. Then both spokes continue the communication bypassing the hub.
On direct spoke to spoke communication, one can distinguish two kinds of traffic. Each traffic will result in NHRP messages exchange involving spokes and hub, as depicted above.
The
protocol addresstoprotocol addresscommunication between spokes. This is done by issuing traffic between those IP addresses. It will result in the creation of a dynamic cache entry.The traffic between private networks located behind GRE interfaces. In addition to the creation of the above mentioned dynamic cache entry, it will result in the creation of a shortcut entry.
Dynamic cache entry¶
Let’s perform traffic between protocol address of each spoke. To achieve this
communication, NHRP protocol will create a dynamic cache entry identifying the
two spokes that want to communicate together. From dataplane perspective, a
NHRP connected route will be setup on the GRE interface, after the mutual
resolution. Example is given below by issuing following cmd ping command:
spoke1 running config# cmd ping 10.255.255.2 source 10.255.255.1 vrf main rate 100
The cache entries will look like below:
spoke1> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre1 nhs 10.255.255.4 44.44.44.44 T
gre1 dynamic 10.255.255.2 22.22.22.22 T
gre1 local 10.255.255.1 - -
spoke2> show nhrp cache
Iface Type Protocol NBMA Flags Identity
gre2 nhs 10.255.255.4 44.44.44.44 T
gre2 local 10.255.255.2 - -
gre2 dynamic 10.255.255.1 44.44.44.44 T
It is worth to be noted that in the last spoke2 dump, NBMA address can be
the hub NBMA address, because the resolution request coming from spoke1
has been forwarded by hub, and no direct resolution request has been received
by spoke2 from spoke1. To be sure that the association between spoke1
protocol address and its NBMA address has been correctly done, the attribute
NBMA-NAT-OA-Address in show nhrp command indicates the real NBMA address
used.
spoke2> show nhrp
[..]
Type: dynamic
Protocol-Address: 10.255.255.1/32
NBMA-Address: 44.44.44.44
NBMA-NAT-OA-Address: 11.11.11.11
We can see that there is now an additional NHRP connection from spoke1 to
spoke2:
hub> show nhrp-connection
Src Dst Flags SAs Identity
44.44.44.44 22.22.22.22 n 0
44.44.44.44 33.33.33.33 n 0
44.44.44.44 11.11.11.11 n 0
spoke1> show nhrp-connection
Src Dst Flags SAs Identity
11.11.11.11 22.22.22.22 n 0
11.11.11.11 44.44.44.44 n 0
Like it has been noted for spoke2 with show nhrp cache dump, the remote entry
to 11.11.11.11 may not be present, while no direct resolution has been identified
by spoke2.
spoke2> show nhrp-connection
Src Dst Flags SAs Identity
22.22.22.22 44.44.44.44 n 0
22.22.22.22 11.11.11.11 n 0
As indicated below, a new NHRP connected route entry has been set up. Note that up to
now, only traffic between Protocol Address of spokes has been initiated; and that
traffic between private networks (192.168.x.0/24) still uses the hub.
spoke1> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 11.11.11.1, wan, 00:45:01
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), 00:45:00
* via 10.255.255.4, gre1 onlink, 00:45:00
C>* 10.255.255.1/32 is directly connected, gre1, 00:45:01
N>* 10.255.255.2/32 [10/0] is directly connected, gre1, 00:12:14
N>* 10.255.255.4/32 [10/0] is directly connected, gre1, 00:45:01
C>* 11.11.11.0/24 is directly connected, wan, 00:45:01
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), 00:45:00
* via 10.255.255.4, gre1 onlink, 00:45:00
C>* 192.168.1.0/24 is directly connected, lan, 00:45:01
spoke2> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 22.22.22.1, wan, 00:45:38
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), 00:45:37
* via 10.255.255.4, gre2 onlink, 00:45:37
N>* 10.255.255.1/32 [10/0] is directly connected, gre2, 00:12:51
C>* 10.255.255.2/32 is directly connected, gre2, 00:45:38
N>* 10.255.255.4/32 [10/0] is directly connected, gre2, 00:45:38
C>* 22.22.22.0/24 is directly connected, wan, 00:45:38
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), 00:45:37
* via 10.255.255.4, gre2 onlink, 00:45:37
C>* 192.168.2.0/24 is directly connected, lan, 00:45:38
Shortcut entry¶
Now, let us perform a cmd ping command between private networks. This command will
result in the capture of this traffic on the hub and will lead to the creation of
a shortcut entry on spokes.
spoke1 running config# cmd ping 192.168.2.1 source 192.168.1.1 vrf main rate 100
The NHRP entries help in creating route entries linked to private networks behind
each GRE device. For instance, sub-network 192.168.2.0/24 is discovered by NHRP,
on spoke1, when traffic is issued from spoke1 to that destination network. On hub,
the traffic is intercepted and sent to spoke2 for NHRP resolution. spoke2 parses
the available networks hidden behind GRE interface, and if present, transmits a
resolution response to spoke1 which creates a nexthop route entry linked to spoke2
dynamic entry, as follows:
spoke1> show nhrp shortcut
Type Prefix Via Identity
dynamic 192.168.2.0/24 10.255.255.2
spoke1> show ipv4-routes
[..]
S>* 0.0.0.0/0 [1/0] via 11.11.11.1, wan, 00:47:58
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), 00:47:57
* via 10.255.255.4, gre1 onlink, 00:47:57
C>* 10.255.255.1/32 is directly connected, gre1, 00:47:58
N>* 10.255.255.2/32 [10/0] is directly connected, gre1, 00:00:48
N>* 10.255.255.4/32 [10/0] is directly connected, gre1, 00:47:58
C>* 11.11.11.0/24 is directly connected, wan, 00:47:58
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), 00:47:57
* via 10.255.255.4, gre1 onlink, 00:47:57
C>* 192.168.1.0/24 is directly connected, lan, 00:47:58
N>* 192.168.2.0/24 [10/0] via 10.255.255.2, gre1 onlink, 00:00:34
spoke2> show nhrp shortcut
Type Prefix Via Identity
dynamic 192.168.1.0/24 10.255.255.1
spoke2> show ipv4-routes
[..]
VRF nhrp:
S>* 0.0.0.0/0 [1/0] via 22.22.22.1, wan, 00:49:39
B> 10.255.255.0/24 [20/0] via 10.255.255.4 (recursive), 00:49:38
* via 10.255.255.4, gre2 onlink, 00:49:38
N>* 10.255.255.1/32 [10/0] is directly connected, gre2, 00:02:15
C>* 10.255.255.2/32 is directly connected, gre2, 00:49:39
N>* 10.255.255.4/32 [10/0] is directly connected, gre2, 00:49:39
C>* 22.22.22.0/24 is directly connected, wan, 00:49:39
B> 192.168.0.0/16 [20/0] via 10.255.255.4 (recursive), 00:49:38
* via 10.255.255.4, gre2 onlink, 00:49:38
N>* 192.168.1.0/24 [10/0] via 10.255.255.1, gre2 onlink, 00:02:34
C>* 192.168.2.0/24 is directly connected, lan, 00:49:39
As can be seen, a NHRP onlink route has been created, and reflects the NHRP shortcut entry created. This entry relies on the NHRP dynamic cache entry previously created when initiating traffic between Protocol Address of spokes.
Note
To bring more clarity, the chapter presented how the cache entry is created, then how the shortcut entry is created, according to each incoming traffic. This said, in reality, launching private network traffic will result in the creation of the two NHRP routes necessary to make spoke-to-spoke communication.
Nexthop route to 192.168.2.0/24 is introduced, and partially overrides
192.168.0.0/16 defined by BGP. As illustrated, the routing decision in NHRP is
made thanks to routing information the protocol has when receiving a resolution
request. The traffic indication was about a specific IP whereas the resulting route
reflects the route of the remote spoke. In our case, the network 192.168.2.0/24
network information is sent from spoke2 to spoke1. All routing information
contained in the vrf where GRE interface sits can be used. A design recommendation
when setting up NHRP network would be to isolate GRE inner network in a specific
VRF.