BGP for L3VPN¶
BGP routing protocol is very rich, and permits exchanging more complex information. With the increasing usage of overlay information (widely used in data centers, but also deployed in ISPs), BGP evolves and is able to to carry tunneling information through L3VPN. L3VPN stands for the ability to encapsulate IP information, into an other payload. Here, the underlay is an IP packet, and the encapsulation used is MPLS. The overlay IP information is the original IP packet with its payload, coming from one of the mutiple virtual private networks.
L3VPN overview¶
L3VPN helps in interconnecting VPNs between several sites, by using a single BGP connection, and at the same time keeping isolation between the different VPNs. L3VPN is based on MPLS technology, and separates the various VPNs connections with MPLS labels handled by BGP. Adding to this, L3VPN offers powerful tools to share (or not) traffic between VPNs by introducing some new RT concepts (see definition below): those tools permit to implement what we call vrf route leaking entries. Those routes can also be used without MPLS for defining local importation and exportation policies between VRs. More generally, those concepts apply to EVPN as per EVPN route target configuration use case.
BGP l3vpn use case example¶
Above drawing illustrates a setup made up of 2 symmetrical sites. Each site is separated with a PE device. In the case the site is a data center, the PE could be replaced with a DC-GW. To simplify, each site is made up of 2 distinct VPNs. L3VPN functionality helps to exchange information about the 2 VPNs, between the 2 sites.
This functionality will subsequently enable data path forwarding. IP Traffic between the 2 PEs will be encapsulated into an MPLS label. On each site, traffic between each CE and the PE is standard IP over ethernet traffic.
From the drawing, the following use cases will be more in detail leveraged successively in that document:
how to interconnect traffic from different VPNs on a same site. This is a specific case from route-leaking. The basic L3VPN commands will be illustrated, as well as the commmands to create Cross-VRF interfaces across VRF. Note that this kind of method is recommended only if the user has interest in interconnecting also VPNs traffic between sites. If this is not the case, an other method to establish cross-VRF routes is recommended at following link: BGP use case for cross-vrf routes.
how to interconnect traffic from a same VPN between sites. This use case will generalise usage of Cross-VRF interfaces with default VRF, as well as explaining how to configure backbone so as to carry MPLS labels.
how to interconnect traffic from different VPNs between sites This use case will generalise the L3VPN use case in an MPLS based framework.
L3VPN terminology¶
It is important to understand some L3VPN terminology. In this paragraph we will give the most important concepts.
- VPN
This acronym refers here to a routing entity, also called VRF Creating a VPN consists in creating a BGP instance in a separated VRF. More information in BGP VRF.
- Route Distinguisher, RD :
This attribute is specific for each VPN. This information is exported along with the L3VPN information of the BGP information
- L3VPN
This refers to creating an overlay with IP packets. In this chapter, L3VPN refers to encapsulating IP traffic over MPLS traffic.
- VPNv4, VPNv6
This refers to RFC 4364: that defines how BGP implements L3VPN with MPLS
- Route Target, RT:
RT and RD share the same format. A VPN can have 2 list of RTs. One is dedicated for import. This will help BGP to import incoming routing entries that come from a remote BGP entity. There is also a list for export, that is sent to remote BGP entities. Route Target is the key element for sharing information across VPNs.
- VRF route leaking:
This refers to the ability to share a route from a VPN to an other entity. See chapter BGP VRF route leak. This ability requires that the VPN that shares a common route, do not have overlapping with the IP provisioning of the networks.
Configuring locally route leaking between VPN¶
BGP configuration¶
Below configuration illustrates a setup made up with 2 VPNs. As can be seen, there is a BGP instance in each of the 2 VR instances, plus the BGP core instance. The 2 instances are configured so as to create leaking between both VPNs. Actually configuration shows the following:
the RD is different, indicating that there are 2 distinct VPNs.
the RT export settings of each VPN is being matched by the RT import settings of the other VPN. This configuration means that route leaking will occur. Practically, export RT of
customer1is11:22, and import RT ofcustomer2is11:22and22:44.11:22value matches, this means thatcustomer2will import information coming fromcustomer1.
vrf main
routing bgp
as 65500
..
..
..
vrf customer1
routing bgp
as 65500
address-family ipv4-unicast
network 192.168.3.0/24
..
l3vpn export route-distinguisher 1:55
l3vpn export route-target 11:22
l3vpn export vpn true
l3vpn import route-target 11:22 route-target 22:44
l3vpn import vpn true
..
..
..
..
..
vrf customer2
routing bgp
as 65500
address-family ipv4-unicast
network 192.168.2.0/24
..
l3vpn export route-distinguisher 2:55
l3vpn export route-target 22:44
l3vpn export vpn true
l3vpn import route-target 11:22 route-target 22:44
l3vpn import vpn true
..
..
..
..
..
When using above configuration, it is mandatory to create BGP core instance. Also, despite RD and RT values could have been the same, it has been deliberately chosen to have distinct values to better understand the mechanisms put in place when dealing with L3VPN importation and exportation. Also, it is common, when a L3VPN setup is put in place between 2 ISPs, that the RD is self to each operator, while the RT will be chosen accordingly by operators.
Note
Above configuration details how routing information is exchanged, but does not explain in detail how data traffic is sent through. The product design choices opted for strongly isolating traffic across VPNs. To pass traffic across VPNs, it is required to create special interfaces by configuration. Those interfaces will make the connection between VPNs. More information will be given about how to create those interfaces in a separate chapter. Next chapters assume the user is familiar with interfaces used for crossing vrfs.
using IPless Virtual Ethernet interfaces¶
Using Cross-VRF interfaces to perform vrf route leaking with BGP requires a
specific semantic between VRs and interface names. VR naming must meet
the requirements of interface naming. Actually, the Cross-VRF interface name
chosen must be equal to the target VR the interface is connected to. To
illustrate, in order to reach VR foo from VR bar, an Cross-VRF interface
named foo has to be created in VR bar. Reversely, an Cross-VRF interface
named bar has to be created in VR foo. In this way, the interface foo
and the interface bar will be connected together. The naming convention
is not only done to reflect the intent of the interface. It is mandatory
to configure it in this way, if one wants to benefit from route leaking
across VRs, using Cross-VRF interfaces, and BGP.
From the user point of view, if a packet is emitted in one interface of a source VR, the same packet will be received in the associated veth interface of destination VR. More information about Cross-VRF interfaces can be found in XVRF Interface types. In order to have the setup working fine, the below configuration has to be appended to the former configuration of previous chapter (BGP VRF leak).
vrf customer1
interface xvrf customer2
link-interface customer1
link-vrf customer2
..
..
..
vrf customer2
interface xvrf customer1
link-interface customer2
link-vrf customer1
..
..
..
With the above configuration applied, VR route leaking is possible.
Subsequently, if BGP peering is done between a CE and the BGP instance
of each VR instance, then route importation and exportation occurs. Below
output demonstrates that the routes from customer2 have been imported to
customer1. The VR route leak are visible with the @1< indicating that
the route entry is originated from VR customer2.
rt1> show bgp vrf customer1 ipv4
BGP table version is 20, local router ID is 1.1.1.1, vrf id 2
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i10.101.0.0/24 1.1.1.2 100 0 i
*>i10.101.1.0/24 1.1.1.2 100 0 i
*>i10.101.2.0/24 1.1.1.2 100 0 i
[..]
*> 10.201.0.0/24 2.2.2.3@1< 100 0 i
*> 10.201.1.0/24 2.2.2.3@1< 100 0 i
*> 10.201.2.0/24 2.2.2.3@1< 100 0 i
[..]
Routing output¶
Once those entries selected in the bgp RIB, nothing prevents the
installation of those VRs routes in the underlying system. Packets going
from a VR to an other VR are using the Cross-VRF interface created. There
is no specific encapsulation, only a route using an interface as gateway.
From above example, the following output displays the routing entries
available in VRF routing table. As can be seen, the route to reach the
remote prefix first reaches the customer2 interface (first line directly
connected, customer2). Then, once the other VR reached, the original
BGP route of customer2 routes the traffic to the correct destination
(via 2.2.2.3 lines). The latter entry is only here for informational
purpose. To get information about routes in VR customer2, use the
associated show command to dump route entris in the associates VR.
rt1> show ipv4-routes vrf customer1
BGP table version is 20, local router ID is 1.1.1.1, vrf id 2
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
VRF customer1:
B>* 10.101.0.0/24 [200/0] via 1.1.1.2, eth1_0, 00:05:06
B>* 10.101.1.0/24 [200/0] via 1.1.1.2, eth1_0, 00:05:06
B>* 10.101.2.0/24 [200/0] via 1.1.1.2, eth1_0, 00:05:06
[..]
B>* 10.201.0.0/24 [200/0] is directly connected, customer2, 00:05:06
* via 2.2.2.3, eth2_0(vrf customer2), 00:05:06
B>* 10.201.1.0/24 [200/0] is directly connected, customer2, 00:05:06
* via 2.2.2.3, eth2_0(vrf customer2), 00:05:06
B>* 10.201.2.0/24 [200/0] is directly connected, customer2, 00:05:06
* via 2.2.2.3, eth2_0(vrf customer2), 00:05:06
[..]
how to interconnect traffic from a same VPN between sites.¶
Being able to interconnect between sites using L3VPN technology is now
possible. As explained in the introduction of that chapter, the traffic between
sites is encapsulated into an MPLS label, negotiated thanks to BGP. More
exactly, the ipv4-vpn address-family configured in BGP will obtain from
remote the label, and the underlay nexthop to use, to reach the remote. That
label will be the encapsulation between 2 VPNs, in one direction. The same
mechanism will apply in reverse direction. Adding to this, this label will be
conveyed by an other framework. Currently, LDP offers that framework by
conveying inner BGP labels in an outer MPLS label negotiated by LDP.
More information on chapter (LDP configuration).
In order to activate L3VPN, use the following command under the main BGP
core instance. L3VPN address-family must be configured on the same VRF
where the LDP configuration is. Usually, the backbone is the main VR
instance.
vrf main
routing bgp
as 65500
neighbor 9.9.9.9 remote-as 65512
neighbor 9.9.9.9 update-source 3.3.3.3
neighbor 9.9.9.9 address-family ipv4-vpn enabled true
..
..
..
Consequently, having an L3VPN peering will trigger importation of L3VPN
entries. For instance, the presence of remote VPNs and associated prefixes
from peer 9.9.9.9 will trigger prefixes importation in the relevant VRs.
Those prefixes, if the remote VPNs match the local VPNs will be imported
in the associated VR. So as to permit that importation, the associated Cross-VRF
interfaces will be created between the main VR and the relevant VRs.
The following configuration illustrates the Cross-VRF interfaces.
vrf main
interface xvrf customer1
link-interface main
link-vrf customer1
..
..
interface xvrf customer2
link-interface main
link-vrf customer2
..
..
..
vrf customer1
interface xvrf main
link-interface customer1
link-vrf main
..
..
..
vrf customer2
interface xvrf main
link-interface customer2
link-vrf main
..
..
..
Also, in order for BGP to be able to export its own labels, BGP must be
configured so as to rely on its own labels, either automatically, or by
choosing its own. Below configuration illustrates the automatic label chosen
by VR customer1, while customer2 chooses to hardset its exportation
label to 300.
vrf customer1
routing bgp
as 65500
address-family ipv4-unicast
network 192.168.2.0/24
..
l3vpn export label auto
..
..
..
..
..
vrf customer2
routing bgp
as 65500
address-family ipv4-unicast
network 192.168.2.0/24
..
l3vpn export label 300
..
..
..
..
..
L3VPN setup using MPLS based framework¶
Above diagram illustrates a topology made up of an MPLS based backbone,
with LSR devices : rt3 and rt4, and LER devices. Each LER device
has a BGP core instance that has ipv4-vpn address-family enabled. Next
to each BGP instance, a BGP instance is created in each VR. Each VR
stands for a private network, either A or B. The color semantic explains
the relationship between the private networks on the various LER devices.
As can be seen with the arrows, each private network is geographically separate, but thanks to L3VPN, the private networks act as if there were only 2 specific private networks. The BGP configuration is depicted below. The LDP and OSPF configuration is out of scope of this chapter.
rt1
vrf main
routing bgp
router-id 5.5.5.5
as 65500
neighbor 9.9.9.9 remote-as 65500
neighbor 15.15.15.15 remote-as 65500
neighbor 9.9.9.9 update-source 5.5.5.5
neighbor 15.15.15.15 update-source 5.5.5.5
neighbor 9.9.9.9 address-family ipv4-vpn enabled true
neighbor 15.15.15.15 address-family ipv4-vpn enabled true
..
..
..
vrf customer1
routing bgp
router-id 1.1.1.1
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 1:55
l3vpn export route-target 1:55
l3vpn export vpn true
l3vpn import route-target 1:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
..
vrf customer2
routing bgp
router-id 2.2.2.2
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 2:55
l3vpn export route-target 2:55
l3vpn export vpn true
l3vpn import route-target 2:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
..
rt2
vrf main
routing bgp
router-id 9.9.9.9
as 65500
neighbor 5.5.5.5 remote-as 65500
neighbor 15.15.15.15 remote-as 65500
neighbor 5.5.5.5 update-source 9.9.9.9
neighbor 15.15.15.15 update-source 9.9.9.9
neighbor 5.5.5.5 address-family ipv4-vpn enabled true
neighbor 15.15.15.15 address-family ipv4-vpn enabled true
..
..
..
vrf customer1
routing bgp
router-id 1.1.1.10
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 1:55
l3vpn export route-target 1:55
l3vpn export vpn true
l3vpn import route-target 1:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
..
vrf customer2
routing bgp
router-id 2.2.2.2
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 2:55
l3vpn export route-target 2:55
l3vpn export vpn true
l3vpn import route-target 2:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
..
rt5
vrf main
routing bgp
router-id 15.15.15.15
as 65500
neighbor 5.5.5.5 remote-as 65500
neighbor 9.9.9.9 remote-as 65500
neighbor 5.5.5.5 update-source 15.15.15.15
neighbor 9.9.9.9 update-source 15.15.15.15
neighbor 5.5.5.5 address-family ipv4-vpn enabled true
neighbor 9.9.9.9 address-family ipv4-vpn enabled true
..
..
..
vrf customer1
routing bgp
router-id 1.1.1.20
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 1:55
l3vpn export route-target 1:55
l3vpn export vpn true
l3vpn import route-target 1:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
..
vrf customer2
routing bgp
router-id 2.2.2.20
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 2:55
l3vpn export route-target 2:55
l3vpn export vpn true
l3vpn import route-target 2:55
l3vpn import vpn true
l3vpn export label 300
..
..
..
..
..
As can be seen, the network prefixes learnt by each BGP instance are
either learnt, by adding extra CE configuration linked to each instance, or
imported thanks to L3VPN technology. Below show dumps the vpnv4 entries:
rt1> show bgp ipv4 vpn
BGP table version is 27, local router ID is 5.5.5.5, vrf id 0
Default local pref 100, local AS 65500
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:55
*> 10.101.0.0/24 1.1.1.2@2< 100 0 i
*> 10.101.1.0/24 1.1.1.2@2< 100 0 i
*> 10.101.2.0/24 1.1.1.2@2< 100 0 i
[..]
*>i10.101.11.0/24 9.9.9.9 100 0 i
*>i10.101.12.0/24 9.9.9.9 100 0 i
*>i10.101.13.0/24 9.9.9.9 100 0 i
*>i10.101.14.0/24 9.9.9.9 100 0 i
[..]
Route Distinguisher: 2:55
*> 10.201.0.0/24 2.2.2.3@1< 100 0 i
*> 10.201.1.0/24 2.2.2.3@1< 100 0 i
*> 10.201.2.0/24 2.2.2.3@1< 100 0 i
*> 10.201.3.0/24 2.2.2.3@1< 100 0 i
[..]
*>i10.201.11.0/24 9.9.9.9 100 0 i
*>i10.201.12.0/24 9.9.9.9 100 0 i
*>i10.201.13.0/24 9.9.9.9 100 0 i
*>i10.201.14.0/24 9.9.9.9 100 0 i
[..]
As can be seen, all the entries have an underlay nexthop that stands for the nexthop of the remote BGP global instance. If local entries are imported, that nexthop will be used in the route-leak to send traffic to the remote BGP instance. Along with the nexthop, an MPLS label is be sent via BGP, and used on top of the MPLS backbone. MPLS stacking will be performed. The MPLS backbone will handle the LSP path.
It is worth to be noted too, that the first 3 visible entries are locally exported
entries. The nexthop to use 1.1.1.2 is located in the VR customer1.
The relationship between the VR name and the VR identifier displayed ( the
@2 value) can be done using the following command:
rt1> show bgp vrfs
Type Id routerId #PeersCfg #PeersEstb Name
L3-VNI RouterMAC Interface
DFLT 0 5.5.5.5 1 1 main
0 00:00:00:00:00:00 unknown
VRF 2 1.1.1.1 1 1 customer1
0 00:00:00:00:00:00 unknown
VRF 1 2.2.2.2 1 1 customer2
0 00:00:00:00:00:00 unknown
The label value 80 is the exported value sent to the remote BGP peers.
That value is received by remote BGP speaker 9.9.9.9 for instance, along
with the undelay network:
rt2> show bgp ipv4 vpn
BGP table version is 20, local router ID is 9.9.9.9, vrf id 0
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:55
*>i10.101.0.0/24 5.5.5.5 100 0 i
UN=5.5.5.5 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.1.0/24 5.5.5.5 100 0 i
UN=5.5.5.5 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.2.0/24 5.5.5.5 100 0 i
UN=5.5.5.5 EC{1:55} label=80 type=bgp, subtype=0
[..]
After having checked that L3VPN peering received the correct information,
it is possible to check against available entries in the bgp vrf customer1
instance like follows:
rt1> show bgp vrf customer1 ipv4
BGP table version is 32, local router ID is 1.1.1.1, vrf id 2
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i10.101.0.0/24 1.1.1.2 100 0 i
*>i10.101.1.0/24 1.1.1.2 100 0 i
*>i10.101.2.0/24 1.1.1.2 100 0 i
[..]
*> 10.101.11.0/24 9.9.9.9@0< 100 0 i
*> 10.101.12.0/24 9.9.9.9@0< 100 0 i
*> 10.101.13.0/24 9.9.9.9@0< 100 0 i
[..]
*> 10.101.22.0/24 15.15.15.15@0< 100 0 i
*> 10.101.23.0/24 15.15.15.15@0< 100 0 i
*> 10.101.24.0/24 15.15.15.15@0< 100 0 i
[..]
Like for the previous dump of vpn entries, it is worth to be noted that remote
vpn entries have their underlay nexthop visible, but annotated with @0<
indicating that this is a VR route leak going to the VPN. Only 1.1.1.2 ip
address is a locally reachable IP address. Actually, this is a CE of that
instance.
Routing output¶
Once those entries selected in the bgp RIB, nothing prevents the installation of those VR routes in the underlying system.
As remind, packets going from a VR to a remote VPN are first mpls
encapsulated once. Then, a second encapsulation takes place, as the backbone is
MPLS based. Then MPLS does the job to forward the packet to the nexthop marked
UN. Reversely, because the BGP vrf instance is at the LER place, the incoming
packets are only encapsulated with the negotiated BGP label. So, to go from the
backbone VR to the local VR, the packet is being popped its mpls label.
From above example, the following output can be extracted from the VR routing
table. As illustrated, the installed route entry performs a double MPLS
encapsulation (82/80). The inner value is the negotiated BGP value. The 82
value is an intermediate value that is being swapped by the negotiated MPLS value
on the backbone.
rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR,
> - selected route, * - FIB route
VRF customer1:
B>* 10.101.0.0/24 [200/0] via 1.1.1.2, eth1_0, 00:27:24
B>* 10.101.1.0/24 [200/0] via 1.1.1.2, eth1_0, 00:27:24
B>* 10.101.2.0/24 [200/0] via 1.1.1.2, eth1_0, 00:27:24
[..]
B>* 10.101.11.0/24 [200/0] is directly connected, main, label 82/80, 00:26:43
via 9.9.9.9(vrf main) (recursive), label 80, 00:26:43
* via 6.6.6.3, eth0_0(vrf main), label 17, 00:26:43
B>* 10.101.12.0/24 [200/0] is directly connected, main, label 82/80, 00:26:43
via 9.9.9.9(vrf main) (recursive), label 80, 00:26:43
* via 6.6.6.3, eth0_0(vrf main), label 17, 00:26:43
B>* 10.101.13.0/24 [200/0] is directly connected, main, label 82/80, 00:26:43
via 9.9.9.9(vrf main) (recursive), label 80, 00:26:43
* via 6.6.6.3, eth0_0(vrf main), label 17, 00:26:43
[..]
B>* 10.101.22.0/24 [200/0] is directly connected, main, label 84/300, 00:26:40
via 15.15.15.15(vrf main) (recursive), label 300, 00:26:40
* via 6.6.6.3, eth0_0(vrf main), label 22, 00:26:40
B>* 10.101.23.0/24 [200/0] is directly connected, main, label 84/300, 00:26:40
via 15.15.15.15(vrf main) (recursive), label 300, 00:26:40
* via 6.6.6.3, eth0_0(vrf main), label 22, 00:26:40
B>* 10.101.24.0/24 [200/0] is directly connected, main, label 84/300, 00:26:40
via 15.15.15.15(vrf main) (recursive), label 300, 00:26:40
* via 6.6.6.3, eth0_0(vrf main), label 22, 00:26:40
[..]
The next command dumps the LFIB of the backbone. As you can see, the 82 value
is replaced by the 17 value, and forwarded thanks to calculated LSP. Actually,
this entry stands for the path to rt2. For reverse direction, incoming packets
are being popped and redirected to the Cross-VRF interface leading to the VRF ( here
customer1).
rt1> show mpls table
Inbound Outbound
Label Type Nexthop Label
-------- ------- --------------- --------
[..]
80 BGP customer1
81 BGP customer2
82 BGP 6.6.6.3 17
83 BGP 6.6.6.3 17
84 BGP 6.6.6.3 22
how to interconnect traffic from different VPNs between sites¶
By integrating examples of the 2 previous chapters, it becomes possible
to perform interconnection between various VPNs between sites. Also,
some VR route leaking across VPNs is done. Note that like for previous
chapters, configuring BGP only is not enough, since conveying MPLS labels
from BGP needs an other MPLS framework to be configured. LDP protocol
is the framework that should be used. More information on chapter (:ref:
LDP configuration <mpls-ldp-configuration>). This being said, next chapter
explores how to simplify, eventually remove the usage of LDP.
L3VPN setup using MPLS based framework with VR leaking.¶
The same topology as for previous chapter is chosen. Black arrows indicate
the traffic that will be authorised to pass between different VPNs. For
instance, network A and network B can access each other. Note that not
all data flow are illustrated in the drawing, but it is also possible to
do VR route leak between network A from rt1 and network B from rt1.
The L3VPN importation and exportation rules for a defined VPN apply for
that VPN, whatever its location is, ie local or remote.
Below configurations are BGP configuration changes. Usually, that kind of
configuration can be used, when some resources are shared with other VRs:
A video server located on that shared resource, or a management vr that is
only able to access to th other VRs.
rt1
vrf main
routing bgp
router-id 5.5.5.5
as 65500
neighbor 9.9.9.9 remote-as 65500
neighbor 15.15.15.15 remote-as 65500
neighbor 9.9.9.9 update-source 5.5.5.5
neighbor 15.15.15.15 update-source 5.5.5.5
neighbor 9.9.9.9 address-family ipv4-vpn enabled true
neighbor 15.15.15.15 address-family ipv4-vpn enabled true
..
..
..
vrf customer1
routing bgp
router-id 1.1.1.1
as 65500
address-family ipv4-unicast
maximum-path ebgp 4
maximum-path ibgp 4
l3vpn export route-distinguisher 1:55
l3vpn export route-target 1:55
l3vpn export vpn true
l3vpn import route-target 1:55 route-target 2:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
..
vrf customer2
routing bgp
router-id 2.2.2.2
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 2:55
l3vpn export route-target 2:55
l3vpn export vpn true
l3vpn import route-target 1:55 route-target 2:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
rt2
vrf main
routing bgp
router-id 9.9.9.9
as 65500
neighbor 5.5.5.5 remote-as 65500
neighbor 15.15.15.15 remote-as 65500
neighbor 5.5.5.5 update-source 9.9.9.9
neighbor 15.15.15.15 update-source 9.9.9.9
neighbor 5.5.5.5 address-family ipv4-vpn enabled true
neighbor 15.15.15.15 address-family ipv4-vpn enabled true
..
..
..
vrf customer1
routing bgp
router-id 1.1.1.10
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 1:55
l3vpn export route-target 1:55
l3vpn export vpn true
l3vpn import route-target 1:55 route-target 2:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
..
vrf customer2
routing bgp
router-id 2.2.2.2
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 2:55
l3vpn export route-target 2:55
l3vpn export vpn true
l3vpn import route-target 1:55 route-target 2:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
..
rt5
vrf main
routing bgp
router-id 15.15.15.15
as 65500
neighbor 5.5.5.5 remote-as 65500
neighbor 9.9.9.9 remote-as 65500
neighbor 5.5.5.5 update-source 15.15.15.15
neighbor 9.9.9.9 update-source 15.15.15.15
neighbor 5.5.5.5 address-family ipv4-vpn enabled true
neighbor 9.9.9.9 address-family ipv4-vpn enabled true
..
..
vrf customer1
routing bgp
router-id 1.1.1.20
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 1:55
l3vpn export route-target 1:55
l3vpn export vpn true
l3vpn import route-target 1:55 route-target 2:55
l3vpn import vpn true
l3vpn export label auto
..
..
..
..
..
vrf customer2
routing bgp
router-id 2.2.2.20
as 65500
address-family ipv4-unicast
l3vpn export route-distinguisher 2:55
l3vpn export route-target 2:55
l3vpn export vpn true
l3vpn import route-target 1:55 route-target 2:55
l3vpn import vpn true
l3vpn export label 300
..
..
..
..
..
The following show extract on rt1 dumps the routing entries of A and B
located on rt2. Two different MPLS labels chosen by BGP of rt2, are
received by rt1, for each network.
rt1> show bgp ipv4 vpn
BGP table version is 20, local router ID is 9.9.9.9, vrf id 0
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 2:55
[..]
*>i10.101.11.0/24 9.9.9.9 100 0 i
UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.12.0/24 9.9.9.9 100 0 i
UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.13.0/24 9.9.9.9 100 0 i
UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.201.11.0/24 9.9.9.9 100 0 i
UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
*>i10.201.12.0/24 9.9.9.9 100 0 i
UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
*>i10.201.13.0/24 9.9.9.9 100 0 i
UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
When applied to the underlying system, the encapsulation for packets
leaving the customer1 VR is different, depending if the target prefix
belongs to A or B. 2 additional temporary labels are allocated and
are swapped as the show mpls labels indicate.
rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR,
> - selected route, * - FIB route
VRF customer1:
B>* 10.101.11.0/24 [200/0] is directly connected, main, label 83/80, 00:00:44
via 9.9.9.9(vrf main) (recursive), label 80, 00:00:44
* via 6.6.6.3, eth0_0(vrf main), label 18, 00:00:44
B>* 10.101.12.0/24 [200/0] is directly connected, main, label 83/80, 00:00:44
via 9.9.9.9(vrf main) (recursive), label 80, 00:00:44
* via 6.6.6.3, eth0_0(vrf main), label 18, 00:00:44
B>* 10.101.13.0/24 [200/0] is directly connected, main, label 83/80, 00:00:44
via 9.9.9.9(vrf main) (recursive), label 80, 00:00:44
* via 6.6.6.3, eth0_0(vrf main), label 18, 00:00:44
B>* 10.201.11.0/24 [200/0] is directly connected, main, label 82/81, 00:08:15
via 9.9.9.9(vrf main) (recursive), label 81, 00:08:15
* via 6.6.6.3, eth0_0(vrf main), label 19, 00:08:15
B>* 10.201.12.0/24 [200/0] is directly connected, main, label 82/81, 00:08:15
via 9.9.9.9(vrf main) (recursive), label 81, 00:08:15
* via 6.6.6.3, eth0_0(vrf main), label 19, 00:08:15
B>* 10.201.13.0/24 [200/0] is directly connected, main, label 82/81, 00:08:15
via 9.9.9.9(vrf main) (recursive), label 81, 00:08:15
* via 6.6.6.3, eth0_0(vrf main), label 19, 00:08:15
rt1> show mpls table
Inbound Outbound
Label Type Nexthop Label
-------- ------- --------------- --------
80 BGP r1-cust1
81 BGP r1-cust2
82 BGP 6.6.6.3 18
83 BGP 6.6.6.3 18
84 BGP 6.6.6.3 18
85 BGP 6.6.6.3 18
An additional specificity of the setup is the possibility to import ECMP
entries coming from 2 separate locations; here, some 32 bit host routes are
retrieved. Some of the entries stand for a server with the same IP, but
geographically at 2 different places, namely rt2 and rt5. This kind of scenario
can be used for servers that require availability, or where load is split in two, to
avoid starvation of the resources of one of the machines.
rt1> show bgp ipv4 vpn
BGP table version is 20, local router ID is 9.9.9.9, vrf id 0
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*=i10.101.18.10/32 15.15.15.15 100 0 i
UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i 9.9.9.9 100 0 i
UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*=i10.101.19.10/32 15.15.15.15 100 0 i
UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i 9.9.9.9 100 0 i
UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*=i10.101.20.10/32 15.15.15.15 100 0 i
UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i 9.9.9.9 100 0 i
rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR,
> - selected route, * - FIB route
VRF customer1:
B>* 10.101.18.10/32 [20/0] is directly connected, main, label 19/80, 00:00:09
via 9.9.9.9(vrf main) (recursive), label 80, 00:00:09
* via 6.6.6.3, eth0_0(vrf main), label 20, 00:00:09
* is directly connected, main, label 22/300, 00:00:09
via 15.15.15.15(vrf main) (recursive), label 300, 00:00:09
* via 6.6.6.3, eth0_0(vrf main), label 21, 00:00:09
B>* 10.101.19.10/32 [20/0] is directly connected, main, label 19/80, 00:00:09
via 9.9.9.9(vrf main) (recursive), label 80, 00:00:09
* via 6.6.6.3, eth0_0(vrf main), label 20, 00:00:09
* is directly connected, main, label 22/300, 00:00:09
via 15.15.15.15(vrf main) (recursive), label 300, 00:00:09
* via 6.6.6.3, eth0_0(vrf main), label 21, 00:00:09
B>* 10.101.20.10/32 [20/0] is directly connected, main, label 19/80, 00:00:09
via 9.9.9.9(vrf main) (recursive), label 80, 00:00:09
* via 6.6.6.3, eth0_0(vrf main), label 20, 00:00:09
* is directly connected, main, label 22/300, 00:00:09
via 15.15.15.15(vrf main) (recursive), label 300, 00:00:09
* via 6.6.6.3, eth0_0(vrf main), label 21, 00:00:09
Interconnecting traffic between direct connections¶
As said in previous interconnection chapters, having an MPLS framework is mandatory to interconnect several VPNs across multiple site. Above examples use LDP. Main reason to use LDP is that MPLS technology must be configured at each hop separating two devices (MPLS is a layer 2 technology). And the border devices should not be aware of the number of hops that separate from remote device. One solution to simplify the configuration and to not rely on external dependencies is to reduce to 1 hop the distance between the two devices that should interconnect. Following examples apply:
device acting as ASBR, and directly connecting to remote ASBR. In that case, eBGP is used.
a GRE tunnel is used to interconnect two devices. As GRE tunnel is a point to point technology, then the traffic flowing across that interface is reduced to 1 hop only, independently of the framework to be crossed.
Using LDP as framework¶
Below configuration illustrates the configuration of the two devices interconnected. The configuration does not include the VPN configuration part.
rt1
vrf main
routing bgp
router-id 5.5.5.5
as 65500
neighbor 9.9.9.9 remote-as 65500
neighbor 9.9.9.9 update-source 5.5.5.5
neighbor 9.9.9.9 address-family ipv4-unicast enabled false
neighbor 9.9.9.9 address-family ipv4-vpn enabled true
..
..
interface loopback loop1
ipv4 address 5.5.5.5/32
..
interface physical eth1
port pci-b0s4
ipv4 address 10.1.2.5/24
..
interface gre gre1
link-interface eth1
ipv4-address 40.0.0.5/24
local 10.1.2.5
remote 10.3.2.9
..
..
routing ospf
router-id 10.1.2.5
network 10.1.2.0/24
..
..
routing static
ipv4-route 9.9.9.9/32 next-hop 40.0.0.9
..
..
routing mpls ldp
router-id 5.5.5.5
address-family ipv4
discovery transport-address 5.5.5.5
interface gre1
..
..
..
..
..
rt2
vrf main
routing bgp
router-id 9.9.9.9
as 65500
neighbor 5.5.5.5 remote-as 65500
neighbor 5.5.5.5 update-source 9.9.9.9
neighbor 5.5.5.5 address-family ipv4-unicast enabled false
neighbor 5.5.5.5 address-family ipv4-vpn enabled true
..
..
interface loopback loop1
ipv4 address 9.9.9.9/32
..
..
interface physical eth1
port pci-b0s4
ipv4 address 10.3.2.9/24
..
..
interface gre gre1
link-interface eth1
ipv4 address 40.0.0.9/24
local 10.3.2.9
remote 10.1.2.5
..
..
routing ospf
router-id 10.3.2.9
network 10.3.2.0/24 area 0
..
..
routing static
ipv4-route 5.5.5.5/32 next-hop 40.0.0.5
..
..
routing mpls ldp
router-id 9.9.9.9
address-family ipv4
discovery transport-address 9.9.9.9
interface gre1
..
..
..
..
..
..
..
As shown below, an implicit-null label has been bound to the route to the
next-hop of remote BGP speaker. You can note that 40.0.0.2 is the gateway
used to reach remote 9.9.9.9.
rt1> show ipv4-routing
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
VRF main:
[..]
C>* 5.5.5.5/32 is directly connected, lo, 00:02:21
S>* 9.9.9.9/32 [1/0] via 40.0.0.2, gre1, label implicit-null, 00:00:19
C>* 40.0.0.0/24 is directly connected, gre1, 00:02:21
[..]
By adding VPN configuration which is not present on above configuration, one
will have two VPN labels negotiated between BGP independently of LDP; 144
is the outgoing label, and 16 is the incoming label.
rt1> show bgp ipv4 vpn
BGP table version is 2, local router ID is 5.5.5.5, vrf id 0
Default local pref 100, local AS 65500
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
[..]
*> 172.16.1.0/24 0.0.0.0@1< 0 32768 ?
*>i172.16.2.0/24 9.9.9.9 0 100 0 ?
The below show ipv4-routes command gives the information on the encapsulation used
by a packet going from local vrf customer1 to remote customer2. Outgoing packets
leaving rt1 device are encapsulated with label 144/implicit-null. You can note
that intermediate step to pass over the vrf border between main and customer1
happens by appending an intermediate label 17 that is popped when directed to
40.0.0.2 destination IP. The additional implicit-null label is chosen since
it stands for the label to use when wanting to reach the nexthop of the VPN route.
Actually, 9.9.9.9 is reachable via 40.0.0.2 by using implicit-null label.
rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
VRF r1-cust1:
C>* 172.16.1.0/24 is directly connected, r1-eth0, 00:11:15
B>* 172.16.2.0/24 [200/0] is directly connected, main, label 17/144, 00:09:07
via 9.9.9.9(vrf main) (recursive), label 144, 00:09:07
* via 40.0.0.2, r1-gre(vrf main), label implicit-null, 00:09:07
Reversely, incoming packets are received with 16/implicit-null label. On rt1,
a switching rule indicates that 16 label is popped, and packet is directed to
xvrf interface customer1.
rt1> show mpls table
Inbound Outbound
Label Type Nexthop Label
-------- ------- --------------- --------
[..]
16 BGP r1-cust1
17 BGP 40.0.0.2 implicit-null
Using Static Label configuration¶
Below configuration illustrates is an alternative to previous configuration.
Without LDP, but by relying on static MPLS configuration, it is possible
to do convey VPN labels transparently across a backbone, inside GRE tunnels.
VPN traffic conveyed will use the static MPLS routes labels. Below static
route used label number 3 that stands for ipv4 implicit-null value.
rt1
vrf main
routing bgp
router-id 5.5.5.5
as 65500
neighbor 9.9.9.9 remote-as 65500
neighbor 9.9.9.9 update-source 5.5.5.5
neighbor 9.9.9.9 address-family ipv4-unicast enabled false
neighbor 9.9.9.9 address-family ipv4-vpn enabled true
..
..
interface loopback loop1
ipv4 address 5.5.5.5/32
..
interface physical eth1
port pci-b0s4
ipv4 address 10.1.2.5/24
..
interface gre gre1
link-interface eth1
ipv4-address 40.0.0.5/24
local 10.1.2.5
remote 10.3.2.9
..
..
routing ospf
router-id 10.1.2.5
network 10.1.2.0/24
..
..
routing static
ipv4-route 9.9.9.9/32 next-hop 40.0.0.9%gre1 label 3
..
..
rt2
vrf main
routing bgp
router-id 9.9.9.9
as 65500
neighbor 5.5.5.5 remote-as 65500
neighbor 5.5.5.5 update-source 9.9.9.9
neighbor 5.5.5.5 address-family ipv4-unicast enabled false
neighbor 5.5.5.5 address-family ipv4-vpn enabled true
..
..
interface loopback loop1
ipv4 address 9.9.9.9/32
..
..
interface physical eth1
port pci-b0s4
ipv4 address 10.3.2.9/24
..
..
interface gre gre1
link-interface eth1
ipv4 address 40.0.0.9/24
local 10.3.2.9
remote 10.1.2.5
..
..
routing ospf
router-id 10.3.2.9
network 10.3.2.0/24 area 0
..
..
routing static
ipv4-route 5.5.5.5/32 next-hop 40.0.0.5%gre1 label 3
..
..
Above configuration results in having a label bound to route to reach BGP
next-hop, that is to say route via 40.0.0.2. In that case, when resolving
route from customer1 VRF, the VPN label would be appended with the
static label. The show commands from above example are the same with or
without LDP.