BGP for L3VPN

BGP routing protocol is very rich, and permits exchanging more complex information. With the increasing usage of overlay information (widely used in data centers, but also deployed in ISPs), BGP evolves and is able to to carry tunneling information through L3VPN.

L3VPN overview

../../../../_images/l3vpn-bgp-config.svg

BGP l3vpn use case example

Above drawing illustrates a setup made up of 2 symmetrical sites. Each site is separated with a PE device. In the case the site is a data center, the PE could be replaced with a DC-GW. To simplify, each site is made up of 2 distinct VPNs. L3VPN functionality helps to exchange information about the 2 VPNs, between the 2 sites.

This functionality will subsequently enable data path forwarding. IP Traffic between the 2 PEs will be encapsulated into an MPLS label. On each site, traffic between each CE and the PE is standard IP over ethernet traffic.

From the drawing, the following use cases will be more in detail leveraged successively in that document:

  • how to interconnect traffic from different VPNs on a same site. This is a specific case from route-leaking. The basic L3VPN commands will be illustrated, as well as the commmands to create Cross-VRF interfaces across VRF.

  • how to interconnect traffic from a same VPN between sites. This use case will generalise usage of Cross-VRF interfaces with default VRF, as well as explaining how to configure backbone so as to carry MPLS labels.

  • how to interconnect traffic from different VPNs between sites This use case will generalise the L3VPN use case in an MPLS based framework.

L3VPN terminology

It is important to understand some L3VPN terminology. In this paragraph we will give the most important concepts.

VPN

This acronym refers here to a routing entity, also called VRF Creating a VPN consists in creating a BGP instance in a separated VRF. More information in BGP VRF.

Route Distinguisher, RD :

This attribute is specific for each VPN. This information is exported along with the L3VPN information of the BGP information

L3VPN

This refers to creating an overlay with IP packets. In this chapter, L3VPN refers to encapsulating IP traffic over MPLS traffic.

VPNv4, VPNv6

This refers to RFC 4364: that defines how BGP implements L3VPN with MPLS

Route Target, RT:

RT and RD share the same format. A VPN can have 2 list of RTs. One is dedicated for import. This will help BGP to import incoming routing entries that come from a remote BGP entity. There is also a list for export, that is sent to remote BGP entities. Route Target is the key element for sharing information across VPNs.

VRF route leaking:

This refers to the ability to share a route from a VPN to an other entity. See chapter BGP VRF rout leak. This ability requires that the VPN that shares a common route, do not have overlapping with the IP provisioning of the networks.

Configuring locally route leaking between VPN

BGP configuration

Below configuration illustrates a setup made up with 2 VPNs. As can be seen, there is a BGP instance in each of the 2 VR instances, plus the BGP core instance. The 2 instances are configured so as to create leaking between both VPNs. Actually configuration shows the following:

  • the RD is different, indicating that there are 2 distinct VPNs.

  • the RT export settings of each VPN is being matched by the RT import settings of the other VPN. This configuration means that route leaking will occur.

vrf main
  routing bgp
    as 65500
    ..
    ..
  ..
vrf customer1
  routing bgp
    as 65500
    address-family ipv4-unicast
      network 192.168.3.0/24
        ..
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 11:22
      l3vpn export vpn true
      l3vpn import route-target 11:22 route-target 22:44
      l3vpn import vpn true
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    as 65500
    address-family ipv4-unicast
      network 192.168.2.0/24
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 11:22
      l3vpn export vpn true
      l3vpn import route-target 11:22 route-target 22:44
      l3vpn import vpn true
      ..
      ..
    ..
    ..
  ..

When using above configuration, it is mandatory to create BGP core instance. Also, despite RD and RT values could have been the same, it has been deliberately chosen to have distinct values to better understand the mechanisms put in place when dealing with L3VPN importation and exportation. Also, it is common, when a L3VPN setup is put in place between 2 ISPs, that the RD is self to each operator, while the RT will be chosen accordingly by operators.

Note

Above configuration details how routing information is exchanged, but does not explain in detail how data traffic is sent through. The product design choices opted for strongly isolating traffic across VPNs. To pass traffic across VPNs, it is required to create special interfaces by configuration. Those interfaces will make the connection between VPNs. More information will be given about how to create those interfaces in a separate chapter. Next chapters assume the user is familiar with interfaces used for crossing vrfs.

using IPless Virtual Ethernet interfaces

Using Cross-VRF interfaces to perform vrf route leaking with BGP requires a specific semantic between VRs and interface names. VR naming must meet the requirements of interface naming. Actually, the Cross-VRF interface name chosen must be equal to the target VR the interface is connected to. To illustrate, in order to reach VR foo from VR bar, an Cross-VRF interface named foo has to be created in VR bar. Reversely, an Cross-VRF interface named bar has to be created in VR foo. In this way, the interface foo and the interface bar will be connected together. The naming convention is not only done to reflect the intent of the interface. It is mandatory to configure it in this way, if one wants to benefit from route leaking across VRs, using Cross-VRF interfaces, and BGP.

From the user point of view, if a packet is emitted in one interface of a source VR, the same packet will be received in the associated veth interface of destination VR. More information about Cross-VRF interfaces can be found in XVRF Interface types. In order to have the setup working fine, the below configuration has to be appended to the former configuration of previous chapter (BGP VRF leak).

vrf customer1
  interface xvrf customer2
    link-interface customer1
    link-vrf customer2
    ..
    ..
  ..
vrf customer2
  interface xvrf customer1
    link-interface customer2
    link-vrf customer1
    ..
    ..
  ..

With the above configuration applied, VR route leaking is possible. Subsequently, if BGP peering is done between a CE and the BGP instance of each VR instance, then route importation and exportation occurs. Below output demonstrates that the routes from customer2 have been imported to customer1. The VR route leak are visible with the @1< indicating that the route entry is originated from VR customer2.

Being able to interconnect between sites using L3VPN technology is now possible. As explained in the introduction of that chapter, the traffic between sites is encapsulated into an MPLS label. More exactly, BGP L3VPN will append a label that will stand for the encapsulation between 2 VPNs, in one direction. Adding to this, this label will be conveyed by an other framework. Currently, LDP offers that framework by conveying inner BGP labels in an outer LDP label. More information on chapter (LDP configuration).

rt1> show bgp vrf customer1 ipv4
BGP table version is 20, local router ID is 1.1.1.1, vrf id 2
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i10.101.0.0/24    1.1.1.2                       100      0 i
*>i10.101.1.0/24    1.1.1.2                       100      0 i
*>i10.101.2.0/24    1.1.1.2                       100      0 i
[..]
*> 10.201.0.0/24    2.2.2.3@1<                    100      0 i
*> 10.201.1.0/24    2.2.2.3@1<                    100      0 i
*> 10.201.2.0/24    2.2.2.3@1<                    100      0 i
[..]

Routing output

Once those entries selected in the bgp RIB, nothing prevents the installation of those VRs routes in the underlying system. Packets going from a VR to an other VR are using the Cross-VRF interface created. There is no specific encapsulation, only a route using an interface as gateway. From above example, the following output displays the routing entries available in VRF routing table. As can be seen, the route to reach the remote prefix first reaches the customer2 interface (first line directly connected, customer2). Then, once the other VR reached, the original BGP route of customer2 routes the traffic to the correct destination (via 2.2.2.3 lines). The latter entry is only here for informational purpose. To get information about routes in VR customer2, use the associated show command to dump route entris in the associates VR.

rt1> show ipv4-routes vrf customer1
BGP table version is 20, local router ID is 1.1.1.1, vrf id 2
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
VRF customer1:
B>* 10.101.0.0/24 [200/0] via 1.1.1.2, eth1_0, 00:05:06
B>* 10.101.1.0/24 [200/0] via 1.1.1.2, eth1_0, 00:05:06
B>* 10.101.2.0/24 [200/0] via 1.1.1.2, eth1_0, 00:05:06
[..]
B>* 10.201.0.0/24 [200/0] is directly connected, customer2, 00:05:06
  *                       via 2.2.2.3, eth2_0(vrf customer2), 00:05:06
B>* 10.201.1.0/24 [200/0] is directly connected, customer2, 00:05:06
  *                       via 2.2.2.3, eth2_0(vrf customer2), 00:05:06
B>* 10.201.2.0/24 [200/0] is directly connected, customer2, 00:05:06
  *                       via 2.2.2.3, eth2_0(vrf customer2), 00:05:06
[..]

how to interconnect traffic from a same VPN between sites.

Being able to interconnect between sites using L3VPN technology is now possible. As explained in the introduction of that chapter, the traffic between sites is encapsulated into an MPLS label, negotiated thanks to BGP. More exactly, the ipv4-vpn address-family configured in BGP will obtain from remote the label, and the underlay nexthop to use, to reach the remote. That label will be the encapsulation between 2 VPNs, in one direction. The same mechanism will apply in reverse direction. Adding to this, this label will be conveyed by an other framework. Currently, LDP offers that framework by conveying inner BGP labels in an outer MPLS label negoatiated by LDP. More information on chapter (LDP configuration).

In order to activate L3VPN, use the following command under the main BGP core instance. L3VPN address-family must be configured on the same VRF where the LDP configuration is. Usually, the backbone is the main VR instance.

vrf main
  routing bgp
    as 65500
    neighbor 9.9.9.9 remote-as 65512
    neighbor 9.9.9.9 update-source 3.3.3.3
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    ..
    ..
  ..

Consequently, having an L3VPN peering will trigger importation of L3VPN entries. For instance, the presence of remote VPNs and associated prefixes from peer 9.9.9.9 will trigger prefixes importation in the relevant VRs. Those prefixes, if the remote VPNs match the local VPNs will be imported in the associated VR. So as to permit that importation, the associated Cross-VRF interfaces will be created between the main VR and the relevant VRs. The following configuration illustrates the Cross-VRF interfaces.

vrf main
  interface xvrf customer1
    link-interface main
    link-vrf customer1
    ..
    ..
  interface xvrf customer2
    link-interface main
    link-vrf customer2
    ..
    ..
  ..
vrf customer1
  interface xvrf main
    link-interface customer1
    link-vrf main
    ..
    ..
  ..
vrf customer2
  interface xvrf main
    link-interface customer2
    link-vrf main
    ..
    ..
  ..

Also, in order for BGP to be able to export its own labels, BGP must be configured so as to rely on its own labels, either automatically, or by choosing its own. Below configuration illustrates the automatic label chosen by VR customer1, while customer2 chooses to hardset its exportation label to 300.

vrf customer1
  routing bgp
    as 65500
    address-family ipv4-unicast
      network 192.168.2.0/24
        ..
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    as 65500
    address-family ipv4-unicast
      network 192.168.2.0/24
        ..
      l3vpn export label 300
      ..
      ..
    ..
    ..
  ..
../../../../_images/bgp-l3vpn-configuration-leak.svg

L3VPN setup using MPLS based framework

Above diagram illustrates a topology made up of an MPLS based backbone, with LSR devices : rt3 and rt4, and LER devices. Each LER device has a BGP core instance that has ipv4-vpn address-family enabled. Next to each BGP instance, a BGP instance is created in each VR. Each VR stands for a private network, either A or B. The color semantic explains the relationship between the private networks on the various LER devices.

As can be seen with the arrows, each private network is geographically separate, but thanks to L3VPN, the private networks act as if there were only 2 specific private networks. The BGP configuration is depicted below. The LDP and OSPF configuration is out of scope of this chapter.

rt1

vrf main
  routing bgp
    router-id 5.5.5.5
    as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 15.15.15.15 remote-as 65500
    neighbor 9.9.9.9 update-source 5.5.5.5
    neighbor 15.15.15.15 update-source 5.5.5.5
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    neighbor 15.15.15.15 address-family ipv4-vpn enabled true
    ..
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.1
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.2
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..

rt2

vrf main
  routing bgp
    router-id 9.9.9.9
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 15.15.15.15 remote-as 65500
    neighbor 5.5.5.5 update-source 9.9.9.9
    neighbor 15.15.15.15 update-source 9.9.9.9
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    neighbor 15.15.15.15 address-family ipv4-vpn enabled true
    ..
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.10
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.2
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..

rt5

vrf main
  routing bgp
    router-id 15.15.15.15
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 5.5.5.5 update-source 15.15.15.15
    neighbor 9.9.9.9 update-source 15.15.15.15
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    ..
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.20
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.20
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 2:55
      l3vpn import vpn true
      l3vpn export label 300
      ..
      ..
    ..
    ..
  ..

As can be seen, the network prefixes learnt by each BGP instance are either learnt, by adding extra CE configuration linked to each instance, or imported thanks to L3VPN technology. Below show dumps the vpnv4 entries:

rt1> show bgp ipv4 vpn
BGP table version is 32, local router ID is 5.5.5.5, vrf id 0
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:55
*> 10.101.0.0/24    1.1.1.2@2<              100      0 i
    UN=1.1.1.2 EC{1:55} label=80 type=bgp, subtype=5
*> 10.101.1.0/24    1.1.1.2@2<              100      0 i
    UN=1.1.1.2 EC{1:55} label=80 type=bgp, subtype=5
*> 10.101.2.0/24    1.1.1.2@2<              100      0 i
    UN=1.1.1.2 EC{1:55} label=80 type=bgp, subtype=5
[..]
*>i10.101.11.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.12.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.13.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
[..]
*>i10.101.22.0/24   15.15.15.15              100      0 i
    UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i10.101.23.0/24   15.15.15.15              100      0 i
    UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i10.101.24.0/24   15.15.15.15              100      0 i
    UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
[..]
Route Distinguisher: 2:55
*> 10.201.0.0/24    2.2.2.3@1<              100      0 i
    UN=2.2.2.3 EC{2:55} label=81 type=bgp, subtype=5
*> 10.201.1.0/24    2.2.2.3@1<              100      0 i
    UN=2.2.2.3 EC{2:55} label=81 type=bgp, subtype=5
*> 10.201.2.0/24    2.2.2.3@1<              100      0 i
    UN=2.2.2.3 EC{2:55} label=81 type=bgp, subtype=5
[..]
*>i10.201.11.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
*>i10.201.12.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
*>i10.201.13.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
[..]

As can be seen, all the entries have an underlay nexthop defined (UN) that usually stands for the nexthop of the remote BGP global instance. If local entries are imported, that nexthop stands will be sent to remote BGP global instance. Along with the nexthop , an MPLS label will be sent via BGP, and used on top of the MPLS backbone. MPLS stacking will be performed. MPLS backbone will handle the LSP path.

It is worth to be noted too, that the first 3 visible entries are locally exported entries. The nexthop to use 1.1.1.2 is located in the VR customer1. The relationship between the VR name and the VR identifier displayed ( the @2 value) can be done using the following command:

rt1> show bgp vrfs
Type  Id     routerId #PeersVfg #PeersEstb Name      L3-VNI Rmac
DFLT  0      5.5.5.5  2       2       main      0      00:00:00:00:00:00
 VRF  2      1.1.1.1  1       1       customer1 0      00:00:00:00:00:00
 VRF  1      2.2.2.2  1       1       customer2 0      00:00:00:00:00:00

The label value 80 is the exported value sent to the remote BGP peers. That value is received by remote BGP speaker 9.9.9.9 for instance, along with the undelay network:

rt2> show bgp ipv4 vpn
BGP table version is 20, local router ID is 9.9.9.9, vrf id 0
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:55
*>i10.101.0.0/24    5.5.5.5              100      0 i
    UN=5.5.5.5 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.1.0/24    5.5.5.5              100      0 i
    UN=5.5.5.5 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.2.0/24    5.5.5.5              100      0 i
    UN=5.5.5.5 EC{1:55} label=80 type=bgp, subtype=0
[..]

After having checked that L3VPN peering received the correct information, it is possible to check against available entries in the bgp vrf customer1 instance like follows:

rt1> show bgp vrf customer1 ipv4
BGP table version is 32, local router ID is 1.1.1.1, vrf id 2
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i10.101.0.0/24    1.1.1.2                       100      0 i
*>i10.101.1.0/24    1.1.1.2                       100      0 i
*>i10.101.2.0/24    1.1.1.2                       100      0 i
[..]
*> 10.101.11.0/24   9.9.9.9@0<                    100      0 i
*> 10.101.12.0/24   9.9.9.9@0<                    100      0 i
*> 10.101.13.0/24   9.9.9.9@0<                    100      0 i
[..]
*> 10.101.22.0/24   15.15.15.15@0<                100      0 i
*> 10.101.23.0/24   15.15.15.15@0<                100      0 i
*> 10.101.24.0/24   15.15.15.15@0<                100      0 i
[..]

Like for the previous dump of vpn entries, it is worth to be noted that remote vpn entries have their underlay nexthop visible, but annotated with @0< indicating that this is a VR route leak going to the VPN. Only 1.1.1.2 ip address is a locally reachable IP address. Actually, this is a CE of that instance.

Routing output

Once those entries selected in the bgp RIB, nothing prevents the installation of those VR routes in the underlying system.

As remind, packets going from a VR to a remote VPN are first mpls encapsulated once. Then, a second encapsulation takes place, as the backbone is MPLS based. Then MPLS does the job to forward the packet to the nexthop marked UN. Reversely, because the BGP vrf instance is at the LER place, the incoming packets are only encapsulated with the negotiated BGP label. So, to go from the backbone VR to the local VR, the packet is being popped its mpls label.

From above example, the following output can be extracted from the VR routing table. As illustrated, the installed route entry performs a double MPLS encapsulation (82/80). The inner value is the negotiated BGP value. The 82 value is an intermediate value that is being swapped by the negotiated MPLS value on the backbone.

rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR,
       > - selected route, * - FIB route

VRF customer1:
B>* 10.101.0.0/24 [200/0] via 1.1.1.2, eth1_0, 00:27:24
B>* 10.101.1.0/24 [200/0] via 1.1.1.2, eth1_0, 00:27:24
B>* 10.101.2.0/24 [200/0] via 1.1.1.2, eth1_0, 00:27:24
[..]
B>* 10.101.11.0/24 [200/0] is directly connected, main, label 82/80, 00:26:43
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:26:43
  *                          via 6.6.6.3, eth0_0(vrf main), label 17, 00:26:43
B>* 10.101.12.0/24 [200/0] is directly connected, main, label 82/80, 00:26:43
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:26:43
  *                          via 6.6.6.3, eth0_0(vrf main), label 17, 00:26:43
B>* 10.101.13.0/24 [200/0] is directly connected, main, label 82/80, 00:26:43
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:26:43
  *                          via 6.6.6.3, eth0_0(vrf main), label 17, 00:26:43
[..]
B>* 10.101.22.0/24 [200/0] is directly connected, main, label 84/300, 00:26:40
                           via 15.15.15.15(vrf main) (recursive), label 300, 00:26:40
  *                          via 6.6.6.3, eth0_0(vrf main), label 22, 00:26:40
B>* 10.101.23.0/24 [200/0] is directly connected, main, label 84/300, 00:26:40
                           via 15.15.15.15(vrf main) (recursive), label 300, 00:26:40
  *                          via 6.6.6.3, eth0_0(vrf main), label 22, 00:26:40
B>* 10.101.24.0/24 [200/0] is directly connected, main, label 84/300, 00:26:40
                           via 15.15.15.15(vrf main) (recursive), label 300, 00:26:40
  *                          via 6.6.6.3, eth0_0(vrf main), label 22, 00:26:40
[..]

The next command dumps the LFIB of the backbone. As you can see, the 82 value is replaced by the 17 value, and forwarded thanks to calculated LSP. Actually, this entry stands for the path to rt2. For reverse direction, incoming packets are being popped and redirected to the Cross-VRF interface leading to the VRF ( here customer1).

rt1> show mpls-table
 Inbound                            Outbound
   Label     Type          Nexthop     Label
--------  -------  ---------------  --------
[..]
      80      BGP         customer1
      81      BGP         customer2
      82      BGP          6.6.6.3        17
      83      BGP          6.6.6.3        17
      84      BGP          6.6.6.3        22

how to interconnect traffic from different VPNs between sites

By integrating examples of the 2 previous chapters, it becomes possible to perform interconnection between various VPNs between sites. Also, some VR route leaking across VPNs is done.

../../../../_images/bgp-l3vpn-configuration-leak.svg

L3VPN setup using MPLS based framework with VR leaking.

The same topology as for previous chapter is chosen. Black arrows indicate the traffic that will be authorised to pass between different VPNs. For instance, network A and network B can access each other. Note that not all data flow are illustrated in the drawing, but it is also possible to do VR route leak between network A from rt1 and network B from rt1. The L3VPN importation and exportation rules for a defined VPN apply for that VPN, whatever its location is, ie local or remote.

Below configurations are BGP configuration changes. Usually, that kind of configuration can be used, when some resources are shared with other VRs: A video server located on that shared resource, or a management vr that is only able to access to th other VRs.

rt1

vrf main
  routing bgp
    router-id 5.5.5.5
    as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 15.15.15.15 remote-as 65500
    neighbor 9.9.9.9 update-source 5.5.5.5
    neighbor 15.15.15.15 update-source 5.5.5.5
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    neighbor 15.15.15.15 address-family ipv4-vpn enabled true
    ..
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.1
    as 65500
    address-family ipv4-unicast
      maximum-path ebgp 4
      maximum-path ibgp 4
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.2
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
  ..

rt2

vrf main
  routing bgp
    router-id 9.9.9.9
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 15.15.15.15 remote-as 65500
    neighbor 5.5.5.5 update-source 9.9.9.9
    neighbor 15.15.15.15 update-source 9.9.9.9
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    neighbor 15.15.15.15 address-family ipv4-vpn enabled true
    ..
  ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.10
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.2
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..

rt5

vrf main
  routing bgp
    router-id 15.15.15.15
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 5.5.5.5 update-source 15.15.15.15
    neighbor 9.9.9.9 update-source 15.15.15.15
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.20
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.20
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label 300
      ..
      ..
    ..
    ..
  ..

The following show extract on rt1 dumps the routing entries of A and B located on rt2. Two different MPLS labels chosen by BGP of rt2, are received by rt1, for each network.

rt1> show bgp ipv4 vpn
BGP table version is 20, local router ID is 9.9.9.9, vrf id 0
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 2:55
[..]
*>i10.101.11.0/24       9.9.9.9          100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.12.0/24       9.9.9.9          100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.13.0/24       9.9.9.9          100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.201.11.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
*>i10.201.12.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
*>i10.201.13.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0

When applied to the underlying system, the encapsulation for packets leaving the customer1 VR is different, depending if the target prefix belongs to A or B. 2 additional temporary labels are allocated and are swapped as the show mpls labels indicate.

rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR,
       > - selected route, * - FIB route

VRF customer1:
B>* 10.101.11.0/24 [200/0] is directly connected, main, label 83/80, 00:00:44
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:00:44
  *                          via 6.6.6.3, eth0_0(vrf main), label 18, 00:00:44
B>* 10.101.12.0/24 [200/0] is directly connected, main, label 83/80, 00:00:44
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:00:44
  *                          via 6.6.6.3, eth0_0(vrf main), label 18, 00:00:44
B>* 10.101.13.0/24 [200/0] is directly connected, main, label 83/80, 00:00:44
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:00:44
  *                          via 6.6.6.3, eth0_0(vrf main), label 18, 00:00:44
B>* 10.201.11.0/24 [200/0] is directly connected, main, label 82/81, 00:08:15
                           via 9.9.9.9(vrf main) (recursive), label 81, 00:08:15
  *                          via 6.6.6.3, eth0_0(vrf main), label 19, 00:08:15
B>* 10.201.12.0/24 [200/0] is directly connected, main, label 82/81, 00:08:15
                           via 9.9.9.9(vrf main) (recursive), label 81, 00:08:15
  *                          via 6.6.6.3, eth0_0(vrf main), label 19, 00:08:15
B>* 10.201.13.0/24 [200/0] is directly connected, main, label 82/81, 00:08:15
                           via 9.9.9.9(vrf main) (recursive), label 81, 00:08:15
  *                          via 6.6.6.3, eth0_0(vrf main), label 19, 00:08:15

rt1> show mpls-table
 Inbound                            Outbound
   Label     Type          Nexthop     Label
--------  -------  ---------------  --------
      80      BGP         r1-cust1
      81      BGP         r1-cust2
      82      BGP          6.6.6.3        18
      83      BGP          6.6.6.3        18
      84      BGP          6.6.6.3        18
      85      BGP          6.6.6.3        18

An additional specificity of the setup is the possibility to import ECMP entries coming from 2 separate locations; here, some 32 bit host routes are retrieved. Some of the entries stand for a server with the same IP, but geographically at 2 different places, namely rt2 and rt5. This kind of scenario can be used for servers that require availability, or where load is split in two, to avoid starvation of the resources of one of the machines.

rt1> show bgp ipv4 vpn
BGP table version is 20, local router ID is 9.9.9.9, vrf id 0
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*=i10.101.18.10/32  15.15.15.15              100      0 i
    UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i                 9.9.9.9              100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*=i10.101.19.10/32  15.15.15.15              100      0 i
    UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i                 9.9.9.9              100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*=i10.101.20.10/32  15.15.15.15              100      0 i
    UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i                 9.9.9.9              100      0 i

rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR,
       > - selected route, * - FIB route

VRF customer1:
B>* 10.101.18.10/32 [20/0] is directly connected, main, label 19/80, 00:00:09
                          via 9.9.9.9(vrf main) (recursive), label 80, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 20, 00:00:09
  *                       is directly connected, main, label 22/300, 00:00:09
                          via 15.15.15.15(vrf main) (recursive), label 300, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 21, 00:00:09
B>* 10.101.19.10/32 [20/0] is directly connected, main, label 19/80, 00:00:09
                          via 9.9.9.9(vrf main) (recursive), label 80, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 20, 00:00:09
  *                       is directly connected, main, label 22/300, 00:00:09
                          via 15.15.15.15(vrf main) (recursive), label 300, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 21, 00:00:09
B>* 10.101.20.10/32 [20/0] is directly connected, main, label 19/80, 00:00:09
                          via 9.9.9.9(vrf main) (recursive), label 80, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 20, 00:00:09
  *                       is directly connected, main, label 22/300, 00:00:09
                          via 15.15.15.15(vrf main) (recursive), label 300, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 21, 00:00:09