BGP for L3VPN

BGP routing protocol is very rich, and permits exchanging more complex information. With the increasing usage of overlay information (widely used in data centers, but also deployed in ISPs), BGP evolves and is able to to carry tunneling information through L3VPN. L3VPN stands for the ability to encapsulate IP information, into an other payload. Here, the underlay is an IP packet, and the encapsulation used is MPLS. The overlay IP information is the original IP packet with its payload, coming from one of the mutiple virtual private networks.

L3VPN overview

L3VPN helps in interconnecting VPNs between several sites, by using a single BGP connection, and at the same time keeping isolation between the different VPNs. L3VPN is based on MPLS technology, and separates the various VPNs connections with MPLS labels handled by BGP. Adding to this, L3VPN offers powerful tools to share (or not) traffic between VPNs by introducing some new RT concepts (see definition below): those tools permit to implement what we call vrf route leaking entries. Those routes can also be used without MPLS for defining local importation and exportation policies between VRs. More generally, those concepts apply to EVPN as per EVPN route target configuration use case.

../../../../_images/l3vpn-bgp-config.svg

BGP l3vpn use case example

Above drawing illustrates a setup made up of 2 symmetrical sites. Each site is separated with a PE device. In the case the site is a data center, the PE could be replaced with a DC-GW. To simplify, each site is made up of 2 distinct VPNs. L3VPN functionality helps to exchange information about the 2 VPNs, between the 2 sites.

This functionality will subsequently enable data path forwarding. IP Traffic between the 2 PEs will be encapsulated into an MPLS label. On each site, traffic between each CE and the PE is standard IP over ethernet traffic.

From the drawing, the following use cases will be more in detail leveraged successively in that document:

  • how to interconnect traffic from different VPNs on a same site. This is a specific case from route-leaking. The basic L3VPN commands will be illustrated, as well as the commmands to create Cross-VRF interfaces across VRF. Note that this kind of method is recommended only if the user has interest in interconnecting also VPNs traffic between sites. If this is not the case, an other method to establish cross-VRF routes is recommended at following link: BGP use case for cross-vrf routes.

  • how to interconnect traffic from a same VPN between sites. This use case will generalise usage of Cross-VRF interfaces with default VRF, as well as explaining how to configure backbone so as to carry MPLS labels.

  • how to interconnect traffic from different VPNs between sites This use case will generalise the L3VPN use case in an MPLS based framework.

L3VPN terminology

It is important to understand some L3VPN terminology. In this paragraph we will give the most important concepts.

VPN

This acronym refers here to a routing entity, also called VRF Creating a VPN consists in creating a BGP instance in a separated VRF. More information in BGP VRF.

Route Distinguisher, RD :

This attribute is specific for each VPN. This information is exported along with the L3VPN information of the BGP information

L3VPN

This refers to creating an overlay with IP packets. In this chapter, L3VPN refers to encapsulating IP traffic over MPLS traffic.

VPNv4, VPNv6

This refers to RFC 4364: that defines how BGP implements L3VPN with MPLS

Route Target, RT:

RT and RD share the same format. A VPN can have 2 list of RTs. One is dedicated for import. This will help BGP to import incoming routing entries that come from a remote BGP entity. There is also a list for export, that is sent to remote BGP entities. Route Target is the key element for sharing information across VPNs.

VRF route leaking:

This refers to the ability to share a route from a VPN to an other entity. See chapter BGP VRF route leak. This ability requires that the VPN that shares a common route, do not have overlapping with the IP provisioning of the networks.

Configuring locally route leaking between VPN

BGP configuration

Below configuration illustrates a setup made up with 2 VPNs. As can be seen, there is a BGP instance in each of the 2 VR instances, plus the BGP core instance. The 2 instances are configured so as to create leaking between both VPNs. Actually configuration shows the following:

  • the RD is different, indicating that there are 2 distinct VPNs.

  • the RT export settings of each VPN is being matched by the RT import settings of the other VPN. This configuration means that route leaking will occur. Practically, export RT of customer1 is 11:22, and import RT of customer2 is 11:22 and 22:44. 11:22 value matches, this means that customer2 will import information coming from customer1.

vrf main
  routing bgp
    as 65500
    ..
    ..
  ..
vrf customer1
  routing bgp
    as 65500
    address-family ipv4-unicast
      network 192.168.3.0/24
        ..
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 11:22
      l3vpn export vpn true
      l3vpn import route-target 11:22 route-target 22:44
      l3vpn import vpn true
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    as 65500
    address-family ipv4-unicast
      network 192.168.2.0/24
        ..
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 22:44
      l3vpn export vpn true
      l3vpn import route-target 11:22 route-target 22:44
      l3vpn import vpn true
      ..
      ..
    ..
    ..
  ..

When using above configuration, it is mandatory to create BGP core instance. Also, despite RD and RT values could have been the same, it has been deliberately chosen to have distinct values to better understand the mechanisms put in place when dealing with L3VPN importation and exportation. Also, it is common, when a L3VPN setup is put in place between 2 ISPs, that the RD is self to each operator, while the RT will be chosen accordingly by operators.

Note

Above configuration details how routing information is exchanged, but does not explain in detail how data traffic is sent through. The product design choices opted for strongly isolating traffic across VPNs. To pass traffic across VPNs, it is required to create special interfaces by configuration. Those interfaces will make the connection between VPNs. More information will be given about how to create those interfaces in a separate chapter. Next chapters assume the user is familiar with interfaces used for crossing vrfs.

using IPless Virtual Ethernet interfaces

Using Cross-VRF interfaces to perform vrf route leaking with BGP requires a specific semantic between VRs and interface names. VR naming must meet the requirements of interface naming. Actually, the Cross-VRF interface name chosen must be equal to the target VR the interface is connected to. To illustrate, in order to reach VR foo from VR bar, an Cross-VRF interface named foo has to be created in VR bar. Reversely, an Cross-VRF interface named bar has to be created in VR foo. In this way, the interface foo and the interface bar will be connected together. The naming convention is not only done to reflect the intent of the interface. It is mandatory to configure it in this way, if one wants to benefit from route leaking across VRs, using Cross-VRF interfaces, and BGP.

From the user point of view, if a packet is emitted in one interface of a source VR, the same packet will be received in the associated veth interface of destination VR. More information about Cross-VRF interfaces can be found in XVRF Interface types. In order to have the setup working fine, the below configuration has to be appended to the former configuration of previous chapter (BGP VRF leak).

vrf customer1
  interface xvrf customer2
    link-interface customer1
    link-vrf customer2
    ..
    ..
  ..
vrf customer2
  interface xvrf customer1
    link-interface customer2
    link-vrf customer1
    ..
    ..
  ..

With the above configuration applied, VR route leaking is possible. Subsequently, if BGP peering is done between a CE and the BGP instance of each VR instance, then route importation and exportation occurs. Below output demonstrates that the routes from customer2 have been imported to customer1. The VR route leak are visible with the @1< indicating that the route entry is originated from VR customer2.

rt1> show bgp vrf customer1 ipv4
BGP table version is 20, local router ID is 1.1.1.1, vrf id 2
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i10.101.0.0/24    1.1.1.2                       100      0 i
*>i10.101.1.0/24    1.1.1.2                       100      0 i
*>i10.101.2.0/24    1.1.1.2                       100      0 i
[..]
*> 10.201.0.0/24    2.2.2.3@1<                    100      0 i
*> 10.201.1.0/24    2.2.2.3@1<                    100      0 i
*> 10.201.2.0/24    2.2.2.3@1<                    100      0 i
[..]

Routing output

Once those entries selected in the bgp RIB, nothing prevents the installation of those VRs routes in the underlying system. Packets going from a VR to an other VR are using the Cross-VRF interface created. There is no specific encapsulation, only a route using an interface as gateway. From above example, the following output displays the routing entries available in VRF routing table. As can be seen, the route to reach the remote prefix first reaches the customer2 interface (first line directly connected, customer2). Then, once the other VR reached, the original BGP route of customer2 routes the traffic to the correct destination (via 2.2.2.3 lines). The latter entry is only here for informational purpose. To get information about routes in VR customer2, use the associated show command to dump route entris in the associates VR.

rt1> show ipv4-routes vrf customer1
BGP table version is 20, local router ID is 1.1.1.1, vrf id 2
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
VRF customer1:
B>* 10.101.0.0/24 [200/0] via 1.1.1.2, eth1_0, 00:05:06
B>* 10.101.1.0/24 [200/0] via 1.1.1.2, eth1_0, 00:05:06
B>* 10.101.2.0/24 [200/0] via 1.1.1.2, eth1_0, 00:05:06
[..]
B>* 10.201.0.0/24 [200/0] is directly connected, customer2, 00:05:06
  *                       via 2.2.2.3, eth2_0(vrf customer2), 00:05:06
B>* 10.201.1.0/24 [200/0] is directly connected, customer2, 00:05:06
  *                       via 2.2.2.3, eth2_0(vrf customer2), 00:05:06
B>* 10.201.2.0/24 [200/0] is directly connected, customer2, 00:05:06
  *                       via 2.2.2.3, eth2_0(vrf customer2), 00:05:06
[..]

how to interconnect traffic from a same VPN between sites.

Being able to interconnect between sites using L3VPN technology is now possible. As explained in the introduction of that chapter, the traffic between sites is encapsulated into an MPLS label, negotiated thanks to BGP. More exactly, the ipv4-vpn address-family configured in BGP will obtain from remote the label, and the underlay nexthop to use, to reach the remote. That label will be the encapsulation between 2 VPNs, in one direction. The same mechanism will apply in reverse direction. Adding to this, this label will be conveyed by an other framework. Currently, LDP offers that framework by conveying inner BGP labels in an outer MPLS label negotiated by LDP. More information on chapter (LDP configuration).

In order to activate L3VPN, use the following command under the main BGP core instance. L3VPN address-family must be configured on the same VRF where the LDP configuration is. Usually, the backbone is the main VR instance.

vrf main
  routing bgp
    as 65500
    neighbor 9.9.9.9 remote-as 65512
    neighbor 9.9.9.9 update-source 3.3.3.3
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    ..
    ..
  ..

Consequently, having an L3VPN peering will trigger importation of L3VPN entries. For instance, the presence of remote VPNs and associated prefixes from peer 9.9.9.9 will trigger prefixes importation in the relevant VRs. Those prefixes, if the remote VPNs match the local VPNs will be imported in the associated VR. So as to permit that importation, the associated Cross-VRF interfaces will be created between the main VR and the relevant VRs. The following configuration illustrates the Cross-VRF interfaces.

vrf main
  interface xvrf customer1
    link-interface main
    link-vrf customer1
    ..
    ..
  interface xvrf customer2
    link-interface main
    link-vrf customer2
    ..
    ..
  ..
vrf customer1
  interface xvrf main
    link-interface customer1
    link-vrf main
    ..
    ..
  ..
vrf customer2
  interface xvrf main
    link-interface customer2
    link-vrf main
    ..
    ..
  ..

Also, in order for BGP to be able to export its own labels, BGP must be configured so as to rely on its own labels, either automatically, or by choosing its own. Below configuration illustrates the automatic label chosen by VR customer1, while customer2 chooses to hardset its exportation label to 300.

vrf customer1
  routing bgp
    as 65500
    address-family ipv4-unicast
      network 192.168.2.0/24
        ..
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    as 65500
    address-family ipv4-unicast
      network 192.168.2.0/24
        ..
      l3vpn export label 300
      ..
      ..
    ..
    ..
  ..
../../../../_images/bgp-l3vpn-configuration-leak.svg

L3VPN setup using MPLS based framework

Above diagram illustrates a topology made up of an MPLS based backbone, with LSR devices : rt3 and rt4, and LER devices. Each LER device has a BGP core instance that has ipv4-vpn address-family enabled. Next to each BGP instance, a BGP instance is created in each VR. Each VR stands for a private network, either A or B. The color semantic explains the relationship between the private networks on the various LER devices.

As can be seen with the arrows, each private network is geographically separate, but thanks to L3VPN, the private networks act as if there were only 2 specific private networks. The BGP configuration is depicted below. The LDP and OSPF configuration is out of scope of this chapter.

rt1

vrf main
  routing bgp
    router-id 5.5.5.5
    as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 15.15.15.15 remote-as 65500
    neighbor 9.9.9.9 update-source 5.5.5.5
    neighbor 15.15.15.15 update-source 5.5.5.5
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    neighbor 15.15.15.15 address-family ipv4-vpn enabled true
    ..
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.1
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.2
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..

rt2

vrf main
  routing bgp
    router-id 9.9.9.9
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 15.15.15.15 remote-as 65500
    neighbor 5.5.5.5 update-source 9.9.9.9
    neighbor 15.15.15.15 update-source 9.9.9.9
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    neighbor 15.15.15.15 address-family ipv4-vpn enabled true
    ..
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.10
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.2
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..

rt5

vrf main
  routing bgp
    router-id 15.15.15.15
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 5.5.5.5 update-source 15.15.15.15
    neighbor 9.9.9.9 update-source 15.15.15.15
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    ..
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.20
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.20
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 2:55
      l3vpn import vpn true
      l3vpn export label 300
      ..
      ..
    ..
    ..
  ..

As can be seen, the network prefixes learnt by each BGP instance are either learnt, by adding extra CE configuration linked to each instance, or imported thanks to L3VPN technology. Below show dumps the vpnv4 entries:

rt1> show bgp ipv4 vpn
BGP table version is 27, local router ID is 5.5.5.5, vrf id 0
Default local pref 100, local AS 65500
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:55
*> 10.101.0.0/24    1.1.1.2@2<              100      0 i
*> 10.101.1.0/24    1.1.1.2@2<              100      0 i
*> 10.101.2.0/24    1.1.1.2@2<              100      0 i
[..]
*>i10.101.11.0/24   9.9.9.9              100      0 i
*>i10.101.12.0/24   9.9.9.9              100      0 i
*>i10.101.13.0/24   9.9.9.9              100      0 i
*>i10.101.14.0/24   9.9.9.9              100      0 i
[..]
Route Distinguisher: 2:55
*> 10.201.0.0/24    2.2.2.3@1<              100      0 i
*> 10.201.1.0/24    2.2.2.3@1<              100      0 i
*> 10.201.2.0/24    2.2.2.3@1<              100      0 i
*> 10.201.3.0/24    2.2.2.3@1<              100      0 i
[..]
*>i10.201.11.0/24   9.9.9.9              100      0 i
*>i10.201.12.0/24   9.9.9.9              100      0 i
*>i10.201.13.0/24   9.9.9.9              100      0 i
*>i10.201.14.0/24   9.9.9.9              100      0 i
[..]

As can be seen, all the entries have an underlay nexthop that stands for the nexthop of the remote BGP global instance. If local entries are imported, that nexthop will be used in the route-leak to send traffic to the remote BGP instance. Along with the nexthop, an MPLS label is be sent via BGP, and used on top of the MPLS backbone. MPLS stacking will be performed. The MPLS backbone will handle the LSP path.

It is worth to be noted too, that the first 3 visible entries are locally exported entries. The nexthop to use 1.1.1.2 is located in the VR customer1. The relationship between the VR name and the VR identifier displayed ( the @2 value) can be done using the following command:

rt1> show bgp vrfs
Type  Id     routerId          #PeersCfg  #PeersEstb  Name
             L3-VNI            RouterMAC              Interface
DFLT  0      5.5.5.5           1          1           main
             0                 00:00:00:00:00:00      unknown
 VRF  2      1.1.1.1           1          1           customer1
             0                 00:00:00:00:00:00      unknown
 VRF  1      2.2.2.2           1          1           customer2
             0                 00:00:00:00:00:00      unknown

The label value 80 is the exported value sent to the remote BGP peers. That value is received by remote BGP speaker 9.9.9.9 for instance, along with the undelay network:

rt2> show bgp ipv4 vpn
BGP table version is 20, local router ID is 9.9.9.9, vrf id 0
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:55
*>i10.101.0.0/24    5.5.5.5              100      0 i
    UN=5.5.5.5 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.1.0/24    5.5.5.5              100      0 i
    UN=5.5.5.5 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.2.0/24    5.5.5.5              100      0 i
    UN=5.5.5.5 EC{1:55} label=80 type=bgp, subtype=0
[..]

After having checked that L3VPN peering received the correct information, it is possible to check against available entries in the bgp vrf customer1 instance like follows:

rt1> show bgp vrf customer1 ipv4
BGP table version is 32, local router ID is 1.1.1.1, vrf id 2
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i10.101.0.0/24    1.1.1.2                       100      0 i
*>i10.101.1.0/24    1.1.1.2                       100      0 i
*>i10.101.2.0/24    1.1.1.2                       100      0 i
[..]
*> 10.101.11.0/24   9.9.9.9@0<                    100      0 i
*> 10.101.12.0/24   9.9.9.9@0<                    100      0 i
*> 10.101.13.0/24   9.9.9.9@0<                    100      0 i
[..]
*> 10.101.22.0/24   15.15.15.15@0<                100      0 i
*> 10.101.23.0/24   15.15.15.15@0<                100      0 i
*> 10.101.24.0/24   15.15.15.15@0<                100      0 i
[..]

Like for the previous dump of vpn entries, it is worth to be noted that remote vpn entries have their underlay nexthop visible, but annotated with @0< indicating that this is a VR route leak going to the VPN. Only 1.1.1.2 ip address is a locally reachable IP address. Actually, this is a CE of that instance.

Routing output

Once those entries selected in the bgp RIB, nothing prevents the installation of those VR routes in the underlying system.

As remind, packets going from a VR to a remote VPN are first mpls encapsulated once. Then, a second encapsulation takes place, as the backbone is MPLS based. Then MPLS does the job to forward the packet to the nexthop marked UN. Reversely, because the BGP vrf instance is at the LER place, the incoming packets are only encapsulated with the negotiated BGP label. So, to go from the backbone VR to the local VR, the packet is being popped its mpls label.

From above example, the following output can be extracted from the VR routing table. As illustrated, the installed route entry performs a double MPLS encapsulation (82/80). The inner value is the negotiated BGP value. The 82 value is an intermediate value that is being swapped by the negotiated MPLS value on the backbone.

rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR,
       > - selected route, * - FIB route

VRF customer1:
B>* 10.101.0.0/24 [200/0] via 1.1.1.2, eth1_0, 00:27:24
B>* 10.101.1.0/24 [200/0] via 1.1.1.2, eth1_0, 00:27:24
B>* 10.101.2.0/24 [200/0] via 1.1.1.2, eth1_0, 00:27:24
[..]
B>* 10.101.11.0/24 [200/0] is directly connected, main, label 82/80, 00:26:43
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:26:43
  *                          via 6.6.6.3, eth0_0(vrf main), label 17, 00:26:43
B>* 10.101.12.0/24 [200/0] is directly connected, main, label 82/80, 00:26:43
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:26:43
  *                          via 6.6.6.3, eth0_0(vrf main), label 17, 00:26:43
B>* 10.101.13.0/24 [200/0] is directly connected, main, label 82/80, 00:26:43
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:26:43
  *                          via 6.6.6.3, eth0_0(vrf main), label 17, 00:26:43
[..]
B>* 10.101.22.0/24 [200/0] is directly connected, main, label 84/300, 00:26:40
                           via 15.15.15.15(vrf main) (recursive), label 300, 00:26:40
  *                          via 6.6.6.3, eth0_0(vrf main), label 22, 00:26:40
B>* 10.101.23.0/24 [200/0] is directly connected, main, label 84/300, 00:26:40
                           via 15.15.15.15(vrf main) (recursive), label 300, 00:26:40
  *                          via 6.6.6.3, eth0_0(vrf main), label 22, 00:26:40
B>* 10.101.24.0/24 [200/0] is directly connected, main, label 84/300, 00:26:40
                           via 15.15.15.15(vrf main) (recursive), label 300, 00:26:40
  *                          via 6.6.6.3, eth0_0(vrf main), label 22, 00:26:40
[..]

The next command dumps the LFIB of the backbone. As you can see, the 82 value is replaced by the 17 value, and forwarded thanks to calculated LSP. Actually, this entry stands for the path to rt2. For reverse direction, incoming packets are being popped and redirected to the Cross-VRF interface leading to the VRF ( here customer1).

rt1> show mpls table
 Inbound                            Outbound
   Label     Type          Nexthop     Label
--------  -------  ---------------  --------
[..]
      80      BGP         customer1
      81      BGP         customer2
      82      BGP          6.6.6.3        17
      83      BGP          6.6.6.3        17
      84      BGP          6.6.6.3        22

how to interconnect traffic from different VPNs between sites

By integrating examples of the 2 previous chapters, it becomes possible to perform interconnection between various VPNs between sites. Also, some VR route leaking across VPNs is done. Note that like for previous chapters, configuring BGP only is not enough, since conveying MPLS labels from BGP needs an other MPLS framework to be configured. LDP protocol is the framework that should be used. More information on chapter (:ref: LDP configuration <mpls-ldp-configuration>). This being said, next chapter explores how to simplify, eventually remove the usage of LDP.

../../../../_images/bgp-l3vpn-configuration-leak.svg

L3VPN setup using MPLS based framework with VR leaking.

The same topology as for previous chapter is chosen. Black arrows indicate the traffic that will be authorised to pass between different VPNs. For instance, network A and network B can access each other. Note that not all data flow are illustrated in the drawing, but it is also possible to do VR route leak between network A from rt1 and network B from rt1. The L3VPN importation and exportation rules for a defined VPN apply for that VPN, whatever its location is, ie local or remote.

Below configurations are BGP configuration changes. Usually, that kind of configuration can be used, when some resources are shared with other VRs: A video server located on that shared resource, or a management vr that is only able to access to th other VRs.

rt1

vrf main
  routing bgp
    router-id 5.5.5.5
    as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 15.15.15.15 remote-as 65500
    neighbor 9.9.9.9 update-source 5.5.5.5
    neighbor 15.15.15.15 update-source 5.5.5.5
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    neighbor 15.15.15.15 address-family ipv4-vpn enabled true
    ..
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.1
    as 65500
    address-family ipv4-unicast
      maximum-path ebgp 4
      maximum-path ibgp 4
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.2
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
  ..

rt2

vrf main
  routing bgp
    router-id 9.9.9.9
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 15.15.15.15 remote-as 65500
    neighbor 5.5.5.5 update-source 9.9.9.9
    neighbor 15.15.15.15 update-source 9.9.9.9
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    neighbor 15.15.15.15 address-family ipv4-vpn enabled true
    ..
  ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.10
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.2
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..

rt5

vrf main
  routing bgp
    router-id 15.15.15.15
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 5.5.5.5 update-source 15.15.15.15
    neighbor 9.9.9.9 update-source 15.15.15.15
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    ..
  ..
vrf customer1
  routing bgp
    router-id 1.1.1.20
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 1:55
      l3vpn export route-target 1:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label auto
      ..
      ..
    ..
    ..
  ..
vrf customer2
  routing bgp
    router-id 2.2.2.20
    as 65500
    address-family ipv4-unicast
      l3vpn export route-distinguisher 2:55
      l3vpn export route-target 2:55
      l3vpn export vpn true
      l3vpn import route-target 1:55 route-target 2:55
      l3vpn import vpn true
      l3vpn export label 300
      ..
      ..
    ..
    ..
  ..

The following show extract on rt1 dumps the routing entries of A and B located on rt2. Two different MPLS labels chosen by BGP of rt2, are received by rt1, for each network.

rt1> show bgp ipv4 vpn
BGP table version is 20, local router ID is 9.9.9.9, vrf id 0
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 2:55
[..]
*>i10.101.11.0/24       9.9.9.9          100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.12.0/24       9.9.9.9          100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.101.13.0/24       9.9.9.9          100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*>i10.201.11.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
*>i10.201.12.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0
*>i10.201.13.0/24   9.9.9.9              100      0 i
    UN=9.9.9.9 EC{2:55} label=81 type=bgp, subtype=0

When applied to the underlying system, the encapsulation for packets leaving the customer1 VR is different, depending if the target prefix belongs to A or B. 2 additional temporary labels are allocated and are swapped as the show mpls labels indicate.

rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR,
       > - selected route, * - FIB route

VRF customer1:
B>* 10.101.11.0/24 [200/0] is directly connected, main, label 83/80, 00:00:44
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:00:44
  *                          via 6.6.6.3, eth0_0(vrf main), label 18, 00:00:44
B>* 10.101.12.0/24 [200/0] is directly connected, main, label 83/80, 00:00:44
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:00:44
  *                          via 6.6.6.3, eth0_0(vrf main), label 18, 00:00:44
B>* 10.101.13.0/24 [200/0] is directly connected, main, label 83/80, 00:00:44
                           via 9.9.9.9(vrf main) (recursive), label 80, 00:00:44
  *                          via 6.6.6.3, eth0_0(vrf main), label 18, 00:00:44
B>* 10.201.11.0/24 [200/0] is directly connected, main, label 82/81, 00:08:15
                           via 9.9.9.9(vrf main) (recursive), label 81, 00:08:15
  *                          via 6.6.6.3, eth0_0(vrf main), label 19, 00:08:15
B>* 10.201.12.0/24 [200/0] is directly connected, main, label 82/81, 00:08:15
                           via 9.9.9.9(vrf main) (recursive), label 81, 00:08:15
  *                          via 6.6.6.3, eth0_0(vrf main), label 19, 00:08:15
B>* 10.201.13.0/24 [200/0] is directly connected, main, label 82/81, 00:08:15
                           via 9.9.9.9(vrf main) (recursive), label 81, 00:08:15
  *                          via 6.6.6.3, eth0_0(vrf main), label 19, 00:08:15

rt1> show mpls table
 Inbound                            Outbound
   Label     Type          Nexthop     Label
--------  -------  ---------------  --------
      80      BGP         r1-cust1
      81      BGP         r1-cust2
      82      BGP          6.6.6.3        18
      83      BGP          6.6.6.3        18
      84      BGP          6.6.6.3        18
      85      BGP          6.6.6.3        18

An additional specificity of the setup is the possibility to import ECMP entries coming from 2 separate locations; here, some 32 bit host routes are retrieved. Some of the entries stand for a server with the same IP, but geographically at 2 different places, namely rt2 and rt5. This kind of scenario can be used for servers that require availability, or where load is split in two, to avoid starvation of the resources of one of the machines.

rt1> show bgp ipv4 vpn
BGP table version is 20, local router ID is 9.9.9.9, vrf id 0
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*=i10.101.18.10/32  15.15.15.15              100      0 i
    UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i                 9.9.9.9              100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*=i10.101.19.10/32  15.15.15.15              100      0 i
    UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i                 9.9.9.9              100      0 i
    UN=9.9.9.9 EC{1:55} label=80 type=bgp, subtype=0
*=i10.101.20.10/32  15.15.15.15              100      0 i
    UN=15.15.15.15 EC{1:55} label=300 type=bgp, subtype=0
*>i                 9.9.9.9              100      0 i

rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR,
       > - selected route, * - FIB route

VRF customer1:
B>* 10.101.18.10/32 [20/0] is directly connected, main, label 19/80, 00:00:09
                          via 9.9.9.9(vrf main) (recursive), label 80, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 20, 00:00:09
  *                       is directly connected, main, label 22/300, 00:00:09
                          via 15.15.15.15(vrf main) (recursive), label 300, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 21, 00:00:09
B>* 10.101.19.10/32 [20/0] is directly connected, main, label 19/80, 00:00:09
                          via 9.9.9.9(vrf main) (recursive), label 80, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 20, 00:00:09
  *                       is directly connected, main, label 22/300, 00:00:09
                          via 15.15.15.15(vrf main) (recursive), label 300, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 21, 00:00:09
B>* 10.101.20.10/32 [20/0] is directly connected, main, label 19/80, 00:00:09
                          via 9.9.9.9(vrf main) (recursive), label 80, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 20, 00:00:09
  *                       is directly connected, main, label 22/300, 00:00:09
                          via 15.15.15.15(vrf main) (recursive), label 300, 00:00:09
  *                         via 6.6.6.3, eth0_0(vrf main), label 21, 00:00:09

Interconnecting traffic between direct connections

As said in previous interconnection chapters, having an MPLS framework is mandatory to interconnect several VPNs across multiple site. Above examples use LDP. Main reason to use LDP is that MPLS technology must be configured at each hop separating two devices (MPLS is a layer 2 technology). And the border devices should not be aware of the number of hops that separate from remote device. One solution to simplify the configuration and to not rely on external dependencies is to reduce to 1 hop the distance between the two devices that should interconnect. Following examples apply:

  • device acting as ASBR, and directly connecting to remote ASBR. In that case, eBGP is used.

  • a GRE tunnel is used to interconnect two devices. As GRE tunnel is a point to point technology, then the traffic flowing across that interface is reduced to 1 hop only, independently of the framework to be crossed.

Using LDP as framework

Below configuration illustrates the configuration of the two devices interconnected. The configuration does not include the VPN configuration part.

rt1

vrf main
  routing bgp
    router-id 5.5.5.5
    as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 9.9.9.9 update-source 5.5.5.5
    neighbor 9.9.9.9 address-family ipv4-unicast enabled false
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    ..
  ..
  interface loopback loop1
    ipv4 address 5.5.5.5/32
    ..
  interface physical eth1
    port pci-b0s4
    ipv4 address 10.1.2.5/24
    ..
  interface gre gre1
    link-interface eth1
    ipv4-address 40.0.0.5/24
    local 10.1.2.5
    remote 10.3.2.9
    ..
    ..
  routing ospf
    router-id 10.1.2.5
    network 10.1.2.0/24
    ..
    ..
  routing static
    ipv4-route 9.9.9.9/32 next-hop 40.0.0.9
    ..
    ..
  routing mpls ldp
    router-id 5.5.5.5
    address-family ipv4
      discovery transport-address 5.5.5.5
      interface gre1
         ..
      ..
    ..
    ..
    ..

rt2

vrf main
  routing bgp
    router-id 9.9.9.9
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 5.5.5.5 update-source 9.9.9.9
    neighbor 5.5.5.5 address-family ipv4-unicast enabled false
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    ..
  ..
  interface loopback loop1
    ipv4 address 9.9.9.9/32
    ..
    ..
  interface physical eth1
    port pci-b0s4
    ipv4 address 10.3.2.9/24
    ..
    ..
  interface gre gre1
    link-interface eth1
    ipv4 address 40.0.0.9/24
    local 10.3.2.9
    remote 10.1.2.5
    ..
    ..
  routing ospf
    router-id 10.3.2.9
    network 10.3.2.0/24 area 0
    ..
    ..
  routing static
    ipv4-route 5.5.5.5/32 next-hop 40.0.0.5
    ..
    ..
  routing mpls ldp
    router-id 9.9.9.9
    address-family ipv4
      discovery transport-address 9.9.9.9
      interface gre1
         ..
      ..
    ..
    ..
    ..
    ..
    ..

As shown below, an implicit-null label has been bound to the route to the next-hop of remote BGP speaker. You can note that 40.0.0.2 is the gateway used to reach remote 9.9.9.9.

rt1> show ipv4-routing
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route


VRF main:
[..]
C>* 5.5.5.5/32 is directly connected, lo, 00:02:21
S>* 9.9.9.9/32 [1/0] via 40.0.0.2, gre1, label implicit-null, 00:00:19
C>* 40.0.0.0/24 is directly connected, gre1, 00:02:21
[..]

By adding VPN configuration which is not present on above configuration, one will have two VPN labels negotiated between BGP independently of LDP; 144 is the outgoing label, and 16 is the incoming label.

rt1> show bgp ipv4 vpn
BGP table version is 2, local router ID is 5.5.5.5, vrf id 0
Default local pref 100, local AS 65500
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:1
[..]
*> 172.16.1.0/24    0.0.0.0@1<         0         32768 ?
*>i172.16.2.0/24    9.9.9.9         0    100      0 ?

The below show ipv4-routes command gives the information on the encapsulation used by a packet going from local vrf customer1 to remote customer2. Outgoing packets leaving rt1 device are encapsulated with label 144/implicit-null. You can note that intermediate step to pass over the vrf border between main and customer1 happens by appending an intermediate label 17 that is popped when directed to 40.0.0.2 destination IP. The additional implicit-null label is chosen since it stands for the label to use when wanting to reach the nexthop of the VPN route. Actually, 9.9.9.9 is reachable via 40.0.0.2 by using implicit-null label.

rt1> show ipv4-routes vrf customer1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route


VRF r1-cust1:
C>* 172.16.1.0/24 is directly connected, r1-eth0, 00:11:15
B>* 172.16.2.0/24 [200/0] is directly connected, main, label 17/144, 00:09:07
                            via 9.9.9.9(vrf main) (recursive), label 144, 00:09:07
  *                         via 40.0.0.2, r1-gre(vrf main), label implicit-null, 00:09:07

Reversely, incoming packets are received with 16/implicit-null label. On rt1, a switching rule indicates that 16 label is popped, and packet is directed to xvrf interface customer1.

rt1> show mpls table
 Inbound                            Outbound
   Label     Type          Nexthop     Label
--------  -------  ---------------  --------
     [..]
      16      BGP         r1-cust1
      17      BGP         40.0.0.2  implicit-null

Using Static Label configuration

Below configuration illustrates is an alternative to previous configuration. Without LDP, but by relying on static MPLS configuration, it is possible to do convey VPN labels transparently across a backbone, inside GRE tunnels. VPN traffic conveyed will use the static MPLS routes labels. Below static route used label number 3 that stands for ipv4 implicit-null value.

rt1

vrf main
  routing bgp
    router-id 5.5.5.5
    as 65500
    neighbor 9.9.9.9 remote-as 65500
    neighbor 9.9.9.9 update-source 5.5.5.5
    neighbor 9.9.9.9 address-family ipv4-unicast enabled false
    neighbor 9.9.9.9 address-family ipv4-vpn enabled true
    ..
  ..
  interface loopback loop1
    ipv4 address 5.5.5.5/32
    ..
  interface physical eth1
    port pci-b0s4
    ipv4 address 10.1.2.5/24
    ..
  interface gre gre1
    link-interface eth1
    ipv4-address 40.0.0.5/24
    local 10.1.2.5
    remote 10.3.2.9
    ..
    ..
  routing ospf
    router-id 10.1.2.5
    network 10.1.2.0/24
    ..
    ..
  routing static
    ipv4-route 9.9.9.9/32 next-hop 40.0.0.9%gre1 label 3
    ..
    ..

rt2

vrf main
  routing bgp
    router-id 9.9.9.9
    as 65500
    neighbor 5.5.5.5 remote-as 65500
    neighbor 5.5.5.5 update-source 9.9.9.9
    neighbor 5.5.5.5 address-family ipv4-unicast enabled false
    neighbor 5.5.5.5 address-family ipv4-vpn enabled true
    ..
  ..
  interface loopback loop1
    ipv4 address 9.9.9.9/32
    ..
    ..
  interface physical eth1
    port pci-b0s4
    ipv4 address 10.3.2.9/24
    ..
    ..
  interface gre gre1
    link-interface eth1
    ipv4 address 40.0.0.9/24
    local 10.3.2.9
    remote 10.1.2.5
    ..
    ..
  routing ospf
    router-id 10.3.2.9
    network 10.3.2.0/24 area 0
    ..
    ..
  routing static
    ipv4-route 5.5.5.5/32 next-hop 40.0.0.5%gre1 label 3
    ..
    ..

Above configuration results in having a label bound to route to reach BGP next-hop, that is to say route via 40.0.0.2. In that case, when resolving route from customer1 VRF, the VPN label would be appended with the static label. The show commands from above example are the same with or without LDP.