BGP for EVPN

Overview

BGP routing protocol is a very rich routing protocols and provide L3VPN and L2VPN features. More information about L3VPN can be read in BGP L3VPN use case example. L2VPN stands for the ability to carry layer data (MAC-level frames) over standard encapsulation protocols. More specifically, the underlay is an IP packet with VXLAN header used as encapsulation technique; while the overlay is a layer 2 frame containing MAC information and IP information. BGP is able to use the benefits of VXLAN tunnels. This permits ISPs to provide network segmentation in VNI (virtual network identifier in a vxlan header) instead of using VLAN for segmenting the network. Also BGP is well suited to handle IP routing of the underlay. Generally, EVPN sits on PE machines.

EVPN is able to carry IP information, but also MAC information in its signaling protocol. The information is collected from routing tables of each VPN, but also neighboring tables, and bridge tables. Actually, layer 2 connectivity can be obtained thanks to information contained in a virtual bridge attached to the vxlan interface.

EVPN is also able to handle BUM traffic, like if it was a local switch. As vxlan interface is a bridge port with possibly multiple tunnel endpoint entries on the same port, outgoing BUM packets are duplicated and sent to various endpoints.

Also, EVPN uses the same semantic as for L3VPN by handling RTs in extended communities, and using RD in NLRI prefixes. This permits easier interconnection between sites.

Initially, the EVPN standard has first been declined with MPLS underlay (with RFC 7432). The main features of EVPN have been proposed, leveraging layer 2 connectivity. Then, the technology evolved, with the increasing usage of overlay technology in data centers. Inter Subnet Forwarding concept has been proposed. This routing mode has been introduced in EVPN, and permits routing overlay information between different sites, similar to what L3VPN technology does with MPLS. Practically, once the routing information exchanged, a bridge interface is used at each side, where packets are routed. Then the MAC layer used to forge overlay packets are the MAC addresses of the bridge interfaces.

EVPN terminology

Ethernet VPN, EVPN:

This refers to creating an overlay with layer 2 frames. In this chapter, it refers to encapsulating IP traffic over VXLAN tunnel. in multi protocol BGP, EVPN refers to a specific address family with AFI identifier set to 25 and SAFI identifier set to 70.

Route Distinguisher, RD:

This attribute is specific for each VPN. This information is exported along with the EVPN information of the BGP information. By configuration, a VPN is often associated to a VNI.

Route Target, RT:

RT and RD share the same format. An EVPN can have 2 list of RTs. One is dedicated for import. This will help importing MAC entries to the appropriate VPN if it deals with route type 2 entries, or IP to the BGP instance associated to the VPN of the instance, if it deals with route type 5 entries. The other one is dedicated for export, and is proposed in the BGP update message where either RT2 or RT2 prefixes are encoded and shared with other VPNs.

Route type 2, RT2:

This prefix refers to the list of attributes used to define a MAC entry in the EVPN concept. It is made up of a RD, a MAC address, an optional associated IP address, an EVI, and an ESI. The two last concepts are not used in below examples, but are respectively related to broadcast domain separation, and multi- homing. In VXLAN topology, the VNI comes along with the prefix and is encoded in the MPLS label field of the prefix (because initially, EVPN protocol has been made for MPLS). So, that value is not an MPLS value, and can be decoded without looking at BOS value like for MPLS.

Route type 5, RT2:

This prefix refers to the list of attributes used to define an IP entry located behind a virtual switch in routing mode. It is made up of a RD, an IP address, an EVI, and an ESI. The two last concepts are not used in below examples, but are respectively related to broadcast domain separation, and multi-homing. In VXLAN topology, the VNI comes along with the prefix and is encoded in the MPLS label field. In the EVPN context, that value is not an MPLS value, and can be decoded without looking at BOS value like for MPLS.

Route type 3, RT3:

This prefix stands for the inclusive multicast ethernet tag route, and is used to signify that a sub network defined by its RD accept BUM traffic. The prefix comes along with the tunnel end-point; that means that BUM traffic can be sent to that tunnel endpoint.

Configuring EVPN

Principles of configuration

The following chapters enter more in detail on how to route or bridge traffic into VXLAN tunnels. The BGP services are differently configured, whether routing mode or bridging mode is used. Let us begin with the bridge and vxlan interfaces.

bridge and vxlan intrerfaces

To be able to perform EVPN, the core technology relies on a VXLAN interface bridged with a bridge interface. The VXLAN interface link-interface must be on the same VRF where the backbone is, that is to sat the main vrf. Also you can note that you have to colocate both VXLAN and bridge interfaces on the same VRF. There is no other restriction regarding in which VRF both interfaces should be. Regarding the VXLAN interface, VNI value will be configured. The destination IP address of VXLAN interface does not need to be configured, as BGP will configure its own destination IP on the underlay. The configured VXLAN and bridge interfaces will be used by BGP to discover which VNI is present on the device, and which information to send to remote peers.

EVPN service

In order to activate EVPN, use the following command under the main BGP core instance. l2vpn-evpn address-family must be configured. Here, the main BGP core instance will play the backbone role.

vrf main
   routing
      bgp
         as 65500
         address-family
            l2vpn-evpn
               advertise-all-vni true
               ..
            ..

The advertise-all-vni keyword will trigger local discovery of all vxlan and bridge interfaces available, so that BGP will retrieve VNI and use that information to send to remote peers. It is to be noted that the discovery takes into account all vr-s instances. So basically, whatever where the VRF is, all VXLAN interfaces will be learnt.

For bridging mode, configuring the BGP main instance is enough. In routing mode, it is usual to configure overlay networking information in separate VRs. For that, if a VRF is dedicated to routing network into a VXLAN interface, then an additional BGP instance attached to the new VRF instance will need to be created in order to perform routing mode. Also, the VRF will be mapped to the VNI by configuration. Subsequently, extra configuration can be done under each BGP instance, directly under the address-family l2vpn-evpn address-family.

Below figure illustrates what does routing mode and bridging mode means. As can be seen, two pe devices are interconnected with EVPN. On each device, a VXLAN interface and a bridged interface are linked together, as well as an ethernet port connected to local host devices. The VRF where both bridge and vxlan interfaces sit does not matter, provided that the link information of the VXLAN interface is on the main VRF.

Two data flows are illustrated. The scheme reuses the same VXLAN interface, but practically, it is necessary to create a VXLAN interface for each kind of connectivity. On the one hand, the blue one stands for layer 2 connectivity. Data traffic is bridged on pe devices, that is to say that traffic is bridged from ethernet port to vxlan port, where traffic is encapsulated into vxlan header and transmitted to remote pe. From hostA to hostB, traffic is full layer 2. On the other hand, the green one stands for layer 3 connectivity. This flow connects two networks together, namely network B and network C. All happens as if the traffic from network B was redirected to gateway in rt2, except that the gateway of rt2 is the remote bridged interface. The forged packet inside VXLAN packet will then be made of the MAC addresses of the two bridged interfaces.

../../../../_images/evpn-bgp-config.svg

BGP evpn use case example

Basic Configuration

EVPN uses a new address-family with AFI identifier set to 25 and SAFI identifier set to 70. Configuring EVPN goes along with the configuration of a bridge interface bridged with a VXLAN interface.

Here below is an example on how to configure a sample BGP configuration with EVPN enabled. As illustrated below, the configuration must include the presence of both vxlan interface and bridge interface.

vrf main
   routing
      bgp
         router-id 10.125.0.1
         as 65500
         address-family
            l2vpn-evpn
               advertise-all-vni true
               vni 11
                 advertise-default-gw true
                 export
                    route-distinguisher 65500:11
                    ..
                 ..
               ..
            ..
         neighbor 10.125.0.3
            remote-as 65500
            address-family l2vpn-evpn enabled true
            /
vrf custom1
   interface
      physical eth0
        ipv4 address 10.125.0.1/24
        port pci-b0s5
        ..
      vxlan vxl11
         vni 11
         local 10.125.0.1
         link-vrf main
         learning true
         link-interface eth0
         link-vrf main
         ..
      ..
   interface
      bridge br11
        link-interface vxl11
      ..
   ..

There is a single global command to enable the EVPN control plane on a VTEP called advertise-all-vni. This will cause the router to learn about all VNIs locally present on the system and the MACs and neighbors (ARP and ND) that pertain to such VNIs and advertise the corresponding information using EVPN procedures to all BGP peers with whom the EVPN address-family has been negotiated. It will also cause any EVPN routes learnt from BGP peers to be installed into the appropriate local VNIs. Received EVPN type-3 routes will translate into the list of remote VTEPs that participate in a particular VNI and received EVPN type-2 routes will get installed as MAC and neighbor entries pertaining to a specific VNI.

It is to be noted that the BGP core instance is used to carry EVPN information, while the other VRs are optionally used to carry the overlay information, be it layer 2 and/or layer 3 information. The mapping between overlay and underlay is based with VNI presence. That implies that the VRs configuration is optional, and for instance, the configuration of a bridge and a virtual interface can be done in the main instance. all traffic that will go through that bridge interface will be subsequently encapsulated, and signaling information will be detected and transmitted within the associated VNI.

Note that we don’t define the remote endpoint of the vxlan interface, as the BGP peer defines it, using the VXLAN interface. That vxlan interface link interface is the same interface where underlay traffic goes through.

To get information about the VXLAN interfaces detected, classified per VNI, the following command can be used to dump the contexts. If the VXLAN interfaces have not been detected, then that implies that a misconfiguration occurred, for instance, if the VXLAN interface has not been bridged. Below example shows that the vxl11 has been detected on vrf custom1, and that a certain number of entries have been learnt, either via bgp learning, or by locally listening for ARP/MAC information ( see MACs and MAC information). The first mac information learnt is the MAC address of the bridged interface.

rt1> show evpn vni all
VNI        Type VxLAN IF              # MACs   # ARPs   # Remote VTEPs  Tenant VRF
11         L2   vxl11                 1        1        1               custom1

rt1> show evpn vni 11
VNI: 11
 Type: L2
 Tenant VRF: custom1
 VxLAN interface: vxl11
 VxLAN ifIndex: 6
 Local VTEP IP: 10.125.0.1
 Mcast group: 0.0.0.0
 Remote VTEPs for this VNI:
  10.125.0.3 flood: HER
 Number of MACs (local and remote) known for this VNI: 1
 Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 3
 Advertise-gw-macip: Yes

rt1> show evpn arp-cache vni 11
Number of ARPs (local and remote) known for this VNI: 1
IP                        Type   State    MAC               Remote VTEP           Seq #'s
fe80::5814:dbff:feba:c854 local  active   f2:de:f1:b6:4e:59                       0/0

To get information about the BGP information exchanged, the following command can be used. Each entry stands for an EVPN route entry. The first number stands for the kind of information shared, ie the route type. For instance, 2 stands for MAC-level information shared (RT2, while 3 (RT3) stands means that this tunnel endpoint is authorized to exchange BUM traffic. The tunnel endpoint can be seen here with the nexthop information. As depicted below, 10.125.0.3 is the tunnel endpoint.

rt1> show bgp l2vpn evpn
BGP table version is 4, local router ID is 10.125.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65500:11
*>i[2]:[0]:[48]:[6e:1c:50:ab:2b:00]:[128]:[fe80::acf9:13ff:fe0d:607a]
                10.125.0.3                    100      0 i
                RT:65500:11 ET:8 Default Gateway ND:Router Flag
*> [2]:[0]:[48]:[f2:de:f1:b6:4e:59]:[128]:[fe80::5814:dbff:feba:c854]
                10.125.0.1                         32768 i
                ET:8 RT:65500:11 Default Gateway ND:Router Flag
*> [3]:[0]:[32]:[10.125.0.1]
                10.125.0.1                         32768 i
                ET:8 RT:65500:11
*>i[3]:[0]:[32]:[10.125.0.3]
                10.125.0.3                    100      0 i
                RT:65500:11 ET:8

Displayed 4 out of 4 total prefixes

It is also possible to do some variants of the call by filtering based on the route type.

rt1> show bgp l2vpn evpn route type multicast
BGP table version is 4, local router ID is 10.125.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
                Extended Community
Route Distinguisher: 65500:11
*> [3]:[0]:[32]:[10.125.0.1]
                10.125.0.1                         32768 i
                ET:8 RT:65500:11
*>i[3]:[0]:[32]:[10.125.0.3]
                10.125.0.3                    100      0 i
                RT:65500:11 ET:8

Displayed 2 prefixes (2 paths) (of requested type)

Also, it is possible to get some details

rt1> show bgp l2vpn evpn route detail
[..]
Route Distinguisher: 65500:11
BGP routing table entry for 65500:11:[2]:[0]:[48]:[6e:1c:50:ab:2b:00]:[128]:[fe80::acf9:13ff:fe0d:607a]
Paths: (1 available, best #1)
  Not advertised to any peer
  Route [2]:[0]:[48]:[6e:1c:50:ab:2b:00]:[128]:[fe80::acf9:13ff:fe0d:607a] VNI 11
  Local
    10.125.0.3 from 10.125.0.3 (10.125.0.3)
      Origin IGP, localpref 100, valid, internal, best (First path received)
      Extended Community: RT:65500:11 ET:8 Default Gateway ND:Router Flag
      Last update: Thu Aug 19 14:05:07 2021
[..]
BGP routing table entry for 65500:11:[3]:[0]:[32]:[10.125.0.3]
Paths: (1 available, best #1)
  Not advertised to any peer
  Local
    10.125.0.3 from 10.125.0.3 (10.125.0.3)
      Origin IGP, localpref 100, valid, internal, best (First path received)
      Extended Community: RT:65500:11 ET:8
      Last update: Thu Aug 19 14:05:07 2021
      PMSI Tunnel Type: Ingress Replication, label: 11
[..]

Route Reflector configuration

It is possible to create a route reflector configuration, then it is not necessary to call advertise-all-vni keyword. Enabling l2vpn-evpn address-family is enough. EVPN routes received from a BGP peer are accepted and maintained in the global EVPN routing table.

Flooding Mode

VXLAN interfaces handled by BGP can be configured with or without acceptance of BUM traffic. By default, head-end replication is used; that implies that all BUM traffic entering to the bridge is sent to the other ports of the bridge. That includes the VXLAN interface. For instance, ARP packets are transmitted through the VXLAN interface. To block that traffic, it is possible to disable flooding, and configure BGP so as to forbid BUM traffic. To inform remote peer that flooding is accepted, RT3 messages are sent. This message indicates that for a given network defined by the RD, BUM traffic is accepted. flooding can be configured as follows:

vrf main
   routing bgp
      address-family l2vpn-evpn
         flooding disabled
         ..
      ..
      ..
   ..

It is possible to reenable flooding by using following command. Consequently, RT3 updates will be propagated.

vrf main
   routing bgp
      address-family l2vpn-evpn
         flooding head-end-replication
         ..
      ..
      ..
   ..

Route Target Configuration

More information about route targets is given in route leaking use case. In L3VPN chapter, it was seen that RD and RTs were used to import and export routes to different VPNs. Here, the same concepts are used, and are applied to all kind of EVPN route types routes. This includes IP routes, but also MAC entries too. Also, the VPN concept is mapped to VNI concept. This is why we refer to VNI.

In addition to the above essential steps, the RD and RTs can be configured for a VNI. By default, RD is automatically derived by using IP4B:NN format, where IP4B stands for the IP address of the router-id used in BGP, and a unique 16 bit field identifier. RTs values is defined as AS:VNI, where AS stands for the AS value of the BGP instance, and VNI is the virtual network identifier of the VXLAN interface. It is possible to redefine the RD value by using another semantic. Below configurations partially reuse the L2 vni configuration, and append a new L3 vni. As described, configuring RD and RTs for L2 (under vni node), and L3 node (under l2vpn-epvn node of separate BGP instance) differs:

vrf main
   routing bgp
      router-id 10.125.0.1
      as 65500
      address-family l2vpn-evpn
         vni 11
           export route-distinguisher 65000:11
/ vrf custom2
   routing bgp
      router-id 10.125.1.1
      as 65500
      address-family l2vpn-evpn
         export route-distinguisher 65000:12
         ..
      ..
      ..
   ..

It is also possible to override route target values, by using following command. Here, the VNI is used for encoding.

vrf main
   routing bgp
      router-id 10.125.0.1
      as 65500
      address-family l2vpn-evpn
         advertise-all-vni true
         vni 11
           export route-target 65000:11
           import route-target 65000:11
 / vrf custom2
   routing bgp
      router-id 10.125.1.1
      as 65500
      address-family l2vpn-evpn
         export route-target 65000:12
         import route-target 65000:12

RFC 8365 explains how RT auto derivation should be done in section 5.1.2.1. The lowest 4 bytes of the AA:NNNN are redefined. The new format is made up of the value 1 in the first 3-bit field ( standing for VXLAN encapsulation), and the VNI value. This encoding is needed for proper interoperability with RT auto-derivation in Junos. To configure this format automatically, use the following command:

vrf main
   routing bgp
      router-id 10.125.0.1
      as 65500
      address-family
         l2vpn-evpn
            auto-route-target rfc8365
 / vrf custom2
   routing bgp
      router-id 10.125.1.1
      as 65500
      address-family l2vpn-evpn
            auto-route-target rfc8365

The RD and RT configuration can be checked, against each VNI discovered.

rt1> show bgp l2vpn evpn vni 11
VNI: 11 (known to the kernel)
 Type: L2
 Tenant-Vrf: custom1
 RD: 10.125.0.3:2
 Originator IP: 10.125.0.3
 Mcast group: 0.0.0.0
 Advertise-gw-macip : Enabled
 Advertise-svi-macip : Disabled
 Import Route Target:
   65500:268435467
 Export Route Target:
   65500:268435467

As said in introduction chapter, EVPN can be used to carry either L2VPN information or L3VPN information. The next chapters respectively discuss how to use EVPN as a L3VPN technology, then as a L2VPN technology.

Inter Subnet Forwarding

Inter Subnet Forwarding is the ability to use a virtual bridge to route information on that bridge. BGP exchanges this information by using RT2 messages. As underlay tunnel carries also MAC-level information, the source and destination MAC addresses used ( and transmitted via BGP) are the MAC addresses of the bridge interface attached. Like for L3VPN, IP prefixes can be assigned to a specific VR, and the bridged interfaces will act as both tunnel endpoint and remote gateway to join a separate remote IP network.

To configure routing, by using a VXLAN interface, its VNI must be configured as an L3VPN vni. By default, each VNI presence detection is seen as a EVPN one. You have to explicitly mention the VNI as a layer 3 VNI.

vrf custom1
   routing l3-vni 11
   interface vxlan vxlan-101 vni 11
   routing bgp as 65500

Subsequently, the VNI layer information is propagated in the system.

rt1> show evpn vni all
VNI       Type VxLAN IF              # MACs   # ARPs   # Remote VTEPs  Tenant VRF
11        L3   vxl11                 1        1        n/a             custom1

The remaining configuration is same as the one presented in the first chapter, ie both a vxlan and slave to a bridge interface in the same VR. To bring clarity, the whole configuration is reused, and is based on ref:BGP EVPN use case example <routing-bgp-evpn-drawing>, where rt1 and rt2 devices configuration are exposed.

rt1

vrf main
   routing
      bgp
         as 65500
         router-id 10.125.0.1
         address-family
            l2vpn-evpn
               advertise-all-vni true
               ..
            ..
         neighbor 10.125.0.2
            remote-as 65500
            address-family
               l2vpn-evpn
                  ..
               ..
            ..
         ..
      ..
   interface physical eth0
      port pci-b0s4
      mtu 1550
      ipv4 address 10.125.0.1/24
      ..
      ..
   ..
vrf custom1
   interface physical eth1
      ipv4 address 10.51.0.1/24
      port pci-b0s6
      ..
   ..
   interface vxlan vxl11
      vni 11
      local 10.125.0.1
      link-interface eth0
      link-vrf main
      ..
   ..
   interface bridge br11
      link-interface vxl11
      ipv4 address 10.50.0.1/24
      ..
   ..
   routing
      l3-vni 11
      bgp
         as 65500
         address-family l2vpn-evpn
              advertise ipv4-unicast
              auto-route-target rfc8365
              export route-distinguisher 65500:11
              ..
           ..
         address-family ipv4-unicast
              redistribute connected
              ..
            ..
         ..
      ..
   ..

rt2

vrf main
   routing
      bgp
         router-id 10.125.0.2
         as 65500
         address-family
            l2vpn-evpn
               advertise-all-vni true
               ..
            ..
         neighbor 10.125.0.1
            remote-as 65500
            address-family
               l2vpn-evpn
                  ..
               ..
            ..
         ..
      ..
   ..
   interface physical eth0
      port pci-b0s4
      mtu 1550
      ipv4 address 10.125.0.2/24
      ..
      ..
   ..
vrf custom1
   interface physical eth1
      ipv4 address 10.52.0.2/24
      port pci-b0s6
      ..
   ..
   interface vxlan vxl11
      vni 11
      local 10.125.0.2
      link-interface eth0
      link-vrf main
      ..
   ..
   interface bridge br11
      link-interface vxl11
      ipv4 address 10.50.0.2/24
      ..
   ..
   routing
      l3-vni 11
      bgp
         as 65500
         address-family l2vpn-evpn
              advertise ipv4-unicast
              auto-route-target rfc8365
              export route-distinguisher 65500:11
              ..
           ..
         address-family ipv4-unicast
              redistribute connected
              ..
            ..
         ..
      ..
   ..

To propagate IP information in the L2VPN, a second BGP instance is created in the VR. That instance will explicitly tell to advertise IP information to the main core instance. That second BGP instance redistributes sub networks from eth1 interface, as depicted in below command:

rt1> show bgp l2vpn evpn
BGP table version is 12, local router ID is 10.125.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65500:11
*> [5]:[0]:[24]:[10.51.0.0]
                10.125.0.1               0         32768 ?
                ET:8 RT:65500:268435467 Rmac:42:f7:18:0a:8e:72
*>i[5]:[0]:[24]:[10.52.0.0]
                10.125.0.2               0    100      0 ?
                RT:65500:268435467 ET:8 Rmac:9a:2a:20:2f:07:69
*> [5]:[0]:[32]:[10.50.0.1]
                10.125.0.1               0         32768 ?
                ET:8 RT:65500:268435467 Rmac:42:f7:18:0a:8e:72
*>i[5]:[0]:[32]:[10.50.0.2]
                10.125.0.2               0    100      0 ?
                RT:65500:268435467 ET:8 Rmac:9a:2a:20:2f:07:69

Displayed 4 out of 4 total prefixes

The below commands show that the imported route entries from remote peers have been accordingly set to the VR. Also, the EVPN entries transmitted in the core instance are dumped too.

rt1> show bgp vrf custom1 ipv4
BGP table version is 10, local router ID is 10.125.0.1, vrf id 11
Default local pref 100, local AS 65500
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.50.0.1/32     0.0.0.0                  0         32768 ?
*>i10.50.0.2/32     10.125.0.2<              0    100      0 ?
*> 10.51.0.0/24     0.0.0.0                  0         32768 ?
*>i10.52.0.0/24     10.125.0.2<              0    100      0 ?

Displayed  4 routes and 4 total paths

That second BGP instance can also be used to connect with remote CE. This permits extending the L3 connectivity behind each PE devices. The propagated routes transmitted to the CE will use the PE as next-hop to reach that subnetwork.

To unconfigure a layer 3 VNI and its associated BGP core instance, use the following command to remove both configuration. Optionally, the bridge and VXLAN interface configuration can be removed:

vrf custom1
   del routing l3-vni
   del routing bgp
   del interface vxlan vxl11
   del interface bridge br11

Bridge Configuration

Bridging permits to aggregate layer 2 networks into a single one, by bridging each network with VXLAN interface. Like for routing, a bridge and a vxlan interface are needed, and need to be bridged, so that BGP populates its VNI list.

Subsequently, the VNI layer information is propagated in the system. BGP uses the VNI information to extract the bridge neighboring information contained to transmit it by using RT2 entries. Based on BGP EVPN use case example, rt1 and rt2 devices configuration are exposed below:

rt1

vrf main
   routing
      bgp
         router-id 10.125.0.1
         as 65500
         address-family
            l2vpn-evpn
               advertise-all-vni true
               auto-route-target rfc8365
               vni 11
                  advertise-default-gw true
                  export route-distinguisher 65500:11
                  ..
               ..
            ..
         neighbor 10.125.0.2
            remote-as 65500
            address-family
               l2vpn-evpn
                  ..
               ..
            ..
         ..
      ..
   interface physical eth0
      port pci-b0s4
      mtu 1550
      ipv4 address 10.125.0.1/24
      ..
      ..
   ..
vrf custom1
   interface physical eth1
      port pci-b0s6
      ..
   ..
   interface vxlan vxl11
      vni 11
      local 10.125.0.1
      link-interface eth0
      link-vrf main
      ..
   ..
   interface bridge br11
      link-interface vxl11
      link-interface eth1
      ipv4 address 10.50.0.1/24
      ..
   ..

rt2

vrf main
   routing
      bgp
         router-id 10.125.0.2
         as 65500
         address-family
            l2vpn-evpn
               advertise-all-vni true
               auto-route-target rfc8365
               vni 11
                  advertise-default-gw true
                  export route-distinguisher 65500:11
                  ..
               ..
            ..
         neighbor 10.125.0.1
            remote-as 65500
            address-family
               l2vpn-evpn
                  ..
               ..
            ..
         ..
      ..
   interface physical eth0
      port pci-b0s4
      mtu 1550
      ipv4 address 10.125.0.2/24
      ..
      ..
   ..
vrf custom1
   interface physical eth1
      port pci-b0s6
      ..
   ..
   interface vxlan vxl11
      vni 11
      local 10.125.0.2
      link-interface eth0
      link-vrf main
      ..
   ..
   interface bridge br11
      link-interface vxl11
      link-interface eth1
      ipv4 address 10.50.0.2/24
      ..
   ..

A summary of the discovered VXLAN interfaces can be seen with following command:

rt1> show evpn vni 11
VNI: 11
 Type: L2
 Tenant VRF: custom1
 VxLAN interface: vxl11
 VxLAN ifIndex: 9
 Local VTEP IP: 10.125.0.1
 Mcast group: 0.0.0.0
 Remote VTEPs for this VNI:
  10.125.0.2 flood: HER
 Number of MACs (local and remote) known for this VNI: 2
 Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 4
 Advertise-gw-macip: Yes

You can note that advertise-default-gw keyword applied to VNI configuration transmits RT2 entries informing about the IPs available on br11 interface. Also, because flooding mode is enabled by default, you can note RT3 entries.

rt1> show bgp l2vpn evpn
BGP table version is 5, local router ID is 10.125.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65500:11
*> [2]:[0]:[48]:[62:20:86:d0:e1:01]:[32]:[10.50.0.1]
                10.125.0.1                         32768 i
                ET:8 RT:65500:268435467 Default Gateway
*> [2]:[0]:[48]:[62:20:86:d0:e1:01]:[128]:[fe80::7ce2:20ff:fe03:9ba]
                10.125.0.1                         32768 i
                ET:8 RT:65500:268435467 Default Gateway ND:Router Flag
*>i[2]:[0]:[48]:[be:21:5b:3e:20:7b]:[32]:[10.50.0.2]
                10.125.0.2                    100      0 i
                RT:65500:268435467 ET:8 Default Gateway
*>i[2]:[0]:[48]:[be:21:5b:3e:20:7b]:[128]:[fe80::bc21:5bff:fe3e:207b]
                10.125.0.2                    100      0 i
                RT:65500:268435467 ET:8 Default Gateway ND:Router Flag
* i[3]:[0]:[32]:[10.125.0.1]
                10.125.0.2                    100      0 i
                RT:65500:268435467 ET:8
*>                  10.125.0.1                         32768 i
                ET:8 RT:65500:268435467

Consequently, the neighboring table is populated with local entries found locally, and remote entries learnt from BGP. For instance, 10.50.0.1 has been detected as a machine in the local network of rt1, and its MAC address and the IP association has been propagated to the remote BGP speaker.

rt1> show evpn arp-cache vni 11
Number of ARPs (local and remote) known for this VNI: 4
IP                        Type   State    MAC               Remote VTEP           Seq #'s
10.50.0.2                 remote active   be:21:5b:3e:20:7b 10.125.0.2            0/0
fe80::7ce2:20ff:fe03:9ba  local  active   62:20:86:d0:e1:01                       0/0
fe80::bc21:5bff:fe3e:207b remote active   be:21:5b:3e:20:7b 10.125.0.2            0/0
10.50.0.1                 local  active   62:20:86:d0:e1:01                       0/0

Also, the MAC table can be seen:

rt1> show evpn mac vni 11
Number of MACs (local and remote) known for this VNI: 2
MAC               Type   Intf/Remote VTEP      VLAN  Seq #'s
be:21:5b:3e:20:7b remote 10.125.0.2                  0/0
62:20:86:d0:e1:01 local  br11                  1     0/0

It is possible to extend the layer 2 network by having a private network behind eth0. On the other way, VTEPs can be increased by multiplying the number of BGP peers and using route-reflector peers. While BUM traffic will be transmitted as if it was a local switch engine, if MAC table gets populated with the right MAC address, then traffic will be transmitted accordingly.