BGP in virtual routers

Multiple BGP instances can be hosted in Virtual Service Router. There are two ways of separating those instances:

  • using VRFs, when strong isolation is needed between each routing instance. BGP per VRF instances are mainly used for connecting to the backbone network.

  • using L3VRFs, when light and scalable isolation is needed. When acting as PE, BGP redistributes VPN routes to CPEs connected to L3VRFs.

BGP configuration, and monitoring in VRF

A BGP router can be instantiated in a separate VRF. The routes learned and configured will be stored in the corresponding forwarding tables. It is possible to create multiple BGP instances on the same machine.

  • Create a BGP instance in a VRF named customer1, by using the following command:

vsr running config# vrf customer1
vsr running vrf customer1# routing bgp
vsr running bgp#! as 65500
vsr running bgp#
[..]

Then you can continue the configuration as usual. The BGP peering and the redistribution will happen on the whole VRF. The interfaces, addresses and routing information of this VRF will be used.

To get more information on a specific VRF, you can use the following command, and the VRF name to get routing information:

vsr> show bgp vrf customer1 ipv4 unicast
BGP table version is 2, local router ID is 1.1.1.1, vrf id 2
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
    i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 3.3.3.0/24       0.0.0.0                  0         32768 i
*> 4.4.4.0/24       0.0.0.0                  0         32768 i

Displayed  2 routes and 2 total paths

BGP use case for VRF

A common use case is to provide a per customer BGP peering. In this case, one machine is used to provide the customer’s access. Each customer has its own network interface (VLAN, physical, etc) located in its dedicated VRF. The following configuration explains how to create multiple BGP instances attached with VLAN interface.

In the below example, 2 instances of BGP are created, each one running over a VLAN interface with its own peer. The same autonomous system can be used for all instances. The BGP contexts don’t share the same system process and don’t share the same forwarding information.

vsr running config# / vrf main interface physical eth2 port pci-b0s5
vsr running config# / vrf customer1 routing bgp as 65555
vsr running config# / vrf customer1 routing bgp router-id 192.168.1.1
vsr running config# / vrf customer1 routing bgp neighbor 192.168.1.2 remote-as 65555
vsr running config# / vrf customer1 interface vlan vlan10 vlan-id 10
vsr running config#! / vrf customer1 interface vlan vlan10 link-interface eth2
vsr running config# / vrf customer1 interface vlan vlan10 link-vrf main
vsr running config# / vrf customer1 interface vlan vlan10 ipv4 address 192.168.1.1/24
vsr running config# / vrf customer2 routing bgp as 65555
vsr running config# / vrf customer2 routing bgp router-id 192.168.2.1
vsr running config# / vrf customer2 routing bgp neighbor 192.168.2.2 remote-as 65555
vsr running config# / vrf customer2 interface vlan vlan20 vlan-id 20
vsr running config#! / vrf customer2 interface vlan vlan20 link-interface eth2
vsr running config# / vrf customer2 interface vlan vlan20 link-vrf main
vsr running config# / vrf customer2 interface vlan vlan20 ipv4 address 192.168.2.1/24

Note

With the above example, it could have been possible to use the same IP mapping for both routing entities. Another benefit of having separate entities is that IP mapping can overlap.

BGP use case for cross-VRF routes

Another common use case is to install cross-VRF routes between 2 BGP instances. The recommended way to do it consists in setting up a mesh of BGP sessions via veth links. The below example illustrates what can be done to install cross-VRF routes between an admin vrf and a service vrf.

vsr running config# vrf admin
vsr running vrf admin# interface physical eth0
vsr running physical eth0#! port pci-b0s5
vsr running physical eth0# ipv4 address 192.168.2.1/24
vsr running physical eth0# ..
vsr running interface# veth service1
vsr running veth service1#! link-interface admin
vsr running veth service1#! link-vrf service1
vsr running veth service1#! ..
vsr running interface#! veth service2
vsr running veth service2#! link-interface admin
vsr running veth service2#! link-vrf service2
vsr running veth service2#! ..
vsr running interface#! loopback localip
vsr running loopback localip#! ipv4 address 169.254.0.100/32
vsr running vrf admin#! routing static ipv4-route 169.254.0.101/32 next-hop service1
vsr running vrf admin#! routing static ipv4-route 169.254.0.102/32 next-hop service2
vsr running vrf admin#! routing bgp
vsr running bgp#! as 12
vsr running bgp#! ebgp-connected-route-check false
vsr running bgp#! router-id 192.168.2.1
vsr running bgp#! address-family ipv4-unicast redistribute connected route-map nolocal
vsr running bgp#! neighbor 192.168.2.2
vsr running neighbor 192.168.2.2#! remote-as 13
vsr running neighbor 192.168.2.2#! address-family ipv4-unicast as-outboun-update
vsr running as-outbound-update#! action remove as-type all
vsr running as-outbound-update#! .. .. .. ..
vsr running bgp#! neighbor-group local update-source localip
vsr running bgp#! neighbor 169.254.0.101
vsr running neighbor 169.254.0.101#! neighbor-group local
vsr running neighbor 169.254.0.101#! remote-as 65501
vsr running neighbor 169.254.0.101#! local-as as-number 65500 no-prepend true replace-as true
vsr running neighbor 169.254.0.101#! neighbor-group local
vsr running neighbor 169.254.0.101#! ..
vsr running bgp#! neighbor 169.254.0.102
vsr running neighbor 169.254.0.102#! remote-as 65502
vsr running neighbor 169.254.0.102#! local-as as-number 65500 no-prepend true replace-as true
vsr running neighbor 169.254.0.102#! neighbor-group local
vsr running neighbor 169.254.0.102#! ..
vsr running bgp#! .. .. ..
vsr running config#! vrf service1
vsr running vrf service1#! interface physical eth1
vsr running physical eth1#! port pci-b0s6
vsr running physical eth1#! ipv4 address 10.1.2.1/24
vsr running physical eth1#! ..
vsr running interface#! veth admin
vsr running veth admin#! link-interface service1
vsr running veth admin#! link-vrf admin
vsr running veth admin#!
vsr running interface#! loopback localip
vsr running loopback localip#! ipv4 address 169.254.0.101/32
vsr running loopback localip#! .. ..
vsr running vrf service1#! routing static ipv4-route 169.254.0.100/32 next-hop admin
vsr running vrf service1#! routing bgp
vsr running bgp#! as 12
vsr running bgp#! ebgp-connected-route-check false
vsr running bgp#! router-id 10.1.2.1
vsr running bgp#! address-family ipv4-unicast redistribute connected route-map nolocal
vsr running bgp#! neighbor 10.1.2.2
vsr running neighbor 10.1.2.2#! remote-as 13
vsr running neighbor 10.1.2.2#! address-family ipv4-unicast as-outbound-update
vsr running as-outbound-update#! action remove as-type all
vsr running as-outbound-update#! .. .. .. ..
vsr running bgp#! neighbor-group local update-source localip
vsr running bgp#! neighbor 169.254.0.100
vsr running neighbor 169.254.0.100#! remote-as 65500
vsr running neighbor 169.254.0.100#! local-as as-number 65501 no-prepend true replace-as true
vsr running neighbor 169.254.0.100#! neighbor-group local
vsr running neighbor 169.254.0.100#! .. .. .. ..
vsr running config#! vrf service2
vsr running vrf service2#! interface physical eth2
vsr running physical eth2#! port pci-b0s6
vsr running physical eth2#! ipv4 address ipv4 address 10.2.2.1/24
vsr running physical eth2#! ..
vsr running interface#! veth admin
vsr running veth admin#! link-interface service2
vsr running veth admin#! link-vrf admin
vsr running veth admin#!
vsr running interface#! loopback localip
vsr running loopback localip#! ipv4 address 169.254.0.102/32
vsr running loopback localip#! .. ..
vsr running vrf service2#! routing static ipv4-route 169.254.0.100/32 next-hop admin
vsr running vrf service2#! routing bgp
vsr running bgp#! as 12
vsr running bgp#! ebgp-connected-route-check false
vsr running bgp#! router-id 10.2.2.1
vsr running bgp#! address-family ipv4-unicast redistribute connected route-map nolocal
vsr running bgp#! neighbor 10.2.2.2
vsr running neighbor 10.2.2.2#! remote-as 13
vsr running neighbor 10.2.2.2#! address-family ipv4-unicast as-outbound-update
vsr running as-outbound-update#! action remove as-type all
vsr running as-outbound-update#! .. .. .. ..
vsr running bgp#! neighbor-group local update-source localip
vsr running bgp#! neighbor 169.254.0.100
vsr running neighbor 169.254.0.100#! remote-as 65500
vsr running neighbor 169.254.0.100#! local-as as-number 65502 no-prepend true replace-as true
vsr running neighbor 169.254.0.100#! neighbor-group local
vsr running neighbor 169.254.0.100#! .. .. .. ..
vsr running config#! routing ipv4-access-list local seq 1 permit 169.254.0.0/24
vsr running config#! routing route-map nolocal
vsr running route-map nolocal#! seq 1 policy deny
vsr running route-map nolocal# seq 1 match ip address access-list local
vsr running route-map nolocal# seq 2 policy permit
vsr running route-map nolocal#

In addition to establishing 2 eBGP sessions with external devices, an internal BGP peering is established between 3 VRFs. The below dump gives an overview of the BGP sessions from the admin VRF perspective.

rt1> show bgp vrf admin summary

IPv4 Unicast Summary:
BGP router identifier 192.168.2.1, local AS number 12 vrf-id 2
BGP table version 3
RIB entries 5, using 920 bytes of memory
Peers 3, using 61 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
169.254.0.101   4      65501        10        18        0    0    0 00:04:30            1        3 N/A
169.254.0.102   4      65502        10        18        0    0    0 00:04:30            1        3 N/A
192.168.2.2     4         13        23        23        0    0    0 00:18:03            0        3 N/A

Note that the above example illustrates 3 eBGP connections with external devices, but also internally with peers on other VRFs. Because /32 bit routes have been used to connect to remote veth peers, routes learned via those peers would not have been installed locally because BGP checks against the recursivity for the eBGP connections. This option has been relaxed, by using the ebgp-connected-route-check command.

To distinguish the external peering from the local peering, the local-as option has been used to replace the original AS number with a private AS number to show to peers from veth links. A side effect of this option is that the AS-PATH list is made up of public and private ASes. To remove private ASes and keep only public ASes, the following command is used by BGP before sending advertisements to external devices: as-outbound-update action remove all.

The below dump gives a RIB extract of the configured device and of the remote device connected to the admin VRF. As you can see, on the remote BGP, the private ASes have been removed from the AS-PATH list.

rt1> show bgp vrf admin ipv4
BGP table version is 3, local router ID is 192.168.2.1, vrf id 2
Default local pref 100, local AS 12
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.1.2.0/24      169.254.0.101            0             0 65501 ?
*> 10.2.2.0/24      169.254.0.102            0             0 65502 ?
*> 20.1.1.0/24      169.254.0.101                          0 65501 13 i
*> 192.168.2.0/24   0.0.0.0                  0         32768 ?

Displayed  3 routes and 3 total paths

rt2> show bgp ipv4
BGP table version is 3, local router ID is 192.168.2.2, vrf id 0
Default local pref 100, local AS 13
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

  Network          Next Hop            Metric LocPrf Weight Path
*> 10.1.2.0/24      192.168.2.1                            0 12 ?
*> 10.2.2.0/24      192.168.2.1                            0 12 ?
*> 20.1.1.0/24      192.168.2.1                            0 12 13 i
*> 192.168.2.0/24   192.168.2.1              0             0 12 ?

Another method to get rid of private ASes would consist in filtering the AS in the incoming AS-PATH list on each local BGP peering.

vsr running config# vrf admin
vsr running vrf admin# routing bgp
vsr running bgp# neighbor 192.168.2.2 address-family ipv4-unicast
vsr running ipv4-unicast# del as-outbound-update
vsr running ipv4-unicast# .. .. ..
vsr running bgp# neighbor-group local
vsr running neighbor-group local# address-family ipv4-unicast
vsr running ipv4-unicast# route-map in route-map-name alteraspath
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! .. .. ..
vsr running config#! vrf service1
vsr running vrf service1#! routing bgp
vsr running bgp#! neighbor 10.1.2.2 address-family ipv4-unicast
vsr running ipv4-unicast#! del as-outbound-update
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! neighbor-group local
vsr running neighbor-group local#! address-family ipv4-unicast
vsr running ipv4-unicast#! route-map in route-map-name alteraspath
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! .. .. ..
vsr running ipv4-unicast#! .. .. ..
vsr running config#! vrf service2
vsr running vrf service2#! routing bgp
vsr running bgp#! neighbor 10.2.2.2 address-family ipv4-unicast
vsr running ipv4-unicast#! del as-outbound-update
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! neighbor-group local
vsr running neighbor-group local#! address-family ipv4-unicast
vsr running ipv4-unicast#! route-map in route-map-name alteraspath
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! .. .. ..
vsr running config#! routing route-map alteraspath
vsr running route-map alteraspath#! seq 100 policy permit
vsr running route-map alteraspath# seq 100 match peer 169.254.0.100
vsr running route-map alteraspath# seq 100 set as-path ignore asn 65500
vsr running route-map alteraspath# seq 101 policy permit
vsr running route-map alteraspath# seq 101 match peer 169.254.0.101
vsr running route-map alteraspath# seq 101 set as-path ignore asn 65501
vsr running route-map alteraspath# seq 102 policy permit
vsr running route-map alteraspath# seq 102 match peer 169.254.0.102
vsr running route-map alteraspath# seq 102 set as-path ignore asn 64502
vsr running route-map alteraspath#

Using a full iBGP network is also another solution to get rid of AS-PATH list handling. In that case, the nexthop-self attribute may be used under each peer address family because the mesh will be incomplete. For instance, the peers behind the admin VRF would not be connected to the peer behind the service1 VRF.

Finally, as the veth link is an ethernet medium, any other routing protocol can be used to transmit routing information to the remote endpoints.

BGP separate instances in L3VRFs

Using L3VRF BGP instances helps to reach good scalability (each instance is smaller, uses less memory and CPU than a VRF instance), but the isolation provided is light (each L3VRF instance of a VRF is part of the same BGP process). The solution guarantees isolation between the configured BGP instances and allows leaking traffic as well. A typical use case is when Virtual Service Router acts as a PE, where VPN routes are redistributed to CPEs.

See also

Overview

It is possible to create multiple BGP instances on the same machine, within a single VRF. The below configuration illustrates how to configure the customer1 instance. You can note that the default BGP instance located at the default L3VRF of the current VRF is present, and even if this instance does nothing, it is necessary to configure it before configuring nested BGP instances in L3VRFs.

The following configuration creates two L3VRF instances named customer1 and customer2.

vsr running config# / vrf main l3vrf customer1 table-id 10
vsr running config# / vrf main l3vrf customer1 routing bgp network-import-check false
vsr running config#! / vrf main l3vrf customer1 routing bgp router-id 1.1.1.1
vsr running config#! / vrf main l3vrf customer1 routing bgp address-family ipv4-unicast network 3.3.3.0/24
vsr running network 3.3.3.0/24#! / vrf main l3vrf customer1 routing bgp address-family ipv4-unicast network 4.4.4.0/24
vsr running network 4.4.4.0/24#! / vrf main l3vrf customer2 table-id 20
vsr running network 4.4.4.0/24#! / vrf main l3vrf customer2 routing bgp router-id 2.2.2.2
vsr running network 4.4.4.0/24#! / vrf main interface physical eth1 port pci-b0s4
vsr running network 4.4.4.0/24#! / vrf main interface physical eth2 port pci-b0s5
vsr running network 4.4.4.0/24#! / vrf main routing bgp as 65500
vsr running network 4.4.4.0/24# / vrf main routing bgp router-id 192.168.0.162

Like for VRF, the BGP peering and the redistribution will happen on the whole L3VRF. All the configured interfaces with addresses, and routing information on that L3VRF will be used.

To get routing information about BGP in that L3VRF instance, the following command will dump all the instances configured.

vsr> show bgp l3vrfs
Type  Id     routerId          #PeersCfg  #PeersEstb  Name
             L3-VNI            RouterMAC              Interface
DFLT  0      192.168.0.162     0          0           default
             0                 00:00:00:00:00:00      unknown
 VRF  17     1.1.1.1           0          0           customer1
             0                 00:00:00:00:00:00      unknown
 VRF  18     2.2.2.2           0          0           customer2
             0                 00:00:00:00:00:00      unknown

Total number of VRFs (including default): 3

To get more information on a specific L3VRF, the following command will dump routing information of the given L3VRF:

vsr> show bgp l3vrf customer1 ipv4
BGP table version is 2, local router ID is 1.1.1.1, vrf id 17
Default local pref 100, local AS 65500
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 3.3.3.0/24       0.0.0.0                  0         32768 i
*> 4.4.4.0/24       0.0.0.0                  0         32768 i

Displayed  2 routes and 2 total paths

The below command disables a specific BGP instance:

vsr running config# / vrf main l3vrf customer1 routing bgp enabled false

Per customer BGP peering use case

This use case assigns a dedicated L3VRF per customer: a BGP instance is created in each L3VRF to peer with its own customer. The below example illustrates 2 BGP instances, each one running on a separate L3VRF, over a dedicated VLAN interface with its own peer. The BGP contexts in a L3VRF share the same system process but will not share the same forwarding information.

vsr running config# / vrf main interface physical eth1 port pci-b0s4
vsr running config# / vrf main routing bgp as 65500
vsr running config# / vrf main l3vrf customer1 table-id 10
vsr running config# / vrf main l3vrf customer1 interface vlan vlan10 ipv4 address 192.168.1.1/24
vsr running config#! / vrf main l3vrf customer1 interface vlan vlan10 vlan-id 10
vsr running config#! / vrf main l3vrf customer1 interface vlan vlan10 link-interface eth1
vsr running config# / vrf main l3vrf customer1 routing bgp router-id 192.168.1.1
vsr running config# / vrf main l3vrf customer1 routing bgp neighbor 192.168.1.2 remote-as 65500
vsr running config# / vrf main l3vrf customer2 table-id 20
vsr running config# / vrf main l3vrf customer2 interface vlan vlan20 ipv4 address 192.168.2.1/24
vsr running config#! / vrf main l3vrf customer2 interface vlan vlan20 vlan-id 20
vsr running config#! / vrf main l3vrf customer2 interface vlan vlan20 link-interface eth1
vsr running config# / vrf main l3vrf customer2 routing bgp as 65501
vsr running config# / vrf main l3vrf customer2 routing bgp router-id 192.168.2.1
vsr running config# / vrf main l3vrf customer2 routing bgp neighbor 192.168.2.2 remote-as 65501

VPN use case

VPNs take advantage of L3VRFs. BGP helps to interconnect VPNs between several sites; L3VPN and EVPN are two of those services implemented by BGP.

See also