BGP in virtual routers¶
Multiple BGP instances can be hosted in Virtual Service Router. There are two ways of separating those instances:
using VRFs, when strong isolation is needed between each routing instance. BGP per VRF instances are mainly used for connecting to the backbone network.
using L3VRFs, when light and scalable isolation is needed. When acting as PE, BGP redistributes VPN routes to CPEs connected to L3VRFs.
BGP configuration, and monitoring in VRF¶
A BGP router can be instantiated in a separate VRF. The routes learned and configured will be stored in the corresponding forwarding tables. It is possible to create multiple BGP instances on the same machine.
Create a BGP instance in a VRF named
customer1, by using the following command:
vsr running config# vrf customer1
vsr running vrf customer1# routing bgp
vsr running bgp#! as 65500
vsr running bgp#
[..]
Then you can continue the configuration as usual. The BGP peering and the redistribution will happen on the whole VRF. The interfaces, addresses and routing information of this VRF will be used. To get routing information about BGP in that VRF instance, you can use the following command, which will dump all the instances configured.
vsr> show bgp vrf customer1 l3vrfs
Type Id routerId #PeersCfg #PeersEstb Name
L3-VNI RouterMAC Interface
DFLT 0 0.0.0.0 1 0 default
0 00:00:00:00:00:00 unknown
To get more information on a specific VRF, you can use the following command, and the VRF name to get routing information:
vsr> show bgp vrf customer1 ipv4 unicast
BGP table version is 2, local router ID is 1.1.1.1, vrf id 2
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 3.3.3.0/24 0.0.0.0 0 32768 i
*> 4.4.4.0/24 0.0.0.0 0 32768 i
Displayed 2 routes and 2 total paths
BGP use case for VRF¶
A common use case is to provide a per customer BGP peering. In this case, one machine is used to provide the customer’s access. Each customer has its own network interface (VLAN, physical, etc) located in its dedicated VRF. The following configuration explains how to create multiple BGP instances attached with VLAN interface.
In the below example, 2 instances of BGP are created, each one running over a VLAN interface with its own peer. The same autonomous system can be used for all instances. The BGP contexts don’t share the same system process and don’t share the same forwarding information.
vsr running config# vrf main
vsr running vrf main# interface physical eth0_0
vsr running physical eth0_0#! port pci-b0s5
vsr running physical eth0_0# .. ..
vsr running vrf main# ..
vsr running config# vrf customer1
vsr running vrf customer1# routing bgp
vsr running bgp#! as 65555
vsr running bgp# router-id 192.168.1.1
vsr running bgp# neighbor 192.168.1.2 remote-as 65555
vsr running bgp# .. ..
vsr running vrf customer1# interface vlan vlan10
vsr running vlan vlan10#! vlan-id 10
vsr running vlan vlan10#! link-interface eth0_0
vsr running vlan vlan10# link-vrf main
vsr running vlan vlan10# ipv4 address 192.168.1.1/24
vsr running vlan vlan10# .. ..
vsr running vrf customer1# ..
vsr running config# vrf customer2
vsr running vrf customer2# routing bgp
vsr running bgp#! as 65555
vsr running bgp# router-id 192.168.2.1
vsr running bgp# neighbor 192.168.2.2 remote-as 65555
vsr running bgp# .. ..
vsr running vrf customer2# interface vlan vlan20
vsr running vlan vlan20#! vlan-id 20
vsr running vlan vlan20#! link-interface eth0_0
vsr running vlan vlan20# link-vrf main
vsr running vlan vlan20# ipv4 address 192.168.2.1/24
vsr running vlan vlan20# .. ..
vsr running vrf customer2#
Note
With the above example, it could have been possible to use the same IP mapping for both routing entities. Another benefit of having separate entities is that IP mapping can overlap.
BGP use case for cross-VRF routes¶
Another common use case is to install cross-VRF routes between 2 BGP instances.
The recommended way to do it consists in setting up a mesh of BGP sessions via
veth links. The below example illustrates what can be done to install cross-VRF
routes between an admin vrf and a service vrf.
vsr running config# vrf admin
vsr running vrf admin# interface physical eth0
vsr running physical eth0#! port pci-b0s5
vsr running physical eth0# ipv4 address 192.168.2.1/24
vsr running physical eth0# ..
vsr running interface# veth service1
vsr running veth service1#! link-interface admin
vsr running veth service1#! link-vrf service1
vsr running veth service1#! ..
vsr running interface#! veth service2
vsr running veth service2#! link-interface admin
vsr running veth service2#! link-vrf service2
vsr running veth service2#! ..
vsr running interface#! loopback localip
vsr running loopback localip#! ipv4 address 169.254.0.100/32
vsr running vrf admin#! routing static ipv4-route 169.254.0.101/32 next-hop service1
vsr running vrf admin#! routing static ipv4-route 169.254.0.102/32 next-hop service2
vsr running vrf admin#! routing bgp
vsr running bgp#! as 12
vsr running bgp#! ebgp-connected-route-check false
vsr running bgp#! router-id 192.168.2.1
vsr running bgp#! address-family ipv4-unicast redistribute connected route-map nolocal
vsr running bgp#! neighbor 192.168.2.2
vsr running neighbor 192.168.2.2#! remote-as 13
vsr running neighbor 192.168.2.2#! address-family ipv4-unicast as-outboun-update
vsr running as-outbound-update#! action remove as-type all
vsr running as-outbound-update#! .. .. .. ..
vsr running bgp#! neighbor-group local update-source localip
vsr running bgp#! neighbor 169.254.0.101
vsr running neighbor 169.254.0.101#! neighbor-group local
vsr running neighbor 169.254.0.101#! remote-as 65501
vsr running neighbor 169.254.0.101#! local-as as-number 65500 no-prepend true replace-as true
vsr running neighbor 169.254.0.101#! neighbor-group local
vsr running neighbor 169.254.0.101#! ..
vsr running bgp#! neighbor 169.254.0.102
vsr running neighbor 169.254.0.102#! remote-as 65502
vsr running neighbor 169.254.0.102#! local-as as-number 65500 no-prepend true replace-as true
vsr running neighbor 169.254.0.102#! neighbor-group local
vsr running neighbor 169.254.0.102#! ..
vsr running bgp#! .. .. ..
vsr running config#! vrf service1
vsr running vrf service1#! interface physical eth1
vsr running physical eth1#! port pci-b0s6
vsr running physical eth1#! ipv4 address 10.1.2.1/24
vsr running physical eth1#! ..
vsr running interface#! veth admin
vsr running veth admin#! link-interface service1
vsr running veth admin#! link-vrf admin
vsr running veth admin#!
vsr running interface#! loopback localip
vsr running loopback localip#! ipv4 address 169.254.0.101/32
vsr running loopback localip#! .. ..
vsr running vrf service1#! routing static ipv4-route 169.254.0.100/32 next-hop admin
vsr running vrf service1#! routing bgp
vsr running bgp#! as 12
vsr running bgp#! ebgp-connected-route-check false
vsr running bgp#! router-id 10.1.2.1
vsr running bgp#! address-family ipv4-unicast redistribute connected route-map nolocal
vsr running bgp#! neighbor 10.1.2.2
vsr running neighbor 10.1.2.2#! remote-as 13
vsr running neighbor 10.1.2.2#! address-family ipv4-unicast as-outbound-update
vsr running as-outbound-update#! action remove as-type all
vsr running as-outbound-update#! .. .. .. ..
vsr running bgp#! neighbor-group local update-source localip
vsr running bgp#! neighbor 169.254.0.100
vsr running neighbor 169.254.0.100#! remote-as 65500
vsr running neighbor 169.254.0.100#! local-as as-number 65501 no-prepend true replace-as true
vsr running neighbor 169.254.0.100#! neighbor-group local
vsr running neighbor 169.254.0.100#! .. .. .. ..
vsr running config#! vrf service2
vsr running vrf service2#! interface physical eth2
vsr running physical eth2#! port pci-b0s6
vsr running physical eth2#! ipv4 address ipv4 address 10.2.2.1/24
vsr running physical eth2#! ..
vsr running interface#! veth admin
vsr running veth admin#! link-interface service2
vsr running veth admin#! link-vrf admin
vsr running veth admin#!
vsr running interface#! loopback localip
vsr running loopback localip#! ipv4 address 169.254.0.102/32
vsr running loopback localip#! .. ..
vsr running vrf service2#! routing static ipv4-route 169.254.0.100/32 next-hop admin
vsr running vrf service2#! routing bgp
vsr running bgp#! as 12
vsr running bgp#! ebgp-connected-route-check false
vsr running bgp#! router-id 10.2.2.1
vsr running bgp#! address-family ipv4-unicast redistribute connected route-map nolocal
vsr running bgp#! neighbor 10.2.2.2
vsr running neighbor 10.2.2.2#! remote-as 13
vsr running neighbor 10.2.2.2#! address-family ipv4-unicast as-outbound-update
vsr running as-outbound-update#! action remove as-type all
vsr running as-outbound-update#! .. .. .. ..
vsr running bgp#! neighbor-group local update-source localip
vsr running bgp#! neighbor 169.254.0.100
vsr running neighbor 169.254.0.100#! remote-as 65500
vsr running neighbor 169.254.0.100#! local-as as-number 65502 no-prepend true replace-as true
vsr running neighbor 169.254.0.100#! neighbor-group local
vsr running neighbor 169.254.0.100#! .. .. .. ..
vsr running config#! routing ipv4-access-list local seq 1 permit 169.254.0.0/24
vsr running config#! routing route-map nolocal
vsr running route-map nolocal#! seq 1 policy deny
vsr running route-map nolocal# seq 1 match ip address access-list local
vsr running route-map nolocal# seq 2 policy permit
vsr running route-map nolocal#
In addition to establishing 2 eBGP sessions with external devices, an internal
BGP peering is established between 3 VRFs. The below dump gives an overview
of the BGP sessions from the admin VRF perspective.
rt1> show bgp vrf admin summary
IPv4 Unicast Summary:
BGP router identifier 192.168.2.1, local AS number 12 vrf-id 2
BGP table version 3
RIB entries 5, using 920 bytes of memory
Peers 3, using 61 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
169.254.0.101 4 65501 10 18 0 0 0 00:04:30 1 3 N/A
169.254.0.102 4 65502 10 18 0 0 0 00:04:30 1 3 N/A
192.168.2.2 4 13 23 23 0 0 0 00:18:03 0 3 N/A
Note that the above example illustrates 3 eBGP connections with external devices,
but also internally with peers on other VRFs. Because /32 bit routes have been
used to connect to remote veth peers, routes learned via those peers would not have
been installed locally because BGP checks against the recursivity for the eBGP
connections. This option has been relaxed, by using the ebgp-connected-route-check
command.
To distinguish the external peering from the local peering, the local-as option has
been used to replace the original AS number with a private AS number to show to peers
from veth links. A side effect of this option is that the AS-PATH list is made up of
public and private ASes. To remove private ASes and keep only public ASes, the
following command is used by BGP before sending advertisements to external devices:
as-outbound-update action remove all.
The below dump gives a RIB extract of the configured device and of the remote device
connected to the admin VRF. As you can see, on the remote BGP, the private ASes
have been removed from the AS-PATH list.
rt1> show bgp vrf admin ipv4
BGP table version is 3, local router ID is 192.168.2.1, vrf id 2
Default local pref 100, local AS 12
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 10.1.2.0/24 169.254.0.101 0 0 65501 ?
*> 10.2.2.0/24 169.254.0.102 0 0 65502 ?
*> 20.1.1.0/24 169.254.0.101 0 65501 13 i
*> 192.168.2.0/24 0.0.0.0 0 32768 ?
Displayed 3 routes and 3 total paths
rt2> show bgp ipv4
BGP table version is 3, local router ID is 192.168.2.2, vrf id 0
Default local pref 100, local AS 13
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 10.1.2.0/24 192.168.2.1 0 12 ?
*> 10.2.2.0/24 192.168.2.1 0 12 ?
*> 20.1.1.0/24 192.168.2.1 0 12 13 i
*> 192.168.2.0/24 192.168.2.1 0 0 12 ?
Another method to get rid of private ASes would consist in filtering the AS in the incoming AS-PATH list on each local BGP peering.
vsr running config# vrf admin
vsr running vrf admin# routing bgp
vsr running bgp# neighbor 192.168.2.2 address-family ipv4-unicast
vsr running ipv4-unicast# del as-outbound-update
vsr running ipv4-unicast# .. .. ..
vsr running bgp# neighbor-group local
vsr running neighbor-group local# address-family ipv4-unicast
vsr running ipv4-unicast# route-map in route-map-name alteraspath
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! .. .. ..
vsr running config#! vrf service1
vsr running vrf service1#! routing bgp
vsr running bgp#! neighbor 10.1.2.2 address-family ipv4-unicast
vsr running ipv4-unicast#! del as-outbound-update
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! neighbor-group local
vsr running neighbor-group local#! address-family ipv4-unicast
vsr running ipv4-unicast#! route-map in route-map-name alteraspath
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! .. .. ..
vsr running ipv4-unicast#! .. .. ..
vsr running config#! vrf service2
vsr running vrf service2#! routing bgp
vsr running bgp#! neighbor 10.2.2.2 address-family ipv4-unicast
vsr running ipv4-unicast#! del as-outbound-update
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! neighbor-group local
vsr running neighbor-group local#! address-family ipv4-unicast
vsr running ipv4-unicast#! route-map in route-map-name alteraspath
vsr running ipv4-unicast#! .. .. ..
vsr running bgp#! .. .. ..
vsr running config#! routing route-map alteraspath
vsr running route-map alteraspath#! seq 100 policy permit
vsr running route-map alteraspath# seq 100 match peer 169.254.0.100
vsr running route-map alteraspath# seq 100 set as-path ignore asn 65500
vsr running route-map alteraspath# seq 101 policy permit
vsr running route-map alteraspath# seq 101 match peer 169.254.0.101
vsr running route-map alteraspath# seq 101 set as-path ignore asn 65501
vsr running route-map alteraspath# seq 102 policy permit
vsr running route-map alteraspath# seq 102 match peer 169.254.0.102
vsr running route-map alteraspath# seq 102 set as-path ignore asn 64502
vsr running route-map alteraspath#
Using a full iBGP network is also another solution to get rid of AS-PATH list
handling. In that case, the nexthop-self attribute may be used under each peer
address family because the mesh will be incomplete. For instance, the peers
behind the admin VRF would not be connected to the peer behind the service1
VRF.
Finally, as the veth link is an ethernet medium, any other routing protocol can
be used to transmit routing information to the remote endpoints.
BGP separate instances in L3VRFs¶
Using L3VRF BGP instances helps to reach good scalability (each instance is smaller, uses less memory and CPU than a VRF instance), but the isolation provided is light (each L3VRF instance of a VRF is part of the same BGP process). The solution guarantees isolation between the configured BGP instances and allows leaking traffic as well. A typical use case is when Virtual Service Router acts as a PE, where VPN routes are redistributed to CPEs.
See also
Overview¶
It is possible to create multiple BGP instances on the same machine, within a
single VRF. The below configuration illustrates how to configure the customer1
instance. You can note that the default BGP instance located at the default
L3VRF of the current VRF is present, and even if this instance does nothing, it
is necessary to configure it before configuring nested BGP instances in L3VRFs.
The following configuration creates two L3VRF instances named customer1 and
customer2.
vsr running config# vrf main
vsr running vrf main# routing bgp
vsr running bgp#! as 65500
vsr running bgp# router-id 192.168.0.162
vsr running bgp# .. ..
vsr running vrf main# l3vrf customer1
vsr running l3vrf customer1#! table-id 10
vsr running l3vrf customer1# # add customer1 routing config
vsr running l3vrf customer1# routing bgp
vsr running bgp# as 65500
vsr running bgp# router-id 192.168.0.162
vsr running bgp# router-id 1.1.1.1
vsr running bgp# address-family ipv4-unicast
vsr running ipv4-unicast# network 3.3.3.0/24
vsr running network 3.3.3.0/24# ..
vsr running ipv4-unicast#! network 4.4.4.0/24
vsr running network 4.4.4.0/24# ..
vsr running ipv4-unicast# .. ..
vsr running bgp# ..
vsr running routing# ..
vsr running l3vrf customer1# ..
vsr running vrf main# l3vrf customer2
vsr running l3vrf customer2#! table-id 20
vsr running l3vrf customer2# # add customer2 routing config
vsr running l3vrf customer2# routing bgp
vsr running bgp# as 65500
vsr running bgp# router-id 2.2.2.2
vsr running bgp# .. .. ..
vsr running vrf main#
Like for VRF, the BGP peering and the redistribution will happen on the whole L3VRF. All the configured interfaces with addresses, and routing information on that L3VRF will be used.
To get routing information about BGP in that L3VRF instance, the following command will dump all the instances configured.
vsr> show bgp l3vrfs
Type Id routerId #PeersCfg #PeersEstb Name
L3-VNI RouterMAC Interface
DFLT 0 192.168.0.162 0 0 default
0 00:00:00:00:00:00 unknown
VRF 6 1.1.1.1 0 0 customer1
0 00:00:00:00:00:00 unknown
VRF 7 2.2.2.2 0 0 customer2
0 00:00:00:00:00:00 unknown
Total number of VRFs (including default): 3
To get more information on a specific L3VRF, the following command will dump routing information of the given L3VRF:
vsr> show bgp l3vrf customer1 ipv4
BGP table version is 2, local router ID is 1.1.1.1, vrf id 6
Default local pref 100, local AS 65500
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 3.3.3.0/24 0.0.0.0 0 32768 i
*> 4.4.4.0/24 0.0.0.0 0 32768 i
Displayed 2 routes and 2 total paths
The below command disables a specific BGP instance:
vsr running config# vrf main
vsr running vrf main# l3vrf customer1 routing bgp
vsr running bgp# enabled false
vsr running bgp#
Per customer BGP peering use case¶
This use case assigns a dedicated L3VRF per customer: a BGP instance is created in each L3VRF to peer with its own customer. The below example illustrates 2 BGP instances, each one running on a separate L3VRF, over a dedicated VLAN interface with its own peer. The BGP contexts in a L3VRF share the same system process but will not share the same forwarding information.
vsr running config# vrf main
vsr running vrf main# interface physical eth0
vsr running physical eth0#! port pci-b0s6
vsr running physical eth0# ..
vsr running interface# ..
vsr running vrf main# routing bgp
vsr running bgp#! as 65500
vsr running bgp# ..
vsr running routing# ..
vsr running vrf main# l3vrf customer1
vsr running l3vrf customer1#! table-id 10
vsr running l3vrf customer1# routing bgp
vsr running bgp# router-id 192.168.1.1
vsr running bgp# neighbor 192.168.1.2 remote-as 65500
vsr running bgp# .. ..
vsr running l3vrf customer1# interface vlan vlan10
vsr running vlan vlan10#! vlan-id 10
vsr running vlan vlan10#! link-interface eth0
vsr running vlan vlan10# ipv4 address 192.168.1.1/24
vsr running vlan vlan10# .. ..
vsr running l3vrf customer1# ..
vsr running vrf main# l3vrf customer2
vsr running l3vrf customer2#! table-id 20
vsr running l3vrf customer2# interface vlan vlan20
vsr running vlan vlan20#! vlan-id 20
vsr running vlan vlan20#! link-interface eth0
vsr running vlan vlan20# ipv4 address 192.168.2.1/24
vsr running vlan vlan20# .. ..
vsr running l3vrf customer2# routing bgp
vsr running bgp# as 65501
vsr running bgp# router-id 192.168.2.1
vsr running bgp# neighbor 192.168.2.2 remote-as 65501
vsr running bgp#
VPN use case¶
VPNs take advantage of L3VRFs. BGP helps to interconnect VPNs between several sites; L3VPN and EVPN are two of those services implemented by BGP.
See also
BGP VPN services, see Connecting VPN with BGP.
Using EVPN technology with BGP and L3VRF, see BGP EVPN.
Using L3VPN technology with BGP and L3VRF, see BGP L3VPN.