Goal
Goal is to build a multi-tenancy compatible EVPN setup with the ability to route between L2 domains and the possibility of external communications.
Assumptions
-
Linux-1, Linux-2, Linux-3 are in the same broadcast domain, VNI 10000
-
Linux-4, Linux-5 are in the same broadcast domain, VNI 20000
-
Linux-1, Linux-2, Linux-3, Linux-4, Linux-5 belong to the same tenant 'tenant-1'
-
L3 routing is allowed within same tenant
-
Connectivity to external nentworks is provided
Prerequirements
Cisco Nexus router have to have some feutures enabled:
nv overlay evpn feature ospf feature bgp feature interface-vlan feature vn-segment-vlan-based feature nv overlay
Underlay
OSPFv2 is used for the underlay, all routers are in the area 0.0.0.0 I.e.:
router ospf main_ospf router-id 192.168.55.3 passive-interface default
interfaces:
interface loopback0 ip address 192.168.55.3/32 ip router ospf main_ospf area 0.0.0.0 icam monitor scale interface Ethernet1/1 description to_r1 no switchport mtu 9202 ip address 10.0.13.2/24 no ip ospf passive-interface ip router ospf main_ospf area 0.0.0.0 no shutdown interface Ethernet1/2 description to_r2 no switchport mtu 9202 ip address 10.0.23.2/24 no ip ospf passive-interface ip router ospf main_ospf area 0.0.0.0 no shutdown
Overlay
BGP overlay uses iBGP. BGP sessions are established between loopbacks. Spines r1 and r2 are route-reflectors. Leaf sw1
router bgp 65000
router-id 192.168.55.1
address-family l2vpn evpn
neighbor 192.168.99.1
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 192.168.99.2
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
Spine r1
router bgp 65000 bgp router-id 192.168.99.1 address-family ipv4 unicast ! address-family ipv6 unicast ! address-family l2vpn evpn ! neighbor-group evpn-group remote-as 65000 update-source Loopback0 ! neighbor 192.168.55.0/24 use neighbor-group evpn-group cluster-id 192.168.99.1 address-family l2vpn evpn route-reflector-client ! ! neighbor 192.168.99.2 remote-as 65000 update-source Loopback0 address-family l2vpn evpn ! ! ! end
VNIs and VLANs
Leaf sw2
vlan 1,10,20,999 vlan 10 vn-segment 10000 vlan 20 vn-segment 20000 vlan 999 vn-segment 999000
Linux-2 is in the VLAN 10, Linux-4 is in the VLAN 20. VLAN 999 is needed for inter VLAN communications.
Interfaces:
vrf context tenant1
vni 999000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:555
route-target import 65000:555 evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:555
route-target import 65000:555 evpn
interface Vlan10
no shutdown
vrf member tenant1
ip address 10.10.10.1/24
fabric forwarding mode anycast-gateway
interface Vlan20
no shutdown
vrf member tenant1
ip address 10.10.20.1/24
fabric forwarding mode anycast-gateway
interface Vlan999
no shutdown
vrf member tenant1
ip forward
ipv6 forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10000
ingress-replication protocol bgp
member vni 20000
ingress-replication protocol bgp
member vni 999000 associate-vrf
interface Ethernet1/8
description to_Linux-4
switchport access vlan 20
interface Ethernet1/9
description to_Linux-2
switchport access vlan 10
vrf context tenant1 is used to route traffic between different broadcast domains. It should be present on every leaf carryin at least one VLAN of the tenant tenant1. anycast-gateway is used to route traffic, thus all leafs should have the same
fabric forwarding anycast-gateway-mac 0010.0010.0010
EVPN vni for L2 domains:
evpn
vni 10000 l2
route-target import auto
route-target export auto
vni 20000 l2
route-target import auto
route-target export auto
route-target import auto and route-target export auto are working since leafs are in the same ASN, for our case RT:65000:10000
and RT:65000:20000.
Community 65000:555 is used to import external routes as type 5.
Routing
Routing between L2 domains is done through type 2 EVPN routes, routing to the external resourses through type 5 EVPN routes.
VNI 999000 and local VLAN 999 are used to route traffic between VNI 10000 and VNI 20000, type 2 EVPN routes are used.
VNI 555 and VLAN 555 are defined on the leaf sw3.
More on external routing
External routing uses VLAN 555 and VNI 555 defined on leat sw3.
Thereafter those routes are imported to L3 VNI on eny leaf where it’s needed through route-target import 65000:555 evpn
vlan 555 vn-segment 555
interface Vlan555 no shutdown vrf member external-shared ip forward ipv6 forward
vrf context external-shared
vni 555
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:999000
route-target import 65000:999000 evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:999000
route-target import 65000:999000 evpn
BGP on leaf sw3 is a bit longer.
router bgp 65000
address-family l2vpn evpn
neighbor 192.168.99.1
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 192.168.99.2
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf external-shared
address-family ipv4 unicast
address-family ipv6 unicast
neighbor 2001::
remote-as 42
address-family ipv6 unicast
neighbor 100.64.64.0
remote-as 42
address-family ipv4 unicast
route-map ADD-RT in
vrf tenant1
address-family ipv4 unicast
redistribute direct route-map ALL
vrf external-shared is to import external routes to the corresponding vrf, and thereafte to vrf context tenant1 or any other vrf where needed.
However for sw3, routes in VLAN10 and VLAN20 are local and thus won’t get to the vrf context external-shared and later on to ISP1 router throut route-target import 65000:999000 means. thus to workaroung that
we need to add vrf tenant1 into the BGP process and redistribute direct.
Compare two outputs:
With vrf tenant1
sw3# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 93, Local Router ID is 192.168.55.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.55.1:32777
* i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[0]:[0.0.0.0]/216
192.168.55.1 100 0 i
*>i 192.168.55.1 100 0 i
* i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[32]:[10.10.10.111]/272
192.168.55.1 100 0 i
*>i 192.168.55.1 100 0 i
*>i[3]:[0]:[32]:[192.168.55.1]/88
192.168.55.1 100 0 i
* i 192.168.55.1 100 0 i
Route Distinguisher: 192.168.55.1:32787
*>i[3]:[0]:[32]:[192.168.55.1]/88
192.168.55.1 100 0 i
* i 192.168.55.1 100 0 i
Route Distinguisher: 192.168.55.2:32777
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[0]:[0.0.0.0]/216
192.168.55.2 100 0 i
* i 192.168.55.2 100 0 i
* i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[32]:[10.10.10.2]/272
192.168.55.2 100 0 i
*>i 192.168.55.2 100 0 i
*>i[3]:[0]:[32]:[192.168.55.2]/88
192.168.55.2 100 0 i
* i 192.168.55.2 100 0 i
Route Distinguisher: 192.168.55.2:32787
* i[2]:[0]:[0]:[48]:[6609.b7b5.b7c4]:[0]:[0.0.0.0]/216
192.168.55.2 100 0 i
*>i 192.168.55.2 100 0 i
*>i[3]:[0]:[32]:[192.168.55.2]/88
192.168.55.2 100 0 i
* i 192.168.55.2 100 0 i
Route Distinguisher: 192.168.55.3:32777 (L2VNI 10000)
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[0]:[0.0.0.0]/216
192.168.55.2 100 0 i
*>l[2]:[0]:[0]:[48]:[5e4f.b35e.9a0c]:[0]:[0.0.0.0]/216
192.168.55.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[0]:[0.0.0.0]/216
192.168.55.1 100 0 i
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[32]:[10.10.10.2]/272
192.168.55.2 100 0 i
*>l[2]:[0]:[0]:[48]:[5e4f.b35e.9a0c]:[32]:[10.10.10.3]/272
192.168.55.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[32]:[10.10.10.111]/272
192.168.55.1 100 0 i
*>i[3]:[0]:[32]:[192.168.55.1]/88
192.168.55.1 100 0 i
*>i[3]:[0]:[32]:[192.168.55.2]/88
192.168.55.2 100 0 i
*>l[3]:[0]:[32]:[192.168.55.3]/88
192.168.55.3 100 32768 i
Route Distinguisher: 192.168.55.3:32787 (L2VNI 20000)
*>l[2]:[0]:[0]:[48]:[523f.4dd8.d542]:[0]:[0.0.0.0]/216
192.168.55.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[6609.b7b5.b7c4]:[0]:[0.0.0.0]/216
192.168.55.2 100 0 i
*>i[3]:[0]:[32]:[192.168.55.1]/88
192.168.55.1 100 0 i
*>i[3]:[0]:[32]:[192.168.55.2]/88
192.168.55.2 100 0 i
*>l[3]:[0]:[32]:[192.168.55.3]/88
192.168.55.3 100 32768 i
Route Distinguisher: 192.168.55.3:3 (L3VNI 555)
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[32]:[10.10.10.2]/272
192.168.55.2 100 0 i
*>i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[32]:[10.10.10.111]/272
192.168.55.1 100 0 i
*>l[5]:[0]:[0]:[8]:[99.0.0.0]/224
192.168.55.3 0 42 i
*>l[5]:[0]:[0]:[22]:[99.0.0.0]/224
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[24]:[99.0.1.0]/224
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[24]:[99.0.2.0]/224
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[24]:[99.0.3.0]/224
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[32]:[2006::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[48]:[2001:1::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[48]:[2001:2::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[48]:[2001:3::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[127]:[2001::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[128]:[2006::2006]/416
192.168.55.3 0 0 42 ?
Route Distinguisher: 192.168.55.3:4 (L3VNI 999000)
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[32]:[10.10.10.2]/272
192.168.55.2 100 0 i
*>i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[32]:[10.10.10.111]/272
192.168.55.1 100 0 i
*>l[5]:[0]:[0]:[24]:[10.10.10.0]/224
192.168.55.3 0 100 32768 ?
*>l[5]:[0]:[0]:[24]:[10.10.20.0]/224
192.168.55.3 0 100 32768 ?
RP/0/RP0/CPU0:isp1#show route
Fri Nov 10 00:01:48.658 UTC
Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
U - per-user static route, o - ODR, L - local, G - DAGR, l - LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, t - Traffic Engineering, (!) - FRR Backup path
Gateway of last resort is not set
S 10.0.0.0/8 [1/0] via 100.64.64.1, 01:38:48
B 10.10.10.0/24 [20/0] via 100.64.64.1, 00:00:06
B 10.10.10.2/32 [20/0] via 100.64.64.1, 00:46:23
B 10.10.10.111/32 [20/0] via 100.64.64.1, 00:52:30
B 10.10.20.0/24 [20/0] via 100.64.64.1, 00:00:06
B 99.0.0.0/8 [200/0] via 0.0.0.0, 01:38:44, Null0
S 99.0.0.0/22 is directly connected, 01:39:30, Null0
C 99.0.1.0/24 is directly connected, 01:39:29, Loopback0
L 99.0.1.1/32 is directly connected, 01:39:29, Loopback0
C 99.0.2.0/24 is directly connected, 01:39:29, Loopback0
L 99.0.2.1/32 is directly connected, 01:39:29, Loopback0
C 99.0.3.0/24 is directly connected, 01:39:29, Loopback0
L 99.0.3.1/32 is directly connected, 01:39:29, Loopback0
C 100.64.64.0/31 is directly connected, 01:38:48, GigabitEthernet0/0/0/0
L 100.64.64.0/32 is directly connected, 01:38:48, GigabitEthernet0/0/0/0
and whout it
sw3# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 95, Local Router ID is 192.168.55.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.55.1:32777
* i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[0]:[0.0.0.0]/216
192.168.55.1 100 0 i
*>i 192.168.55.1 100 0 i
* i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[32]:[10.10.10.111]/272
192.168.55.1 100 0 i
*>i 192.168.55.1 100 0 i
*>i[3]:[0]:[32]:[192.168.55.1]/88
192.168.55.1 100 0 i
* i 192.168.55.1 100 0 i
Route Distinguisher: 192.168.55.1:32787
*>i[3]:[0]:[32]:[192.168.55.1]/88
192.168.55.1 100 0 i
* i 192.168.55.1 100 0 i
Route Distinguisher: 192.168.55.2:32777
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[0]:[0.0.0.0]/216
192.168.55.2 100 0 i
* i 192.168.55.2 100 0 i
* i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[32]:[10.10.10.2]/272
192.168.55.2 100 0 i
*>i 192.168.55.2 100 0 i
*>i[3]:[0]:[32]:[192.168.55.2]/88
192.168.55.2 100 0 i
* i 192.168.55.2 100 0 i
Route Distinguisher: 192.168.55.2:32787
* i[2]:[0]:[0]:[48]:[6609.b7b5.b7c4]:[0]:[0.0.0.0]/216
192.168.55.2 100 0 i
*>i 192.168.55.2 100 0 i
*>i[3]:[0]:[32]:[192.168.55.2]/88
192.168.55.2 100 0 i
* i 192.168.55.2 100 0 i
Route Distinguisher: 192.168.55.3:32777 (L2VNI 10000)
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[0]:[0.0.0.0]/216
192.168.55.2 100 0 i
*>l[2]:[0]:[0]:[48]:[5e4f.b35e.9a0c]:[0]:[0.0.0.0]/216
192.168.55.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[0]:[0.0.0.0]/216
192.168.55.1 100 0 i
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[32]:[10.10.10.2]/272
192.168.55.2 100 0 i
*>l[2]:[0]:[0]:[48]:[5e4f.b35e.9a0c]:[32]:[10.10.10.3]/272
192.168.55.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[32]:[10.10.10.111]/272
192.168.55.1 100 0 i
*>i[3]:[0]:[32]:[192.168.55.1]/88
192.168.55.1 100 0 i
*>i[3]:[0]:[32]:[192.168.55.2]/88
192.168.55.2 100 0 i
*>l[3]:[0]:[32]:[192.168.55.3]/88
192.168.55.3 100 32768 i
Route Distinguisher: 192.168.55.3:32787 (L2VNI 20000)
*>l[2]:[0]:[0]:[48]:[523f.4dd8.d542]:[0]:[0.0.0.0]/216
192.168.55.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[6609.b7b5.b7c4]:[0]:[0.0.0.0]/216
192.168.55.2 100 0 i
*>i[3]:[0]:[32]:[192.168.55.1]/88
192.168.55.1 100 0 i
*>i[3]:[0]:[32]:[192.168.55.2]/88
192.168.55.2 100 0 i
*>l[3]:[0]:[32]:[192.168.55.3]/88
192.168.55.3 100 32768 i
Route Distinguisher: 192.168.55.3:3 (L3VNI 555)
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[32]:[10.10.10.2]/272
192.168.55.2 100 0 i
*>i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[32]:[10.10.10.111]/272
192.168.55.1 100 0 i
*>l[5]:[0]:[0]:[8]:[99.0.0.0]/224
192.168.55.3 0 42 i
*>l[5]:[0]:[0]:[22]:[99.0.0.0]/224
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[24]:[99.0.1.0]/224
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[24]:[99.0.2.0]/224
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[24]:[99.0.3.0]/224
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[32]:[2006::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[48]:[2001:1::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[48]:[2001:2::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[48]:[2001:3::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[127]:[2001::]/416
192.168.55.3 0 0 42 ?
*>l[5]:[0]:[0]:[128]:[2006::2006]/416
192.168.55.3 0 0 42 ?
Route Distinguisher: 192.168.55.3:4 (L3VNI 999000)
*>i[2]:[0]:[0]:[48]:[4e33.5b83.835b]:[32]:[10.10.10.2]/272
192.168.55.2 100 0 i
*>i[2]:[0]:[0]:[48]:[6e43.b74d.3712]:[32]:[10.10.10.111]/272
192.168.55.1 100 0 i
RP/0/RP0/CPU0:isp1#show route
Fri Nov 10 00:01:00.516 UTC
Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
U - per-user static route, o - ODR, L - local, G - DAGR, l - LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, t - Traffic Engineering, (!) - FRR Backup path
Gateway of last resort is not set
S 10.0.0.0/8 [1/0] via 100.64.64.1, 01:38:00
B 10.10.10.2/32 [20/0] via 100.64.64.1, 00:45:35
B 10.10.10.111/32 [20/0] via 100.64.64.1, 00:51:42
B 99.0.0.0/8 [200/0] via 0.0.0.0, 01:37:56, Null0
S 99.0.0.0/22 is directly connected, 01:38:42, Null0
C 99.0.1.0/24 is directly connected, 01:38:41, Loopback0
L 99.0.1.1/32 is directly connected, 01:38:41, Loopback0
C 99.0.2.0/24 is directly connected, 01:38:41, Loopback0
L 99.0.2.1/32 is directly connected, 01:38:41, Loopback0
C 99.0.3.0/24 is directly connected, 01:38:41, Loopback0
L 99.0.3.1/32 is directly connected, 01:38:41, Loopback0
C 100.64.64.0/31 is directly connected, 01:38:00, GigabitEthernet0/0/0/0
L 100.64.64.0/32 is directly connected, 01:38:00, GigabitEthernet0/0/0/0
Basically vrf tenant1 advetises whole subnet towards ISP1 and not individual type 2 IPs like for leafs sw1 and sw2, but that the acceptable workaroung.
Verification
Packet capture showing inter VLAN/VNI communication through VNI 999000
Frame 44587: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface -, id 0
Ethernet II, Src: 0c:c7:00:00:1b:08 (0c:c7:00:00:1b:08), Dst: 0c:3a:4f:a4:00:03 (0c:3a:4f:a4:00:03)
Internet Protocol Version 4, Src: 192.168.55.1, Dst: 192.168.55.3
User Datagram Protocol, Src Port: 49872, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
Group Policy ID: 0
VXLAN Network Identifier (VNI): 999000
Reserved: 0
Ethernet II, Src: 0c:c7:00:00:1b:08 (0c:c7:00:00:1b:08), Dst: 0c:36:00:00:1b:08 (0c:36:00:00:1b:08)
Destination: 0c:36:00:00:1b:08 (0c:36:00:00:1b:08)
Source: 0c:c7:00:00:1b:08 (0c:c7:00:00:1b:08)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.10.10.111, Dst: 10.10.20.5
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 84
Identification: 0xd3b1 (54193)
000. .... = Flags: 0x0
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 63
Protocol: ICMP (1)
Header Checksum: 0x7570 [validation disabled]
[Header checksum status: Unverified]
Source Address: 10.10.10.111
Destination Address: 10.10.20.5
Internet Control Message Protocol
Full configs
ISP1
RP/0/RP0/CPU0:isp1#show running-config Fri Nov 10 00:07:46.391 UTC Building configuration... !! IOS XR Configuration 7.3.1 !! Last configuration change at Mon Nov 6 21:28:18 2023 by admin ! hostname isp1 username admin group root-lr group cisco-support secret 10 $6$VftYn/OCtQHC5n/.$2amSbAY5JnckN37abVwr6Xy.e2OCXQ/6kUUIiLYRm/gU14zSpBtXczVR2ya3JaAk18NnWilLpu0ifFocbjCHN/ ! call-home service active contact smart-licensing profile CiscoTAC-1 active destination transport-method http ! ! interface Loopback0 ipv4 address 99.0.1.1 255.255.255.0 ipv4 address 99.0.2.1 255.255.255.0 secondary ipv4 address 99.0.3.1 255.255.255.0 secondary ipv6 address 2001:1::/48 ipv6 address 2001:2::/48 ipv6 address 2001:3::/48 ipv6 address 2006::2006/128 ! interface MgmtEth0/RP0/CPU0/0 shutdown ! interface GigabitEthernet0/0/0/0 description to_sw1 ipv4 address 100.64.64.0 255.255.255.254 ipv6 address 2001::/127 ! interface GigabitEthernet0/0/0/1 shutdown ! interface GigabitEthernet0/0/0/2 shutdown ! interface GigabitEthernet0/0/0/3 shutdown ! route-policy IBGP-FILTER-V4-V6 pass end-policy ! router static address-family ipv4 unicast 10.0.0.0/8 100.64.64.1 99.0.0.0/22 Null0 254 tag 9999 ! address-family ipv6 unicast 2006::/32 Null0 254 tag 9999 ! ! router bgp 42 address-family ipv4 unicast network 99.0.0.0/8 aggregate-address 99.0.0.0/8 redistribute connected redistribute static ! address-family ipv6 unicast network 2000::/3 network 2001::/16 aggregate-address 2006::/32 redistribute connected redistribute static ! neighbor 2001::1 remote-as 65000 address-family ipv6 unicast route-policy IBGP-FILTER-V4-V6 in route-policy IBGP-FILTER-V4-V6 out ! ! neighbor 100.64.64.1 remote-as 65000 address-family ipv4 unicast route-policy IBGP-FILTER-V4-V6 in route-policy IBGP-FILTER-V4-V6 out ! ! ! end
Spine r1
RP/0/RP0/CPU0:r1#show running-config Fri Nov 10 00:10:33.807 UTC Building configuration... !! IOS XR Configuration 7.3.1 !! Last configuration change at Fri Nov 3 20:13:23 2023 by admin ! hostname r1 username admin group root-lr group cisco-support secret 10 $6$duKZI/iJ7JJG3I/.$lFoQpquJREMaXysffr5BfK54RdFNjNvt8YCCj3oPqUSFYJTw0ePQ7RTKws7LleeTESJosd3l5Rd7tZzv.jWv4/ ! call-home service active contact smart-licensing profile CiscoTAC-1 active destination transport-method http ! ! interface Loopback0 ipv4 address 192.168.99.1 255.255.255.255 ! interface MgmtEth0/RP0/CPU0/0 shutdown ! interface GigabitEthernet0/0/0/0 description to_sw1 mtu 9216 ipv4 address 10.0.11.1 255.255.255.0 ! interface GigabitEthernet0/0/0/1 description to_sw2 mtu 9216 ipv4 address 10.0.12.1 255.255.255.0 ! interface GigabitEthernet0/0/0/2 description to_sw3 mtu 9216 ipv4 address 10.0.13.1 255.255.255.0 ! interface GigabitEthernet0/0/0/3 shutdown ! router ospf main_osfp router-id 192.168.99.1 area 0.0.0.0 interface Loopback0 passive enable ! interface GigabitEthernet0/0/0/0 ! interface GigabitEthernet0/0/0/1 ! interface GigabitEthernet0/0/0/2 ! ! ! router bgp 65000 bgp router-id 192.168.99.1 address-family ipv4 unicast ! address-family ipv6 unicast ! address-family l2vpn evpn ! neighbor-group evpn-group remote-as 65000 update-source Loopback0 ! neighbor 192.168.55.0/24 use neighbor-group evpn-group cluster-id 192.168.99.1 address-family l2vpn evpn route-reflector-client ! ! neighbor 192.168.99.2 remote-as 65000 update-source Loopback0 address-family l2vpn evpn ! ! ! end
Spine r2
RP/0/RP0/CPU0:r2#show running-config Fri Nov 10 00:13:34.104 UTC Building configuration... !! IOS XR Configuration 7.3.1 !! Last configuration change at Fri Nov 3 20:13:51 2023 by admin ! hostname r2 username admin group root-lr group cisco-support secret 10 $6$Kq/iWRUf6s80W...$eYGtToQGVNrq3GHeqkIlVyBCCZIu2z7A.DOx8qntM5bCRz9t3RSRhF/MHzYHoHrl72SC3cUPdFKqZhWduuTgd. ! address-family ipv4 unicast ! address-family ipv6 unicast ! call-home service active contact smart-licensing profile CiscoTAC-1 active destination transport-method http ! ! interface Loopback0 ipv4 address 192.168.99.2 255.255.255.255 ! interface MgmtEth0/RP0/CPU0/0 shutdown ! interface GigabitEthernet0/0/0/0 description to_r1 mtu 9216 ipv4 address 10.0.21.1 255.255.255.0 ! interface GigabitEthernet0/0/0/1 description to_r2 mtu 9216 ipv4 address 10.0.22.1 255.255.255.0 ! interface GigabitEthernet0/0/0/2 description to_sw3 mtu 9216 ipv4 address 10.0.23.1 255.255.255.0 ! interface GigabitEthernet0/0/0/3 shutdown ! router ospf main_osfp router-id 192.168.99.2 area 0.0.0.0 interface Loopback0 passive enable ! interface GigabitEthernet0/0/0/0 ! interface GigabitEthernet0/0/0/1 ! interface GigabitEthernet0/0/0/2 ! ! ! router bgp 65000 bgp router-id 192.168.99.2 address-family ipv4 unicast ! address-family ipv6 unicast ! address-family l2vpn evpn ! neighbor-group evpn-group remote-as 65000 update-source Loopback0 ! neighbor 192.168.55.0/24 use neighbor-group evpn-group cluster-id 192.168.99.2 address-family l2vpn evpn route-reflector-client ! ! neighbor 192.168.99.1 remote-as 65000 update-source Loopback0 address-family l2vpn evpn ! ! ! end
Leaf sw1
sw1# show running-config
!Command: show running-config
!Running configuration last done at: Thu Nov 9 23:35:06 2023
!Time: Fri Nov 10 00:11:14 2023
version 10.2(5) Bios:version
switchname sw1
vdc sw1 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
username admin password 5 $5$EPLIOB$VcXBBD0zdD4Pt3YXX2yqBUTM6M3fJwBdJxwrYUfM6D/
role network-admin
ip domain-lookup
copp profile strict
snmp-server user admin network-admin auth md5 3209EFD2820344D265E19A4B9A10ED749A
CB priv aes-128 0040F182CD18429038CCD85C9F56E910C88D localizedV2key
rmon event 1 log trap public description FATAL(1) owner PMON@FATAL
rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 log trap public description ERROR(3) owner PMON@ERROR
rmon event 4 log trap public description WARNING(4) owner PMON@WARNING
rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO
system default switchport
fabric forwarding anycast-gateway-mac 0010.0010.0010
vlan 1,10,20,999
vlan 10
vn-segment 10000
vlan 20
vn-segment 20000
vlan 999
vn-segment 999000
vrf context management
vrf context tenant1
vni 999000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:555
route-target import 65000:555 evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:555
route-target import 65000:555 evpn
interface Vlan1
interface Vlan10
no shutdown
vrf member tenant1
ip address 10.10.10.1/24
fabric forwarding mode anycast-gateway
interface Vlan20
no shutdown
vrf member tenant1
ip address 10.10.20.1/24
fabric forwarding mode anycast-gateway
interface Vlan999
no shutdown
vrf member tenant1
ip forward
ipv6 forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10000
ingress-replication protocol bgp
member vni 20000
ingress-replication protocol bgp
member vni 999000 associate-vrf
interface Ethernet1/1
description to_r1
no switchport
mtu 9202
ip address 10.0.11.2/24
no ip ospf passive-interface
ip router ospf main_ospf area 0.0.0.0
no shutdown
interface Ethernet1/2
description to_r2
no switchport
mtu 9202
ip address 10.0.21.2/24
no ip ospf passive-interface
ip router ospf main_ospf area 0.0.0.0
no shutdown
interface Ethernet1/3
interface Ethernet1/4
interface Ethernet1/5
interface Ethernet1/6
interface Ethernet1/7
interface Ethernet1/8
interface Ethernet1/9
switchport access vlan 10
interface Ethernet1/10
interface Ethernet1/11
interface Ethernet1/12
interface Ethernet1/13
interface Ethernet1/14
interface Ethernet1/15
interface Ethernet1/16
interface Ethernet1/17
interface Ethernet1/18
interface Ethernet1/19
interface Ethernet1/20
interface Ethernet1/21
interface Ethernet1/22
interface Ethernet1/23
interface Ethernet1/24
interface Ethernet1/25
interface Ethernet1/26
interface Ethernet1/27
interface Ethernet1/28
interface Ethernet1/29
interface Ethernet1/30
interface Ethernet1/31
interface Ethernet1/32
interface Ethernet1/33
interface Ethernet1/34
interface Ethernet1/35
interface Ethernet1/36
interface Ethernet1/37
interface Ethernet1/38
interface Ethernet1/39
interface Ethernet1/40
interface Ethernet1/41
interface Ethernet1/42
interface Ethernet1/43
interface Ethernet1/44
interface Ethernet1/45
interface Ethernet1/46
interface Ethernet1/47
interface Ethernet1/48
interface Ethernet1/49
interface Ethernet1/50
interface Ethernet1/51
interface Ethernet1/52
interface Ethernet1/53
interface Ethernet1/54
interface Ethernet1/55
interface Ethernet1/56
interface Ethernet1/57
interface Ethernet1/58
interface Ethernet1/59
interface Ethernet1/60
interface Ethernet1/61
interface Ethernet1/62
interface Ethernet1/63
interface Ethernet1/64
interface mgmt0
vrf member management
interface loopback0
ip address 192.168.55.1/32
ip router ospf main_ospf area 0.0.0.0
icam monitor scale
line console
line vty
boot nxos bootflash:/nxos64-cs.10.2.5.M.bin
router ospf main_ospf
router-id 192.168.55.1
passive-interface default
router bgp 65000
router-id 192.168.55.1
address-family l2vpn evpn
neighbor 192.168.99.1
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 192.168.99.2
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
evpn
vni 10000 l2
route-target import auto
route-target export auto
vni 20000 l2
route-target import auto
route-target export auto
no system default switchport shutdown
Leaf sw2
sw2# show running-config
!Command: show running-config
!No configuration change since last restart
!Time: Fri Nov 10 00:12:16 2023
version 10.2(5) Bios:version
switchname sw2
vdc sw2 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
username admin password 5 $5$MABCHN$gUqF9QuVnSFIDj/pFMmyAP50tJidjCcvsT2DZ84N406
role network-admin
ip domain-lookup
copp profile strict
snmp-server user admin network-admin auth md5 01461793D54A024F0402260F6F6A68FE17
E7 priv aes-128 494278ACA1CCD181E9D9AECAB2DAF73B8D35 localizedV2key
rmon event 1 log trap public description FATAL(1) owner PMON@FATAL
rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 log trap public description ERROR(3) owner PMON@ERROR
rmon event 4 log trap public description WARNING(4) owner PMON@WARNING
rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO
system default switchport
fabric forwarding anycast-gateway-mac 0010.0010.0010
vlan 1,10,20,999
vlan 10
vn-segment 10000
vlan 20
vn-segment 20000
vlan 999
vn-segment 999000
vrf context management
vrf context tenant1
vni 999000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
interface Vlan1
interface Vlan10
no shutdown
vrf member tenant1
ip address 10.10.10.1/24
fabric forwarding mode anycast-gateway
interface Vlan20
no shutdown
vrf member tenant1
ip address 10.10.20.1/24
fabric forwarding mode anycast-gateway
interface Vlan999
no shutdown
vrf member tenant1
ip forward
ipv6 forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10000
ingress-replication protocol bgp
member vni 20000
ingress-replication protocol bgp
member vni 999000 associate-vrf
interface Ethernet1/1
description to_r1
no switchport
mtu 9202
ip address 10.0.12.2/24
no ip ospf passive-interface
ip router ospf main_ospf area 0.0.0.0
no shutdown
interface Ethernet1/2
description to_r2
no switchport
mtu 9202
ip address 10.0.22.2/24
no ip ospf passive-interface
ip router ospf main_ospf area 0.0.0.0
no shutdown
interface Ethernet1/3
interface Ethernet1/4
interface Ethernet1/5
interface Ethernet1/6
interface Ethernet1/7
interface Ethernet1/8
description to_Linux-4
switchport access vlan 20
interface Ethernet1/9
description to_Linux-2
switchport access vlan 10
interface Ethernet1/10
interface Ethernet1/11
interface Ethernet1/12
interface Ethernet1/13
interface Ethernet1/14
interface Ethernet1/15
interface Ethernet1/16
interface Ethernet1/17
interface Ethernet1/18
interface Ethernet1/19
interface Ethernet1/20
interface Ethernet1/21
interface Ethernet1/22
interface Ethernet1/23
interface Ethernet1/24
interface Ethernet1/25
interface Ethernet1/26
interface Ethernet1/27
interface Ethernet1/28
interface Ethernet1/29
interface Ethernet1/30
interface Ethernet1/31
interface Ethernet1/32
interface Ethernet1/33
interface Ethernet1/34
interface Ethernet1/35
interface Ethernet1/36
interface Ethernet1/37
interface Ethernet1/38
interface Ethernet1/39
interface Ethernet1/40
interface Ethernet1/41
interface Ethernet1/42
interface Ethernet1/43
interface Ethernet1/44
interface Ethernet1/45
interface Ethernet1/46
interface Ethernet1/47
interface Ethernet1/48
interface Ethernet1/49
interface Ethernet1/50
interface Ethernet1/51
interface Ethernet1/52
interface Ethernet1/53
interface Ethernet1/54
interface Ethernet1/55
interface Ethernet1/56
interface Ethernet1/57
interface Ethernet1/58
interface Ethernet1/59
interface Ethernet1/60
interface Ethernet1/61
interface Ethernet1/62
interface Ethernet1/63
interface Ethernet1/64
interface mgmt0
vrf member management
ip address 10.0.0.2/24
interface loopback0
ip address 192.168.55.2/32
ip router ospf main_ospf area 0.0.0.0
icam monitor scale
line console
line vty
boot nxos bootflash:/nxos64-cs.10.2.5.M.bin
router ospf main_ospf
router-id 192.168.55.2
passive-interface default
router bgp 65000
address-family l2vpn evpn
neighbor 192.168.99.1
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 192.168.99.2
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
evpn
vni 10000 l2
route-target import auto
route-target export auto
vni 20000 l2
route-target import auto
route-target export auto
no system default switchport shutdown
Leaf sw3
sw3# show running-config | no-more
!Command: show running-config
!Running configuration last done at: Fri Nov 10 00:01:22 2023
!Time: Fri Nov 10 00:09:15 2023
version 10.2(5) Bios:version
switchname sw3
vdc sw3 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
username admin password 5 $5$EHHCCC$XE.MliFvH6EEc5GGywgWyb4Qx6NysPdtudqLMBFSQXB role network-admin
ip domain-lookup
copp profile strict
snmp-server user admin network-admin auth md5 204F143CE4DF43D72D285A335226CE7CCE28 priv aes-128 480A6B1FD1E5408864374037136D9D239E0C localizedV2key
rmon event 1 log trap public description FATAL(1) owner PMON@FATAL
rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 log trap public description ERROR(3) owner PMON@ERROR
rmon event 4 log trap public description WARNING(4) owner PMON@WARNING
rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO
system default switchport
fabric forwarding anycast-gateway-mac 0010.0010.0010
vlan 1,10,20,555,999
vlan 10
vn-segment 10000
vlan 20
vn-segment 20000
vlan 555
vn-segment 555
vlan 999
vn-segment 999000
ip prefix-list NETWORK seq 5 permit 99.0.0.0/8 le 32
route-map ADD-RT permit 10
match ip address prefix-list NETWORK
set community 65534:65534
route-map ALL permit 10
vrf context external-shared
vni 555
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:999000
route-target import 65000:999000 evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:999000
route-target import 65000:999000 evpn
vrf context management
vrf context tenant1
vni 999000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:555
route-target import 65000:555 evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
route-target import 65000:555
route-target import 65000:555 evpn
interface Vlan1
interface Vlan10
no shutdown
vrf member tenant1
ip address 10.10.10.1/24
fabric forwarding mode anycast-gateway
interface Vlan20
no shutdown
vrf member tenant1
ip address 10.10.20.1/24
fabric forwarding mode anycast-gateway
interface Vlan555
no shutdown
vrf member external-shared
ip forward
ipv6 forward
interface Vlan999
no shutdown
vrf member tenant1
ip forward
ipv6 forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 555 associate-vrf
member vni 10000
ingress-replication protocol bgp
member vni 20000
ingress-replication protocol bgp
member vni 999000 associate-vrf
interface Ethernet1/1
description to_r1
no switchport
mtu 9202
ip address 10.0.13.2/24
no ip ospf passive-interface
ip router ospf main_ospf area 0.0.0.0
no shutdown
interface Ethernet1/2
description to_r2
no switchport
mtu 9202
ip address 10.0.23.2/24
no ip ospf passive-interface
ip router ospf main_ospf area 0.0.0.0
no shutdown
interface Ethernet1/3
interface Ethernet1/4
interface Ethernet1/5
no switchport
vrf member external-shared
ip address 100.64.64.1/31
ipv6 address 2001::1/127
no shutdown
interface Ethernet1/6
interface Ethernet1/7
interface Ethernet1/8
description to_Linux-5
switchport access vlan 20
interface Ethernet1/9
description to_Linux-3
switchport access vlan 10
interface Ethernet1/10
interface Ethernet1/11
interface Ethernet1/12
interface Ethernet1/13
interface Ethernet1/14
interface Ethernet1/15
interface Ethernet1/16
interface Ethernet1/17
interface Ethernet1/18
interface Ethernet1/19
interface Ethernet1/20
interface Ethernet1/21
interface Ethernet1/22
interface Ethernet1/23
interface Ethernet1/24
interface Ethernet1/25
interface Ethernet1/26
interface Ethernet1/27
interface Ethernet1/28
interface Ethernet1/29
interface Ethernet1/30
interface Ethernet1/31
interface Ethernet1/32
interface Ethernet1/33
interface Ethernet1/34
interface Ethernet1/35
interface Ethernet1/36
interface Ethernet1/37
interface Ethernet1/38
interface Ethernet1/39
interface Ethernet1/40
interface Ethernet1/41
interface Ethernet1/42
interface Ethernet1/43
interface Ethernet1/44
interface Ethernet1/45
interface Ethernet1/46
interface Ethernet1/47
interface Ethernet1/48
interface Ethernet1/49
interface Ethernet1/50
interface Ethernet1/51
interface Ethernet1/52
interface Ethernet1/53
interface Ethernet1/54
interface Ethernet1/55
interface Ethernet1/56
interface Ethernet1/57
interface Ethernet1/58
interface Ethernet1/59
interface Ethernet1/60
interface Ethernet1/61
interface Ethernet1/62
interface Ethernet1/63
interface Ethernet1/64
interface mgmt0
vrf member management
ip address 10.0.0.2/24
interface loopback0
ip address 192.168.55.3/32
ip router ospf main_ospf area 0.0.0.0
icam monitor scale
line console
line vty
boot nxos bootflash:/nxos64-cs.10.2.5.M.bin
router ospf main_ospf
router-id 192.168.55.3
passive-interface default
router bgp 65000
address-family l2vpn evpn
neighbor 192.168.99.1
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 192.168.99.2
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf external-shared
address-family ipv4 unicast
address-family ipv6 unicast
neighbor 2001::
remote-as 42
address-family ipv6 unicast
neighbor 100.64.64.0
remote-as 42
address-family ipv4 unicast
route-map ADD-RT in
vrf tenant1
address-family ipv4 unicast
redistribute direct route-map ALL
evpn
vni 10000 l2
route-target import auto
route-target export auto
vni 20000 l2
route-target import auto
route-target export auto
no system default switchport shutdown