AVD example for a MPLS-VPN based WAN Network¶
Introduction¶
This example is the logical second step in introducing AVD to new users, following the Introduction to Ansible and AVD section. New users with access to virtual routers (using Arista vEOS-lab or cEOS) can learn how to generate configuration and documentation for a complete fabric environment. Users with access to physical routers will have to adapt a few settings. This is all documented inline in the comments included in the YAML files. If a lab with virtual or physical routers is not accessible, this example can also be used to only generate the output from AVD if desired.
The example includes and describes all the AVD files and their content used to build a MPLS-VPN WAN network covering two sites using the following:
- Four (virtual) p routers.
- Three (virtual) pe routers serving aggregation devices and CPEs.
- Two (virtual) route reflectors act as route servers for the WAN.
This example does not include Integration with CloudVision to keep everything as simple as possible. In this case, the Ansible host will communicate directly with the routers using eAPI.
Installation¶
Requirements to use this example:
- Follow the installation guide for AVD
- Run the following playbook to copy the AVD examples to your current working directory, for example
ansible-avd-examples
:
This will show the following:
~/ansible-avd-examples# ansible-playbook arista.avd.install_examples
PLAY [Install Examples]**********************************************************************************************
TASK [Copy all examples to ~/ansible-avd-examples]*******************************************************************
changed: [localhost]
PLAY RECAP
*********************************************************************************************************************
localhost : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Info
If the content of any file is modified and the playbook is rerun, the file will not be overwritten. However, if any file in the example is deleted and the playbook is rerun, Ansible will re-create the file.
After the playbook has run successfully, the directory structure of the example should look like below, the contents of which will be covered in later sections:
ansible-avd-examples/ (or wherever the playbook was run)
|── isis-ldp-ipvpn
├── ansible.cfg
├── build.yml
├── deploy.yml
├── documentation
├── group_vars
├── images
├── intended
├── inventory.yml
├── README.md
└── switch-basic-configurations
Overall design overview¶
Physical topology¶
The drawing below shows the physical topology used in this example. The interface assignment shown here is referenced across the entire example, so keep that in mind if this example must be adapted to a different topology. Finally, the Ansible host is connected to the dedicated out-of-band management port (Management1 when using vEOS-lab):
IP ranges used¶
Out-of-band management IP allocation for WAN1 | 172.16.1.0/24 |
---|---|
Default gateway | 172.16.1.1 |
p1 | 172.16.1.11 |
p2 | 172.16.1.12 |
p3 | 172.16.1.13 |
p4 | 172.16.1.14 |
pe1 | 172.16.1.101 |
pe2 | 172.16.1.102 |
pe3 | 172.16.1.103 |
rr1 | 172.16.1.151 |
rr2 | 172.16.1.152 |
Point-to-point links between network nodes | (Underlay) |
WAN1 | 10.255.3.0/24 |
Loopback0 interfaces for router ID (p) | 10.255.0.0/27 |
Loopback0 interfaces for overlay peering (pe) | 10.255.1.0/27 |
Loopback0 interfaces for overlay peering (rr) | 10.255.2.0/27 |
L3 Interfaces | 10.0-1.1.0/24 |
For example pe1 Ethernet3.10 has the IP address: |
10.0.1.1 |
For example pe3 Ethernet4 has the IP address: |
10.1.1.9 |
ISIS-LDP design¶
BGP design¶
Basic EOS config¶
Basic connectivity between the Ansible host and the routers must be established before Ansible can be used to push configurations. You must configure the following on all routers:
- A hostname configured purely for ease of understanding.
- An IP enabled interface - in this example, the dedicated out-of-band management interface is used.
- A username and password with the proper access privileges.
Below is the basic configuration file for p1
:
! ansible-avd-examples/single-dc-l3ls/switch-basic-configurations/p1-basic-configuration.txt
! Basic EOS config
!
! Hostname of the device
hostname p1
!
! Configures username and password for the ansible user
username ansible privilege 15 role network-admin secret sha512 $6$7u4j1rkb3VELgcZE$EJt2Qff8kd/TapRoci0XaIZsL4tFzgq1YZBLD9c6f/knXzvcYY0NcMKndZeCv0T268knGKhOEwZAxqKjlMm920
!
! Defines the VRF for MGMT
vrf instance MGMT
!
! Defines the settings for the Management1 interface through which Ansible reaches the device
interface Management1
description OOB_MANAGEMENT
no shutdown
vrf MGMT
! IP address - must be set uniquely per device
ip address 172.16.1.11/24
!
! Static default route for VRF MGMT
ip route vrf MGMT 0.0.0.0/0 172.16.1.1
!
! Enables API access in VRF MGMT
management api http-commands
protocol https
no shutdown
!
vrf MGMT
no shutdown
!
end
!
! Save configuration to flash
copy running-config startup-config
Note
The folder isis-ldp-ipvpn/switch-basic-configurations/
contains a file per device for the initial configurations.
Ansible inventory, group vars, and naming scheme¶
The following drawing shows a graphic overview of the Ansible inventory, group variables, and naming scheme used in this example:
Note
The CPE’s and aggregation nodes are not configured by AVD, but the ports used to connect to them are.
Group names use uppercase and underscore syntax:
- FABRIC
- WAN1
- WAN1_P_ROUTERS
- WAN1_PE_ROUTERS
- WAN1_RR_ROUTERS
All hostnames use lowercase, for example:
- p4
- pe1
- rr2
The drawing also shows the relationships between groups and their children:
- For example,
p1
,p2
,p3
, andp4
are all children of the group calledWAN1_P_ROUTERS
.
Additionally, groups themselves can be children of another group, for example:
WAN1_P_ROUTERS
is a child of the groupWAN1
.WAN1_PE_ROUTERS
is a child of bothWAN1
andNETWORK_SERVICES
.
This naming convention makes it possible to extend anything easily, but as always, this can be changed based on your preferences. Just ensure that the names of all groups and hosts are unique.
Content of the inventory.yml file¶
This section describes the entire ansible-avd-examples/isis-ldp-ipvpn/inventory.yml
file used to represent the above topology.
The hostnames specified in the inventory must exist either in DNS or in the hosts file on your Ansible host to allow successful name lookup and be able to reach the routers directly. A successful ping from the Ansible host to each inventory host verifies name resolution(e.g., ping p1
).
Alternatively, if there is no DNS available, or if devices need to be reached using a fully qualified domain name (FQDN), define ansible_host
to be an IP address or FQDN for each device - see below for an example:
---
all:
children:
FABRIC:
children:
WAN1:
children:
WAN1_P_ROUTERS:
hosts:
p1:
ansible_host: 172.16.1.11
p2:
ansible_host: 172.16.1.12
p3:
ansible_host: 172.16.1.13
p4:
ansible_host: 172.16.1.14
WAN1_PE_ROUTERS:
hosts:
pe1:
ansible_host: 172.16.1.101
pe2:
ansible_host: 172.16.1.102
pe3:
ansible_host: 172.16.1.103
WAN1_RR_ROUTERS:
hosts:
rr1:
ansible_host: 172.16.1.151
rr2:
ansible_host: 172.16.1.152
NETWORK_SERVICES:
children:
WAN1_PE_ROUTERS:
The above is included in this example, purely to make it as simple as possible. However, in the future, please do not carry over this practice to a production environment, where an inventory file for an identical topology should look as follows when using DNS:
---
all:
children:
FABRIC:
children:
WAN1:
children:
WAN1_P_ROUTERS:
hosts:
p1:
p2:
p3:
p4:
WAN1_PE_ROUTERS:
hosts:
pe1:
pe2:
pe3:
WAN1_RR_ROUTERS:
hosts:
rr1:
rr2:
NETWORK_SERVICES:
children:
WAN1_PE_ROUTERS:
NETWORK_SERVICES
- Creates a group named
NETWORK_SERVICES
. Ansible variable resolution resolves this group name to the identically named group_vars file (ansible-avd-examples/isis-ldp-ipvpn/group_vars/NETWORK_SERVICES.yml
).- The file’s contents are specifications of tenant VRFs and their associated routed interfaces, BGP peers, and OSPF interfaces, then applied to the group’s children. In this case, the group
WAN1_PE_ROUTERS
.
Defining device types¶
Since this example covers building a MPLS WAN network, AVD must know about the device types, for example, p, pe, rr routers, etc. The devices are already grouped in the inventory, so the device types are specified in the group variable files with the following names and content:
For example, all routers that are children of the WAN1_P_ROUTERS group defined in the inventory will be of type p
.
Setting fabric-wide configuration parameters¶
The ansible-avd-examples/isis-ldp-ipvpn/group_vars/FABRIC.yml
file defines generic settings that apply to all children of the FABRIC
group as specified in the inventory described earlier.
The first section defines how the Ansible host connects to the devices:
ansible_connection: ansible.netcommon.httpapi # (1)!
ansible_network_os: arista.eos.eos # (2)!
ansible_user: ansible # (3)!
ansible_password: ansible
ansible_become: true
ansible_become_method: enable # (4)!
ansible_httpapi_use_ssl: true # (5)!
ansible_httpapi_validate_certs: false # (6)!
- The Ansible host must use eAPI
- Network OS which in this case is Arista EOS
- The username/password combo
- How to escalate privileges for write access
- Use SSL
- Do not validate SSL certificates
The following section specifies variables that generate configuration to be applied to all devices in the fabric:
fabric_name: FABRIC # (1)!
underlay_routing_protocol: isis-ldp
overlay_routing_protocol: ibgp
local_users: # (2)!
- name: ansible
privilege: 15
role: network-admin
sha512_password: $6$QJUtFkyu9yoecsq.$ysGzlb2YXaIMvezqGEna7RE8CMALJHnv7Q1i.27VygyKUtSeX.n2xRTyOtCR8eOAl.4imBLyhXFc4o97P5n071
- name: admin
privilege: 15
role: network-admin
no_password: true
bgp_peer_groups: # (3)!
mpls_overlay_peers:
password: Q4fqtbqcZ7oQuKfuWtNGRQ==
p2p_uplinks_mtu: 1500 # (4)!
- The name of the fabric for internal AVD use. This name must match the name of an Ansible Group (and therefore a corresponding group_vars file) covering all network devices.
- Local users/passwords and their privilege levels. In this case, the
ansible
user is set with the passwordansible
, and anadmin
user is set with no password. - BGP peer groups and their passwords (all passwords are “arista”).
- Point-to-point interface MTU, in this case, is set to 1500 since the example uses vEOS, but when using hardware, this should be set to 9214 instead.
Setting device-specific configuration parameters¶
The ansible-avd-examples/isis-ldp-ipvpn/group_vars/WAN1.yml
file defines settings that apply to all children of the WAN1
group as specified in the inventory described earlier. However, this time the settings defined are no longer fabric-wide but are limited to WAN1. This example is of limited benefit with only a single data center. Still, it allows us to scale the configuration to a scenario with multiple data centers in the future.
---
mgmt_gateway: 172.16.1.1 # (1)!
p:
defaults:
platform: vEOS-lab # (2)!
loopback_ipv4_pool: 10.255.0.0/27 # (3)!
nodes: # (4)!
- name: p1
id: 1 # (5)!
mgmt_ip: 172.16.1.11/24 # (6)!
- name: p2
id: 2
mgmt_ip: 172.16.1.12/24
- The default gateway for the management interface of all devices in WAN1 is defined.
platform
references default settings defined in AVD specific to certain switch platforms.loopback_ipv4_pool
defines the IP scope from which AVD assigns IPv4 addresses for Loopback0.nodes
defines the actual p routers, using the hostnames defined in the inventory.id
is used to calculate the various IP addresses, for example, the IPv4 address for the Loopback0 interface. In this case, p1 will get the IPv4 address 10.255.0.1/27 assigned to the Loopback0 interface.mgmt_ip
defines the IPv4 address of the management interface. As stated earlier, Ansible will perform name lookups using the hostnames specified in the inventory unless using theansible_host
option. However, there is no automatic mechanism to grab the result of the name lookup and use that to generate the management interface configuration.
The following section covers the pe routers. Significantly more settings need to be set compared to the p routers:
# PE router group
pe:
defaults:
platform: vEOS-lab # (1)!
loopback_ipv4_pool: 10.255.1.0/27 # (2)!
virtual_router_mac_address: 00:1c:73:00:dc:00 # (3)!
mpls_route_reflectors: [ rr1, rr2 ] # (4)!
isis_system_id_prefix: '0000.0001' # (5)!
spanning_tree_mode: none # (6)!
node_groups: # (7)!
- group: WAN1-PE1-2
nodes:
- name: pe1
id: 1
mgmt_ip: 172.16.1.101/24
- name: pe2
id: 2
mgmt_ip: 172.16.1.102/24
- group: WAN1-PE3
nodes:
- name: pe3
id: 3
mgmt_ip: 172.16.1.103/24
platform
references default settings defined in AVD specific to certain switch platforms.loopback_ipv4_pool
defines the IP scope from which AVD assigns IPv4 addresses for Loopback0. Please note that this IP pool differs from the one used for the p routers in this example. If you want to reuse the same IP pool for multiple node types to avoid setting the same IP addresses for several devices, we can define the optionloopback_ipv4_offset
.virtual_router_mac_address
defines the MAC address used for the anycast gateway on the various subnets. This is the MAC address connected endpoints will learn when ARPing for their default gateway. It is irrelevant for the vpn-ipv4/6 services used in this example but is still mandatory to set.mpls_route_reflectors
defines which route reflectors the pe nodes peer with for overlay route distribution.isis_system_id_prefix
is mandatory to set when using ISIS for the underlay routing protocol. It is used to calculate the ISIS NET ID.spanning_tree_mode
defines the spanning tree mode. In this case, we are not using spanning tree since we have only routed interfaces on our pe routers.node_groups
defines settings common to more than one node. In the l3ls-evpn design this has more utility than here, which is used to define MLAG pairs. In the MPLS design, it is mainly used to logically group devices for organizational purposes.
Finally, more of the same, but this time for the rr routers:
rr:
defaults:
platform: vEOS-lab
loopback_ipv4_pool: 10.255.2.0/27
mpls_route_reflectors: [ rr1, rr2 ] # (1)!
isis_system_id_prefix: '0000.0002'
spanning_tree_mode: none
node_groups:
- group: WAN1_RR1-2
nodes:
- name: rr1
id: 1
mgmt_ip: 172.16.1.151/24
- name: rr2
id: 2
mgmt_ip: 172.16.1.152/24
mpls_route_reflectors
is used here to make the rr nodes peer with each other.
Defining underlay connectivity between network nodes¶
A free-standing list of core_interfaces
dictionaries and their associated profiles and ip pools defines the underlay connectivity between nodes.
core_interfaces:
p2p_links_ip_pools:
- name: core_pool # (1)!
ipv4_pool: 10.255.3.0/24
p2p_links_profiles:
- name: core_profile # (2)!
mtu: 1500
isis_metric: 50
ip_pool: core_pool
isis_circuit_type: level-2
isis_authentication_mode: md5
isis_authentication_key: $1c$sTNAlR6rKSw=
p2p_links: # (3)!
- nodes: [ pe1, p1 ] # (4)!
interfaces: [ Ethernet1, Ethernet1 ]
profile: core_profile # (5)!
id: 1 # (6)!
- nodes: [ pe1, p2 ]
interfaces: [ Ethernet2, Ethernet2 ]
profile: core_profile
id: 2
- The IP pool
name
is used to assign a name to the IP pool, this is later called in the profile to associate the pool to the profile. - The profile
name
is used to assign a name to the link profile, which is later called under the p2p link definitions to inherit settings from the profile. - Each list item in
p2p_links
is a dictionary that defines one routed point-to-point underlay link and its associated parameters. nodes
is used to identify which nodes are connecting.profile
is used here to inherit common settings for the link from the profile.id
is used to extract a single /31 subnet for the link from the IP pool mentioned by the profile. Each link that shares an IP pool must have a unique ID to prevent overlapping IP addressing.
Specifying network services (VRFs and routed interfaces) and endpoint connectivity in the VPN-IPv4 fabric¶
---
tenants:
# Definition of tenants. Additional level of abstraction to VRFs
- name: CUSTOMER1
vrfs:
# VRF definitions inside the tenant.
- name: C1_VRF1
# VRF ID definition.
vrf_id: 10
# Select address families for the VRF.
address_families:
- vpn-ipv4
# Enable OSPF on selected PEs in the VRF.
ospf:
enabled: true
nodes:
- pe1
- pe2
- pe3
l3_interfaces:
# L3 interfaces
- interfaces: [ Ethernet3.10, Ethernet4.10, Ethernet2 ]
nodes: [ pe1, pe2, pe3 ]
description: C1_L3_SERVICE
enabled: true
ip_addresses: [ 10.0.1.1/29, 10.0.1.2/29, 10.0.1.9/30 ]
# Enable OSPF on the interfaces.
ospf:
enabled: true
- name: CUSTOMER2
vrfs:
- name: C2_VRF1
vrf_id: 20
address_families:
- vpn-ipv4
l3_interfaces:
# L3 interfaces
- interfaces: [ Ethernet3.20, Ethernet4.20, Ethernet4 ]
nodes: [ pe1, pe2, pe3 ]
description: C2_L3_SERVICE
enabled: true
ip_addresses: [ 10.1.1.1/29, 10.1.1.2/29, 10.1.1.9/30 ]
# Define BGP peers inside the VRF.
bgp_peers:
- ip_address: 10.1.1.3
remote_as: 65123
description: C2_ROUTER1
send_community: standard
maximum_routes: 100
nodes: [ pe1, pe2 ]
- ip_address: 10.1.1.10
remote_as: 65124
description: C2_ROUTER2
send_community: standard
maximum_routes: 100
nodes: [ pe3 ]
All tenant VRFs and routed interfaces for endpoint connectivity in the network are defined here.
Two tenants called CUSTOMER1
and CUSTOMER2
are specified. Each of these tenants has a single VRF defined, and under those VRFs, we define the routed interfaces, tenant (PE-CE) routing protocols and address families in use:
- name: C1_VRF1
vrf_id: 10
address_families:
- vpn-ipv4
ospf:
enabled: true
nodes:
- pe1
- pe2
- pe3
l3_interfaces:
- interfaces: [ Ethernet3.10, Ethernet4.10, Ethernet2 ]
nodes: [ pe1, pe2, pe3 ]
description: C1_L3_SERVICE
enabled: true
ip_addresses: [ 10.0.1.1/29, 10.0.1.2/29, 10.0.1.9/30 ]
ospf:
enabled: true
This defines C1_VRF1
, with a VRF ID of 10
, enables OSPF routing for PE-CE connections inside the VRF on selected pe routers and defines routed interfaces that are used to connect to the CE devices/aggregation nodes. Each interface has an IP address assigned, a description, and has OSPF routing enabled.
The lists of interfaces, nodes, and ip_addresses used in the above definition of the l3 interface are read by the ansible logic as follows: interface Ethernet3.10
belongs to the node pe1
and has the IP address of 10.0.1.1/29
. In other words, the list indices are used to form the basic parameters for one interface.
The playbook¶
In this example, the deploy playbook looks like the following:
---
# Please note, comments below are intended for site documentation only
- name: Run AVD # (1)!
hosts: FABRIC
gather_facts: false
tasks:
- name: Generate AVD Structured Configurations and Fabric Documentation # (2)!
ansible.builtin.import_role:
name: arista.avd.eos_designs
- name: Generate Device Configurations and Documentation # (3)!
ansible.builtin.import_role:
name: arista.avd.eos_cli_config_gen
- name: Deploy Configurations to Devices # (4)!
ansible.builtin.import_role:
name: arista.avd.eos_config_deploy_eapi
- At the highest level, the name and scope of the playbook are set, which in this example is the entire fabric. For instance,
FABRIC
is a group name defined in the inventory. If the playbook should only apply to a subset of devices, it can be changed here. - This task uses the role
arista.avd.eos_designs
, which generates structured configuration for each device. This structured configuration can be found in theansible-avd-examples/isis-ldp-ipvpn/intended/structured_configs
folder. - This task uses the role
arista.avd.eos_cli_config_gen
, which generates the actual Arista EOS CLI configurations found in theansible-avd-examples/isis-ldp-ipvpn/intended/configs
folder, along with the device-specific and fabric wide documentation found in theansible-avd-examples/isis-ldp-ipvpn/documentation/
folder. In addition, it relies on the structured configuration generated byarista.avd.eos_designs
. - This task uses the role
arista.avd.eos_config_deploy_eapi
that pushes the generated configuration to the devices in scope.
Testing AVD output without a lab¶
Example of using the build playbook without devices (local tasks):
---
# Please note, comments below are intended for site documentation only
- name: Run AVD
hosts: FABRIC
gather_facts: false
tasks:
- name: Generate AVD Structured Configurations and Fabric Documentation
ansible.builtin.import_role:
name: arista.avd.eos_designs
- name: Generate Device Configurations and Documentation
ansible.builtin.import_role:
name: arista.avd.eos_cli_config_gen
The build playbook will generate all of the output (variables, configurations, documentation) but will not attempt to communicate with any devices.
Please look through the folders and files described above to learn more about the output generated by AVD.
Executing the playbook¶
The execution of the playbook should produce the following output:
user@ubuntu:~/isis-ldp-ipvpn$ ansible-playbook deploy.yml
PLAY [Run AVD] *****************************************************************************************************************************************************************************
TASK [arista.avd.eos_designs : Collection arista.avd version 3.5.0 loaded from /home/user/.ansible/collections/ansible_collections] ******************************************************
ok: [p1]
TASK [arista.avd.eos_designs : Create required output directories if not present] **********************************************************************************************************
ok: [p1 -> localhost] => (item=/home/user/Documents/git_projects/ansible-avd-examples/isis-ldp-ipvpn/intended/structured_configs)
ok: [p1 -> localhost] => (item=/home/user/Documents/git_projects/ansible-avd-examples/isis-ldp-ipvpn/documentation/fabric)
(...)
If similar output is not shown, make sure:
- The documented requirements are met.
- The latest
arista.avd
collection is installed.
Troubleshooting¶
VPN-IPv4 Overlay not working¶
If after doing the following steps:
- Manually copy/paste the switch-basic-configuration to the devices.
- Run the playbook and push the generated configuration to the fabric.
- Login to a pe or rr device, for example, pe1 and run the command
show bgp vpn-ipv4 summary
to view VPN routes.
The following error message is shown:
This is caused by AVD pushing the configuration line service routing protocols model multi-agent
, which enables the multi-agent routing process supporting VPN-IPv4 and EVPN. This change requires a reboot of the device.
VPN-IPv4 Overlay in Arista Cloud Test (ACT)¶
Suppose you are running this lab in the Arista Cloud Test service, and the overlay services are not working (no connectivity from CPE to CPE) after performing the abovementioned steps. In that case, you may need to change the default forwarding engine of the vEOS nodes.
Add the following line to the starting configurations for each node:
Currently, this command must be manually entered into the device configurations before trying to push the command with AVD. After you have entered it manually on each node, add the following YAML to group_vars/WAN1.yml and run the deployment playbook:
Retest the services. They should now work, provided the CPEs and aggregation node are correctly configured.