In this blog post I am laying my study notes on the topic Cisco ACI Domains. We learn the concept of workloads, bare metal servers and virtualisation servers first. Second, we discover the types of networking domains Cisco ACI offers. Third, we read about how to configure an external L2 connection. And we finish with an example of configuring an L3 external connection.

Definitions

  • A domain in ACI defines where and how to use a VLAN Pool.
  • For each VLAN Pool you associate one domain. It is a 1:1 mapping.
  • There are 4 types of domains:
    • physical domains
    • VMM (virtual) domains
    • External Bridged domains
    • External Routed domains
  • EPG(s) must be attached to a domain. An EPG attaches to only one domain. A domain however can be attached to one or more EPGs.
  • To be able to logically connect a physical server to ACI fabric you need to configure a Physical Domain on APIC
  • To be able to logically connect a virtualized server to ACI fabric you need to configure a VMM Domain on APIC

Bare Metal Host Concept

A Bare metal host (BMH) is a physical server that does not have any virtualisation capabilities. It is a regular server, the kind of which that you may see sitting shy in a modern datacenter in a corner, near big chassis and blade systems. Th

The bare metal server provides a physical workload. Cisco ACI offers the possibility to manage this physical workload when we configure APIC to integrate with the bare metal host. Part of this integration is the configuration of a physical domain.

In lab environments, we can emulate a bare metal host by connecting a L3 switch to the fabric and:

  • setting L2 connectivity between the switch and the fabric (EPG with static path binding on the ACI fabric + L2 trunking on the switch)
  • associating a physical domain to the AAEP (not an external routed domain).

Physical Domain

  • A physical domain connects a bare metal host to the ACI fabric.
  • the default physical domain that comes with ACI is named phys.

To configure a physical domain, go to Fabric -> Access Policies -> Physical and External Domains

Right-click -> create Physical Domain

Virtual Domain, aka VMM Domain

  • connects a virtualized server (aka virtualization host) to the fabric.
  • is configured on APIC as a VMM Domain.
  • connects a Virtual Machine Manager such as Vmware vCenter (the management console in a vSphere Cluster) or Microsoft SCVMM to APIC.
    • when you create a VMM Domain on APIC – as part of integrating a VMM to ACI-, a virtual switch will be created on the virtualized server, and the virtual environment parameters (in the example of VMware: vCenter name, vNICs, list of VMs) becomes visible on the ACI. For this, you have to choose at the configuration menu between VMware DVS or Cisco AVS.
    • must be assigned a VLAN or VXLAN pool. The VLAN pool will provide the VLAN IDs that will be assigned dynamically to Port Groups on the vSphere Virtual Switch for example in case of Vmware integration with ACI

External Routed Domains and External Bridged Domains

  • An External Bridge Domain (aka L2 Domain) is used when an external L2 network, such as a switch, is physically connected to the ACI fabric. Similarly, an External Routed Domain (aka L3 Domain) is used when a L3 network, such as a router or a firewall, is physically connected to the fabric.

Connecting an ACI Fabric to an External L2 Bridged Network

We have two logical possibilities:

Extending an EPG with Static Binding

  • we statically assign the port connected to the L2 external network to an internal EPG. On the opposite device we simply configure 802.1q trunk links.
  • this method does not require contracts to communicate with the external switched network

L2Out

  • the configuration of a L2 Out involves the configuration of an ACI construct named External Bridged Network or External L2 Network or Bridged Outside, and associating it with an internal Bridge Domain which we want to “extend” to the outside world. That is why we name this method as Bridge Domain extension.
  • and we have three physical connection possibilities:
    • regular L2 port
    • port channel
    • vPC

Configuring L2Out

First, the L2out requires access policies to be configured:

  • Interface Policies
  • Interfac Policy Group
  • Interface Profile
  • Switch Profile
  • VLAN Pool
  • AAEP
  • External Bridged Domain:

Once the External Bridged Domain is created, we can not return back to the menu and associate an AAEP for it:

However we still are able to associate an AAEP to the External Bridged Domain on the AAEP configuration page:

We go back to the External Bridged Domain configuration page and I see the associated AAEP:

Associate the VLAN Pool to the External Bridged Domain:

  • Configure the External Bridged Network:

After giving it a Name we associate it to the External Bridged Domain we’ve configured:

We fill in the Bridge Domain that we want to extend to the outside switched network:

We configure the VLAN encapsulation that is transiting between the Leaf port and the external switching device:

This VLAN ID must be within the range of the VLAN Pool associated to the External Bridged Domain we’ve configured earlier. So in our case, VLAN ID 24 falls within the range [23-55]:

In Node and Interface Protocol Profiles we choose vPC, because we are having the connection between ACI and the external switching network over a vPC interface:

Click Next:

Click Next then Finish

Connecting an ACI Fabric to an External Routed Network

This is performed by setting up what is called the L3 Out connection.

  • is a construct that connects the ACI fabric to one or more external router(s) for the purpose of exchanging routes. The interconnection can be on one or more border leafs.
  • the interconnection subnet between ACI fabric and the external router(s) is on the ACI fabric side configured as either:
    • regular L3 port (aka routed port), or
    • SVI, or
    • sub-interface.
  • In an ACI multi-tenant design we can:
    • either attach one L3 Out connection to each tenant, or
    • we can centralize one L3 Out connection on a shared tenant (either the tenant common or any tenant that provides shared services to other tenants)
      • if the external routed network is shared, then we need to configure contracts between the external routed network and the EPGs, and the scope of the contracts must be set to VRFor global, depending on whether the EPGs are assigned the same VRF or are in different tenants.
  • ACI fabric makes adjacencies with the external routers, e.g. OSPF adjacency, BGP adjacency, etc.
  • is defined unter the Networking tab.
  • we can control (allow or filter out) route prefixes into and out of the ACI fabric
  • requires the configuration beforehand of:
  • requires the definition of an external network, aka External EPG, which itself will contain the list of non-ACI network prefixes that will be visible from within the ACI fabric. Many companies define only one external EPG and within it they define the route 0.0.0.0/0
  • An external entity (router, switch, server…) is attached to the ACI fabric through:
    • one port
    • vPC
    • Port Channel

NB

  • Both Physical and Virtual Domains require VLAN pool(s) each.
  • One difference between virtual and physical Domains: a physical Domain does not use a virtual machine manager.
  • When we don’t fully integrate a virtual workload to ACI using VMM Integration, then the virtual workload is treated in ACI similarly to a physical workload.
  • When defining a new VMM Domain, you have the choice between VMware DVS or Cisco AVS.
    If you opt for AVS, then a couple of requirements must be met beforehand: ACI must be installed and all fabric switches registered
    minimum MTU value is 1600, and must be configured on all devices in the path between ACI fabric and AVS.

Conclusion

Cisco ACI was meant not to be a silo technology, but rather a building block that connects and integrates to physical and virtual workload platforms. ACI integrates with existing L2 and L3 network blocks too.

How is ACI in your organization connected to the rest of the network?

Categories: Cisco DCACI

Keyboard Banger

Keyboard Banger is a network engineer from Africa. He has been working in network support and administration since 2008. He started writing study notes about certification exams and technology topics a couple of years ago. When he's not writing articles, he can be found wandering on technical forums.

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *