Cisco ACI Notes

Published by Keyboard Banger on

This is a collection of my Cisco ACI notes during my studies.

  • unenforced network: you get this type of network when you set the “Policy Control Enforcement Preference” to “Unenforced” at the VRF level. This disables the default security policiy on the VRF, which means all EPGs associated with the VRF can theoretically communicate together.
  • ACI builds by default a zero-based trust network, i.e communication is not allowed unless specified, which is opposite to the traditional network which is trust-based.
  • When we don’t fully integrate a virtual workload to ACI using VMM Integration, then the virtual workload is treated in ACI similarly to a physical workload.
  • Cisco AVS, Application Virtual Switch, is a multihypervisor virtual switch that comes with ACI free of charge and is built on the successful 1000V switch. It provides additional features to ACI. It is completely managed by ACI, unlike the 1000V that was managed by a Virtual Supervisor Module VSM.
  • When integrating vmware virtual workloads with ACI, you have the choice between DVS or Cisco AVS.
  • If you opt for AVS, then a couple of requirements must be met beforehand:
    • ACI must be installed and all fabric switches registered
    • minimum MTU value is 1600, and must be configured on all devices in the path between ACI fabric and AVS.
  • When integrating a physical workload (bare-metal servers) with ACI we should very likely configure policies on each physical NIC or virtual NIC on the server.
  • To integrate ACI with Microsoft platforms, we have two options:
    • integration with Microsoft SCVMM
    • integration with Azure Pack
      • provides ready-to-use management portal and admisitrator portal
      • reflects the same experience as Microsoft Azure cloud.
  • when companies migrate from a traditional network to ACI, they follow a staged approach. Often, they opt for the network-centric mode first. Second, they gradually migrate to the application-centric mode going through the hybrid mode. they can start by creating a Bridge Domain and subnet for every VLAN and put their servers that were in this VLAN on an EPG. Then they can gradually group those servers according to another criteria such as by application or by business need, rather than by VLAN as it was in the traditional network.
  • The network-centric mode
    • can be a one-tenant or a multi-tenant setup.
    • in a multi-tenant design, if you want your tenants to communicate in the future, then you must design non-verlapping subnets in them. You can not let overlapping subnets in your tenants and want to have communication between your tenants because in this case you need a certain address translation function between both tenants, and ACI does not support NAT to this day.
    • in this mode we take our existing VLANs and subnets from the old network and create them in ACI; i.e. we “reproduce” the network in ACI.
    • in a single tenant design, L2out and L3out can be placed in your tenant or in the tenant Common. But the recommendation is to place it in the tenant Common. Reason: you may want in the future to add further tenants and with the L2out and L3out already placed in the common tenant, you won’t be forced to redesign your tenants.
  • The hybrid mode is the combination of the network-centric mode and some features from the application-centric mode.
  • Blacklist model vs whitelist model: in an organisation the corporate security guidelines and policies follow one of the following security models: blacklist or whitelist. In a blacklist model everything is open unless specifically denied. In the whitelist model every communication is denied unless specifically authorized. A quick analogy to understand the whitelist model is Cisco IOS Access Lists. wh
  • There an entry “deny any” at the end of the ACL. Cisco ACI employs the whitelist model by default.
  • A packet in the fabric is encapsulated within an UDP datagram
  • VXLAN
    • VXLAN in the Cisco ACI fabric is different from the standard VXLAN protocol.
    • VXLAN offers 16 Million subnets. Each segment is distinguished with a VNI VXLAN Network Identifier.
    • VNI defines a L2 broadcast segment or a L3 context
    • There is a VNI-to-VLAN mapping on each fabric leaf.
    • End Hosts attached to leafs can communicate together using VLANs or VXLAN, depending on the design. Communication between Leafs/Spines uses VXLAN.
  • BUM traffic
    • Unknown Unicast traffic is handeled in two ways
      • flood
      • Hardware Proxy: unknown unicast frames are forwarded to the spines, where the addresses are analyzed. If there is no match in the table the frames are dropped.
    • Unknown Multicast traffic is a multicast traffic crossing the ACI fabric without a IGMP Join message.
  • ARP traffic in the ACI fabric is handeled in two ways:
    • flood: MAC addresses are learned from the L2 traffic
    • Unicast routing: MAC addresses are learned from L2 traffic and IP addresses are learned from L3 traffic
  • VLAN allocation in APIC is either static or dynamic. Dynamic allocation is recommended.
  • To be able to logically connect a physical server to ACI fabric you need to configure a Physical Domain on APIC
  • To be able to logically connect a virtualized server to ACI fabric you nedd to configure a VMM Domain on APIC.
  • VLAN pool
    • is the VLAN range that will be assigned statically or dynamically
    • recommended to design VLAN pools based on funtional role, e.g Firewall_VLP.
    • attaches to one or more domains. Beware that when using the same VLAN Pool for more than one domain, the VLAN significance is local to each domain. The recommendation however is not to reuse the same VLAN Pool.
  • Domains
    • define where and how to use a VLAN Pool
    • for each VLAN Pool you define one domain: 1:1 mapping.
    • there are 4 types of domains: physical, virtual, L2out, L3out.
    • EPG(s) must be attached to a domain.
    • a domain can attach one or more EPGs.
  • Physical Domain:
    • connects a physical external entity to the fabric
    • Cisco recommends one physical domain when connecting the ACI fabric to an external network
  • Virtual Domain aka VMM Domain
    • connects a virtualized server (aka virtualization host) to the fabric.
    • is configured on APIC as a VMM Domain.
    • connects a Virtual Machine Manager such as Vmware vCenter or Microsoft SCVMM to APIC.
      • once you create a VMM Domain on APIC, a virtual switch is created on the virtualized server, and the virtual environment parameters (vCenter name, vNICs, list of VMs) becomes visible on the ACI.
      • must be assigned a VLAN or VXLAN pool. The VLAN pool will provide the VLAN IDs that will be assigned dynamically to Port Groups on the vSphere Virtual Switch for example in case of Vmware integration with ACI.
  • Both Physical and Virtual Domains require VLAN pool(s) each.
  • One difference between virtual and physical Domains: a physical Domain does not use a virtual machine manager.
  • L2out domain and L3out domain:
    • are L2 and L3 connections from ACI fabric to the core network of the company.
    • In a multi-tenant design we can:
      • either attach one L3out connection to each tenant, or
      • we can centralize one L3out connection on tenant Common.
  • At the level of L2out and L3out:
    • ACI fabric makes adjacencies with the core network
    • we can control (allow or filter out) route prefixes into and out of the ACI fabric
  • When a physical server is connecting to the ACI fabric, then configure static VLAN mapping.
  • When a virtualized server is connecting to the ACI fabric, then use dynamic VLAN allocation.
  • Physical workload: is a subset of compute, storage and network ressources dedicated to a single entity. In the IT industry we distinguish physical workloads and virtual workloads. A Physical workload ist then the subset of compute, storage and network dedicated to physical machine(s). A Virtual Workload is the same subset being dedicated to virtual machine(s).
  • An external entity (router, switch, server…) is attached to the ACI fabric through:
    • one port
    • vPC
    • Port Channel
  • Bridge Domain
    • is a container of subnets, i.e. we define the IP subnets here.
    • we define subnets with their gateway IP addresses.
    • we can group subnets altogether in a same bridge domain, or separate them in different bridge domains. The second approach is necessary if you need to place firewall policies between subnets.
    • points to one and only one VRF.
    • can be configured without a VRF (APIC GUI allows it), which can be added later.
    • on the APIC, it is configured under Tenant -> Networking.
    • can be created through context menus (on the left) or graphically with drag and drop. If there is more than one VRF already created, then you must pay attention while dragging the bridge domain symbol: you must release the bridge domain symbol over the desired VRF.
  • AEP: Attach Access Entity Profile (see how to configure AEP in ACI)
    • is required for attaching the external entity to the fabric.
    • links an infrastructure policy group to fabric interface(s), where an external entity connects. External entities with similar infrastructure policy requirements should be assigned the same AEP.
    • encapsulates one or more Domains, so it is a one-to-many relationship.
    • requires a VLAN Pool to be associated to the domain.
  • Tenants
    • provide security by isolating what is defined under tenant A from tenant B.
    • With tenants we can run many logical networks on the same physical network
    • provide a separation of the control and management plane, ie each tenant has its own control plane and management plane.
    • There are various tenant design possibilities. We can design our fabric to run with a single tenant or with many. Possible criteria for a multi-tenant design, can be:
      • department naming: Sales, Engineering, Tests
      • location in the product lifecycle: Production, Test-Env,
      • security zones: standard, DMZ,…
      • etc.
    • There are default tenants already preconfigured on the APIC:
      • management tenant
      • infrastructure tenant
      • common tenant: here we can define common network policies and services that will be used across tenants. Some of these services could be DHCP, DNS, Active Directory, etc.
    • in ACI, management of tenants can be performed on a per-tenant basis. And we can assign tenant management on a user or group basis.
  • Infrastructure administrator vs tenant administrator
    • Infrastructure administrator manages and controls VLAN namespaces for all tenants. He has access to all tenants.
    • Tenant administrator has access only to his allowed tenant(s) and his/their ressources.
  • NTP must be configured and synchronized on APIC and all fabric nodes. Here is a quick tutorial on setting up NTP on ACI.
  • New nodes being added to the fabric are automatically discovered by APIC through LLDP. As soon as they pop up in the APIC GUI Interface you can add or block them from joining the fabric, based on their Serial Numbers.
  • New fabric nodes send DHCP requests and receive replies from APIC.
  • APIC sends TEP addresses to the new leafs
  • VTEP aka TEP
    • Tunnel EndPoint addresses
    • the address pool is laid down during initial APIC setup and is recommended to be a /16 or a /17 subnet. By default it is 10.0.0.0/16 subnet. ACI version 2 allows to have a VTEP address pool of a /22 subnet.
  • GIPO
    • Global IP Outside
    • used to propagate fabric-internal multicast traffic
  • Giving lower numerical IDs to the spines is recommended. The subsequent higher IDs should be reserved for the leafs.
  • All fabric nodes and APICs should be connected to an OOB network for management purposes.
  • Access to leaf switches through console cable is possible but offers only read capabilities.
  • OS image management occurs on the APIC, which supports TFTP
  • in ACI there is no need to:
    • configure loopback addresses on new switches
    • configure IGP protocol and neighborships
    • configure custom routing timers
    • configure list of allowed VLANs on trunks.
  • Swtiches in a Pod share the same VTEP prefix
  • VRF, aka private network:
    • pronounced V-R-F or way fancier “Vurf” :)
    • is equivalent to the legacy VRF concept.
    • is pointed by one or more bridge domains
    • can be created with context menus or graphically with drag and drop
    • you can configure the VRF on ACI first or the bridge domain first. The order does not matter because you can attach the bridge domain to the VRF later.
    • Subnets must be unique only VRF-wide. It means:
      • you can’t have a subnet Subnet_S1 on two bridge domains belonging to the same VRF
      • VRF_A and VRF_D can both have a subnet with the same subnet range of IP addresses.
    • APIC can be instructed to confine the subnets within a given VRF, or to propagate them to other VRFs, or to allow redistributing them to the outside network through a L3out connection.
    • APIC automatically creates an Infrastructure VRF to communicate with fabric switches
  • Management of the fabric can be performed also using an external management station connected to the fabric on tenant “mgmt”. In this scenario you must:
    • configure a VLAN Pool, an AEP, a phyiscal domain
    • assign the VLAN Pool to the domain
    • encapsulate the domain under the AEP
  • Application Network Profile, aka ANP or Application Profile
    • must be configured before even creating EPGs
    • while creating the Application Network Profile you can create EPGs as well. Or you can leave creating EPGs later on.
  • End Point Groups, aka EPG
    • while creating an EPG, a bridge domain must be associated to it.
    • on the APIC GUI, using the Topology tab: after dropping an EPG symbol in the window and configuring it, it will not be created unless you press the Submit button.
    • Between EPGs all communication is by default denied. This is the default behaviour when you configure a VRF. That means that ACI acts like a firewall at line rate denying traffic between EPGs.
    • can be attached to one or more VMM Domains
    • when attached to a VMM Vmware Domain, then Port Groups will be automatically created on the vSphere Virtual Switch. The VMware administrator will then have to manually assign VMs to Port Groups.
    • has the option to activate/deactivate Microsegmentation during the configuration menu of the EPG itself.
  • Provisioning a switch port in traditional networks is completely different from the ACI world:
    • in a traditional switch you configure interfaces separately
    • in ACI, you configure many constructs and objects at first, sch as domain, AEP, VLAN Pool, Switch Profile, Interface Profile… which may seem a burden at first. But its power lays with its flexibility and extensibility. For example if you want to add an interface with similar configuration to a previous one, simply add it to the Interface Profile.
  • an Application in the ACI model ist not a virtual/physical machine, but the combination of:
    • workloads, either physical or virtual
    • L2 – L7 policies: VLANs, subnets, L4 ports, ACL, QoS policies, filtering policies, load balancing policies,…
  • ACI fabric contains 2 to 6 spines: 2, 4, 6.
  • ACI fabric operates on a whitelist model: no communication is allowed unless specified.
  • Frames in ACI are routed, but the L2 switching semantics are preserved.
  • Contracts:
    • policies i.e rules intended to regulate the communication between EPGs. The policies include one or more of the following constructs:
      • permit
      • deny
      • log
      • mark
      • QoS
      • redirect
      • service graph
    • can be as simple as one rule or complex. Complex contracts contain Subjects, Labels, Filters and Actions.
      • a Subject can be seen as a function. An example of this would be a web server providing HTTP, HTTPS and FTP services.
    • can be grouped together in a bundle; an EPG providing the “services” ist said to be a Provider, or Provider EPG. The EPG benefitting from the “services” ist said to be a Consumer, thus a Consumer EPG.
    • can be leveraged by more than one EPG, i.e an EPG A and an EPG B point to the same Contract.
    • can be defined globally on the fabric, or under a tenant.
    • a particular kind of contracts is called Taboo. A Taboo defines a list of deny actions and is applied to an EPG, i.e that EPG will be denied access to these ressources. Taboos enforce the black list model which I described in the beginning of this post. Taboos are especially used by clients that are using the blacklist model already and performing a migration to ACI infrastructure.
  • L4-7 Service insertion
    • is the process of introducing L4-7 services in the data path of a packet in ACI fabric
    • requires a service graph and the mapping of it to concrete devices
    • connected L4-7 devices are managed in either:
      • managed mode:
        • a complete horizontal integration is performed.
        • ACI pushes policies to the device and redirects traffic to it.
        • advantage of providing company-wide consistent policies
        • advantage of deploying the L4-7services anywhere in the fabric without caring about the physical location of the device.
        • if the device provides contexts, then access to the admin context must be configured on ACI.
        • ACI dynamically manages VLAN assignment
        • ACI collects statistical data such as health scores
      • unmanaged mode: policies are not managed by ACI. They are created by a the L4-7 device administrator.
    • in ACI we can choose one of three L4-7 Service Insertion modes:
      • with device package: this is the full integration of the L4-7 device with ACI fabric. The device package is developed by the third-party vendor and includes two files: an XML files describing the capabilities of the device, and a python file describing the integration with ACI.
      • service manager mode: security policies are defined on the L4-7 device by its own administrator. Then the policies are integrated and orchestrated by ACI
      • no device package or service manager: the L4-7 device is completely managed by its own administrator. ACI administrator only creates the necessary VLANs. This modes is helpful in “political” environments, where each department still wants to completely manage its equipement.
  • Service graph
    • is a collection of Abstract Nodes
    • is connected to EPG with a contract
  • Meta device
    • is a symbolic representation of the L4-7 device that is connecting to the fabric
    • its function, whether it is a firewall, or a load balancer or else, is defined in the Abstract Node
  • Logical Device (LDev)
    • is a cluster of two or more Concrete Devices
  • Concrete Device (CDev)
    • is a one-to-one representation of the L4-7 device
    • Interfaces of the Concrete Device (vnsCIf) represent one-to-one the interfaces of the real device but in the format {slot_port}, like interface 1_2 or interface 1_4
    • Each Interfac on the Concrete Device maps to an interface on the Logical Device (LIf)
  • ACI fabric design on multiple sites:
    • stretched fabric: the ACI fabric is stretched on both sites. We have always one APIC cluster: one APIC is installed on one site and two APIC on the other site. Some leafs from site A physically connect to some spines of site B.
    • multi-pod: considered an evolution of stretched fabric. It involves an InterPod Network IPN. Communication between spines of both Pods occurs with MultiProtocol BGP over Ethernet VPN (MP-BGP EVPN)
    • dual fabric design: each site has its own APIC cluster and own ACI fabric. Both ACI fabrics are connected over the L2 or L3 networks, which are carried by some leafs at each site.
    • multi-site design: this is an evolution of the dal fabric design. Both ACI fabrics are connected over the WAN. The WAN is connected at the spines of each site.
  • MP-BGP
    • is used to carry routing information to and from the fabric
    • internal MP-BGP and iBGP use the same ASN
    • runs on selected spines chosen by the APIC administrator
  • Infrastructure VLAN
    • is used solely within the fabric
    • must be unique on the whole network, including end host VLANs
    • recommended but not mandatory: use VLAN ID 3967.
  • Initial install of an APIC
    • initially we must configure CIMC with certain parameters. Once CIMC IP settings are done, we can configure APIC further with the CIMC virtual console (CIMC GUI) or through a direct connection (keyboard and mouse).
    • physically APIC requires 2x 10G connections to the fabric, 2x connections to the Out-of-Band management network and one CIMC connection.
    • configure parameters such as
      • Controller ID
      • Controller Name
      • Fabric Name
      • Infrastructure VLAN
      • VTEP address pool
      • Out-of-Band Management IP address and gateway
      • Multicast group address
    • once the following parameters are set up during the script install, you can not change them later unless you rebuild the fabric:
      • Infrastructure VLAN
      • VTEP address pool
    • APIC are staged one after the other, unless we are in a multipod setting, where not all APICs are on the same physical location. In this case we must set up the Pod environments first then stage the remote APICs.
    • always provision 3 or 5 APICs per cluster, in order to preserve the minority/majority vote and avoid split-brain APIC scenarios.
    • APIC clusters of 5 allow to have active/standby operation: 3 APICs are active and 2 are standby.
    • Always have a minimum of 3 active APICs in an APIC cluster.
  • ACI Basic vs Advanced GUI
    • Basic GUI
      • use cases:
        • for small ACI deployments
        • for network administrators who do not need full ACI features such as L4-7 integration.
      • allows configuration of tenants, leaf ports and access profiles
      • allows to configure one port at a time
      • is above ACI v3.0 not supported anymore
    • Advanced GUI
      • allows to configure multiple ports through the access selector and Interface Profiles
      • recommended to be used.
  • Extending the ACI fabric:
    • new leafs should have a leaf ID numbered higher than 100
    • new spines should have a Spine ID numbered higher than 200.
  • Fabric Extenders can be attached to ACI fabric. Not all FEXs are supported by ACI. A FEX can attach to only one leaf.
  • Tenant policies:
    • define the behaviour of an application whenever traffic hits the fabric
    • trigger the access policies
  • Fabric policies:
    • necessary for the internal working of the fabric.
    • example: NTP
  • Use a naming convention that is consistent across the fabric(s)
  • Access policies:
    • perform the configuration of the physical interfaces
    • are not active on the ports until triggered by a tenant policy
  • Interface Policy Groups
    • no more than one Interface Policy Group should be assigned per port Channel or vPC  (1:1 mapping)
  • Interface Profiles
    • aka Access Port Selectors
    • there are Leaf Interface Profiles and Spine Interface Profiles
    • select interfaces on one or more switches
    • an interface profile is only an interface selector; i.e it does not configure interfaces.
  • Switch Profiles
    • select one or more switches from which the ports will be configured
    • a switch profile is only a switch selector; i.e. it does not configure switches
  • the chronological order of configuring access policies is:
    • 1- VLAN Pools
    • 2- Domains
    • 3- AEP
    • 4- Interface policy groups
    • 5- Interface profiles
    • 6- Switch profiles
  • APIC firmware upgrade:
    • you define an APIC firmware policy first, that defines when to perform the upgrade
    • you can launch the firmware upgrade immediately or plan it using a Scheduler
  • Switch firmware upgrade
    • you define maintenance groups and maintenance policies
    • Clients with critical ACI environments use the four-group method; i.e. they define 4 maintenance groups and start by upgrading Leaf groups first;
      • red Leaf group
      • blue Leaf group
      • red Spine group
      • blue Spine group
    • The maintenance policy defines when and how to perform the upgrade:
        • the “when”: immediately or using a Scheduler
        • the “how”: the upgrade process if launched must obey the following rules:
          • not exceeding the concurrCap value, which is the maximum simultaneous switches being upgraded at a time
          • only one switch per VPC
  • Configuration management in ACI is performed with either snapshots or backups.
    • Snapshots
      • are fast (couple of clicks) to store a config or to restore it
      • store settings of the fabric or of the tenants
      • do not store the complete configuration
    • Backups
      • used to store and restore the ACI configuration
      • require an external server
      • in a backup operation, a backup file is generated. In a restore, an import file is needed
      • require an export policy in the save operation, and an import policy in the restore operation.
      • two types of backup/restore operation:
        • best effort:
          • when a difference in Shads is encountered, the portion of data is ignored.
          • when the imported config belongs to an ACI version is different from the current running, the difference is ignored.
        • atomic:
          • when a difference in Shads is encountered, the portion of data is ignored.
          • when the imported config belongs to an ACI version different from the current running, the backup/restore operation is aborted.
  • Endpoint learning
    • there are three so-called Station tables
      • local station table:
        • each leaf has a local station table
        • contains all endpoints connected to the local leaf
      • global station table
        • each leaf has a global station table
        • contains cached information about some remote endpoints. Leafs are not supposed to possess forwading information about all endpoints in the fabric.
      • proxy station table
        • resides on the spines
        • all the spines have the same proxy station table
        • contains forwarding information about all endpoints attached to the leafs.
  • at the leaf ingress, the packet is always encapsulated into a Cisco VXLAN packet and routed in the overlay network using IS-IS protocol. At the leaf egress, the packet is decapsulated from Cisco VXLAN and encapsulated to whichever the encapsulation at the endpoint is (VLAN, VXLAN, NVGRE)
  • Software overlay network: the logically built overlay network between virtual switches located on hypervisors
    • When a virtualized server is a dual hypervisor, then each hypervisor runs its own software network overlay, and both network overlays do not communicate with each other.
    • the software overlay network does not communicate with the physical network either unless a software gateway is installed.
  • Dockers (equivalent to VM) in the Linux Docker technology doe not have their own TCP/IP stack but rather a namespace in the TCP/IP stack of the host machine.
  • OpFlex: determines if a virtual switch on the hypervisor is connected to a leaf port. Remember that a virtual switch connects to physical NIC ports of the virtualization host, and the physical NIC ports connect to the ACI fabric.
  • Microsegmentation EPG
    • The purpose of this feature is to automate the assignment of selected Virtual Machines to a particular EPG using rules, instead of the VMware administrator having to manually assigning them.
    • Each rule is in the format “match-any | match-all {u-attribute}”, where u-attributes are the microsegmentation attributes
    • the list of available u-attributes of an uEPG attaching to a physical domain: only two:
      • IP Address
      • MAC Address
    • the list of available u-attributes of an uEPG attaching to a VMM domain is richer:
      • IP Address
      • MAC Address
      • VM Name
      • VM OS
      • VM tag
    • a rule can be a pure “match-any” filter, a pure “match-all” filter, or a combination of both.
    • if there are many clauses in the rule, than beware of the precedence among the u-attributes, e.g. the u-attribute “VM Name” has a higher precedence than “VM tag”. So if the u-attribute “VM Name” matches first, further clauses of the rule won’t be inspected by APIC.
    • available for both physical and VMM Domains
    • aka uEPG

I’ve distilled all these Notes from:

Categories: Cisco ACI

Keyboard Banger

Keyboard Banger is a network engineer from Africa. He has been working in network support and administration since 2008. He started writing study notes about certification exams and technology topics a couple of years ago. When he's not writing articles, he can be found wandering on technical forums.

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *