Network Experiences

VXLAN Reference

VXLAN (Virtual Extensible LAN) is a network virtualization technology designed to address scalability limitations of VLANs, creating logical Layer 2 networks over a Layer 3 IP-based network.

Purpose: Enables scalable, flexible, and efficient network segmentation and multi-tenant architectures for modern data centers.

Key Features and Benefits

Scalability
  • VLAN Limitation: Traditional VLANs (IEEE 802.1Q) are restricted to 4,096 IDs, insufficient for large environments.
  • VXLAN Solution: Uses a 24-bit VXLAN Network Identifier (VNI), enabling up to 16 million unique segments.
Seamless Mobility
  • VM Mobility: VXLAN allows virtual machines (VMs) to move across different physical servers or locations without changing their IP addresses or MAC addresses, ensuring seamless mobility. This is especially beneficial in dynamic cloud environments where VMs might need to be moved between hosts frequently.
Tenant Isolation
  • Multi-Tenant Support: Each VNI represents a unique, isolated Layer 2 network, providing strong network segmentation, which is particularly valuable in multi-tenant data centers.
Layer 2 Over Layer 3
  • Encapsulation: Encapsulates Layer 2 frames within Layer 3 UDP packets, enabling:
  • Extension of Layer 2 networks across Layer 3 infrastructure.
  • Features like ECMP for traffic engineering, improving resiliency and load balancing.
Elimination of Spanning Tree Protocol
  • Since VXLAN operates over Layer 3, it avoids issues with Spanning Tree Protocol, reducing complexity in large-scale environments.
Multicast/Broadcast Handling

Efficient handling of multicast/broadcast traffic using IP multicast in the underlay.

Multi-Data Center Deployments
  • DC Interconnect (DCI): Enables Layer 2 communication across geographically distributed data centers. Applications, VMs, and storage systems can interact as if they were on the same Layer 2 network.
  • Disaster Recovery: Facilitates workload failover between data centers without requiring network reconfiguration, making it a key enabler of disaster recovery solutions.
Key Use Cases – When is VXLAN needed
  1. Multi-Tenant Data Centers: Network isolation for different tenants in cloud environments.
  2. Network Virtualization: Integral to platforms like VMware NSX and OpenStack.
  3. Extending Layer 2 Across Data Centers: Maintains connectivity between geographically dispersed sites.
  4. Software-Defined Networking (SDN): Enables network segmentation and automation at scale.

Scenarios for Deployment

Single Data Center
  • Overcomes VLAN scalability limits.
  • Enables Layer 2 extension across racks or pods.
  • Leverages Layer 3 traffic engineering.
Multiple Data Centers
  • Provides Layer 2 extension and workload mobility between geographically dispersed locations.
  • Essential for disaster recovery and multi-site applications.

Is VXLAN Only Needed for Multiple DCs?

  • No, VXLAN is not only needed for multi-data center environments. It is also highly valuable within a single data center to provide scalability, flexibility, mobility, and network isolation. In multi-data center scenarios, VXLAN becomes essential for Layer 2 extension and workload mobility across geographically dispersed locations, ensuring seamless operation between multiple sites. Even within a single data center, VXLAN can be highly beneficial:
    • To overcome VLAN scalability limits.
    • To achieve better Layer 2 extension and workload mobility across different racks or pods.
    • To leverage Layer 3 traffic engineering benefits in a large-scale, highly virtualized environment.

Technical Components

VNI – VXLAN Network Identifier:
  • 24-bit identifier for unique network segments.
  • allows for up to 16 million VXLAN segments, each functioning as an isolated Layer 2 network. The VNI is used to differentiate between different VXLAN segments.
VTEP – VXLAN Tunnel Endpoints:

Switches and routers that participate in VXLAN have a special interface called a VTEP. The VTEP provides the connection between the underlay and the overlay. VTEPs are the devices (typically switches or routers) responsible for encapsulating and decapsulating VXLAN packets. Each VTEP maps the Layer 2 VLAN traffic into Layer 3 VXLAN traffic and vice versa.

  • Source VTEP: Encapsulates Ethernet frames from the virtual network into UDP packets for transmission across the IP network.
  • Destination VTEP: Decapsulates VXLAN packets back into Ethernet frames for delivery to the target device in the Layer 2 network.
  • Encapsulation: Original Layer 2 frames are encapsulated with VXLAN, UDP, and IP headers for transport across the underlay.
Packet Format

A VXLAN packet consists of the following headers:

  • Original Ethernet Frame: The payload that is encapsulated.
  • VXLAN Header: Contains the VNI to distinguish different virtual networks.
  • UDP Header: VXLAN uses UDP (User Datagram Protocol) for encapsulation, typically with port 4789.
  • IP Header: Encapsulates the packet with a source and destination IP address for Layer 3 transport.
  • Outer Ethernet Frame: The outermost Layer 2 frame used for physical delivery of the packet across the network.
Control Plane
  • Flood-and-Learn Model: Original VXLAN relied on IP multicast.
  • BGP EVPN: Modern VXLAN deployments use EVPN for dynamic and scalable MAC/IP address distribution, replacing the flood-and-learn approach.
Overlay Network
  • Creates a logical Layer 2 network over an IP-based underlay.
  • The underlying IP network (referred to as the “underlay”) forwards VXLAN-encapsulated traffic between VTEPs, while the overlay maintains logical Layer 2 connectivity between devices.
  • Uses IP multicast or replication for efficient broadcast/multicast traffic handling.
Learning and Forwarding
  • MAC Address Learning: In VXLAN, each VTEP learns the MAC addresses of devices in its local network and maps them to corresponding VNIs and remote VTEPs. This is similar to how traditional Layer 2 switches learn MAC addresses.
  • Unicast and Multicast:
    • Unicast Traffic: Encapsulated and forwarded to the remote VTEP directly based on the destination MAC address and IP address learned by the source VTEP.
    • Multicast/Broadcast Traffic: VXLAN supports forwarding Layer 2 multicast and broadcast traffic by leveraging IP multicast in the underlay network.

Challenges

  • Packet Overhead: Additional headers increase packet size, potentially leading to fragmentation and as a result the MTU needs to be increased
  • Control Plane Complexity: While VXLAN’s data plane is simple, the control plane can be complex without protocols like EVPN to handle MAC address learning and distribution.
  • Multicast Dependency: The original VXLAN model relied on IP multicast for broadcast and unknown unicast traffic, which can be inefficient or unavailable in some network environments. Solutions like EVPN help overcome this.

Data Plane vs. Control Plane

VXLAN (Data Plane)
  • Role: Encapsulates Layer 2 frames for transport across Layer 3.
  • Function: It is responsible for the actual forwarding of data packets. VXLAN tunnels the traffic by encapsulating it using UDP, allowing it to traverse a Layer 3 network as if it were on the same Layer 2 segment.
  • Tenant Isolation: VNIs (VXLAN Network Identifiers) are used to segregate different tenants’ traffic at the data plane level. The VXLAN data plane isolates tenants by keeping each tenant’s traffic within its designated VXLAN tunnel (based on the VNI).
BGP EVPN (Control Plane)

Role: BGP EVPN manages and distributes the reachability information (MAC and IP addresses) across the network.

  • Function: It advertises MAC-to-IP mappings, and routes between different switches and devices, making sure that the network is aware of how to reach different endpoints efficiently. This replaces the traditional flood-and-learn approach with a more scalable, efficient method.
  • Tenant Routing and Segmentation: BGP EVPN is responsible for signaling which VNIs belong to which tenants and for handling routing across VXLAN segments. It integrates with VRFs to provide Layer 3 routing segmentation, ensuring that tenant traffic remains isolated not only at Layer 2 (VXLAN) but also at Layer 3 (routing).
Summary:
  • VXLAN: Handles the data plane by transporting packets through the network.
  • BGP EVPN: Manages the control plane by distributing the MAC/IP information and routing the packets efficiently across the VXLAN network.

In