All about the path

Recently on the twittersphere there was a short exchange  as to why would  a security professional care about VXLAN when they don’t care about ASICs in a switch.

In stead of putting the conversation up (I’m lazy and can’t be bothered screen capturing it) I thought I’d share my $0.10 worth here.

In short, you always need to be concerned about the path of data. Just like Network Architects want to know packet paths for engineering purposes a Security Professional also is concerned with what systems or processes does it cross and where are the enforcement (choke) points. But most importantly, you need to know the data path so you can secure it.

I’ll start by briefly explaining what VXLAN is, its deficiencies and then a quick look as to why a Security professional needs to be concerned with the data paths, irrespective of whether or not they are virtual or physical.

What is VXLAN?

VXLAN (IETF REF: is “A Framework for Overlaying Virtualised Layer 2 Networks over Layer 3 Networks“. This is a VMware and Cisco (amongst others) initiative. That’s the fluffy way of saying it is a tunneling protocol, more precisely a L4 tunneling protocol.

Why have VXLAN?

Personally I wouldn’t. It is a VMware kludge to a very specific problem. However, if you are a VMware proponent then you have your uses. For any one dealing with large virtualised environments, especially multi-tennant ones, you will find that you run into the limitations of your L2 network (even a well designed one) really quickly; 4000 VLANs doesn’t go far in multi-tennant environments .

  • How do you migrate VMs across L3 boundaries?
  • How do you move devices about without changing IP addresses?
  • Do all application support the readdressing of VMs?

Let’s take a fairly simple example of vMotion; this is not the ideal example but sufficient to get the concept across. vMotion is the movement of one Virtual Machine (VM) from a physical server to another physical server. Figure 1, below, shows a fairly simple setup with servers connected to the same switch, which we will assume are in the same Layer 2 Domain.

Figure 1 – Servers with VMs in a single L2 Domain

I won’t go into the gratuitous detail on how vMotion works, you can look that up for yourself, but for the VM to move from one physical machine to another there needs to be Layer 2 (L2) adjacency. Figure 2 below shows how the machine is replicated. The path between the virtual Machine on the left (highlighted in Green); through the Hypervisor (in this instance I’ve used VMware) out the physical NIC into the switch; out the switch; into the first NIC of the second server; up through it’s Hypervisor and finally coming to rest.

Stay with me there is a point to all of this.. I hope.

Figure 2 – VMotion path through switch.

Now, throw a router in the middle (see figure 3) and the whole process breaks. Because there isn’t the ability for L2 adjacency, it crosses the L3 boundary, the machine cannot move, let alone retain its IP address, etc.

Figure 3 – VM Servers in two L2 domains, separated by router.


As mentioned previously VXLAN is essentially a tunneling protocol. It is taking a Layer 2 frames and wrapping it in a Layer 4 Datagram, Figure 4 depicts 2 VXLAN networks, green and pink. This, with some other wizardry gets over some of the aforementioned limitations, the virtual machines are tied to a VXLAN ID by the Hypervisor (they need to be the same) which is then passed to the VTEP (VXLAN Tunnel End Point) which then wraps everything in a UDP packet (OK that is slightly simplified).

Figure 4 – VXLAN allows pseudo L2 connectivity across L3.

Now there is an L2 adjacency between the two servers, they can now replicate VMs between each other whilst maintaining the same L2 and L3 addresses. See Figure 5.

Figure 5 – vMotion across L3 boundaries is now possible with VXLAN.

Security Issues:

This is where I hope the rant has some value – If you didn’t care how the protocol works (even in a rudimentary sense) or how the data flowed in the scenarios above you wouldn’t understand what the limitations, and therefore potential security flaws, are or understand what issues it solves. That said VXLAN doesn’t solve any of the existing security issues we find in Ethernet today. It still requires processes to lock down deficiencies in the technology.

  1. Just like Ethernet you can spoofing ARPs;
  2. Broadcast storms still possible (turned into multicast within VXLAN domain);
  3. No security mechanisms built into VXLAN;
  4. If you have access to the network you can spoof anything? If you are on the same network as a VXLAN segment; and
  5. Through the very nature of the VXLAN function you are also now removed from the ability to provide hardware handoff for L2-L7 security measures (think ACLs, VACLs, Firewalls, NAT, Load-balancing, etc). You need to have this gated between VXLAN domains via a VM that spans both.

Even ignoring all these new issues (yes they are new issues, is this a new way of accessing an environment, of course it is!) what about policies, enforcement points, network taps, etc, that may be already in place, in-line/path with the new VXLAN tunnel that you are proposing?

These are all why a security engineer/architect/designer needs to care about the path.

Now to the comment “security guys don’t care about the path through a switch’s ASIC…”, generally speaking, most won’t care, but they should. Just because it is a switch doesn’t mean that nothing happens to a packet once it goes in and then goes out the other side. ASICs are customised, programmable chips that allow the device to perform a number of different functions; if you’ve ever looked into this you will know that no two platforms are the same (even from the same vendor).

In the diagram below (which is a fictitious switch construct) you can see that there are some capabilities built into the different components of the switch, specifically the ASICs.

Getting traffic from one port to another port isn’t necessarily a simple thing. Can I get straight from Eth1 to Eth2 directly, or do I have to go up through the switching fabric to the L2 switching engine? What happens when I have to route (below shows a path of a packet when going via the routing engine)? Does, or can, the packet be modified/duplicated inside the switch and if so what happens? At what point in the packet’s path through the switch is policy enforced, and in what order?

The list goes on and on.

Most enterprise switches, even basic ones, will allow you to mirror ports, I’m pretty sure that Security folk would care about what ports are set up as mirrors and more importantly, what’s at the other end of that port.


The scenario’s above are fairly simple, more advanced switches allow far more sophisticated things to happen. You can rewrite packets, pass off packets to other processors (Routing, Load balancing, Packet inspection, policy enforcement, etc). Again you need to know the path through the switch and what touches a packet to understand the risk and potential impact to that application.


Chris Neal sums up my rant nicely below: