Travelling Techie

Adventures in VMware

User Tools

Site Tools


Network Virtualization
Compute Virtualization
Storage Virtualization
Operating Systems
VMware Classes
Other Classes


Recent Changes

About The Author

Brandon Neill is a VMware Certified Instructor and Consultant. He specializes in NSX and vRealize Automation. In addition to teaching Official VMware Classes, he provides contract training and consulting services.


This is an old revision of the document!


I've never cared much for the blog format, so when I started writing online I decided to create a wiki instead. There will be blog entries as well, but most information will be hierarchically written to (hopefully) make it easier to find. This is very much still a work in progress but I'm adding new information all the time. My primary areas of focus are vSphere Networking and NSX, with vRealize Automation thrown in there too. There will also be a smattering of general vSphere knowledge and maybe even some storage. I've also included information for students taking one of my VMware Classes or watching one of my Lynda Classes.


Josh Townsend’s recent post on vmtoday, PCoIP Packet Loss? Don’t blame the network is a fantastic example of troubleshooting. In addition, it illustrates something I ran in to frequently as an Escalation Engineer. Namely, that just because a problem is exhibiting the signs of a network issue, sometimes you have to expand your search. Once you have eliminated the possible network problems, you have to be willing to look at other areas that might cause similar symptoms.

A very good example of that is packet loss. In addition to network problems (and the very occasional insidious vmkernel problem), anything that causes a vm to pause, even momentarily, can cause it to drop packets.

Storage is one possibility that can cause this, as Josh pointed out. Another possibility is too many vCPUs. While not exactly a pause, if the VM can’t get all it’s processors scheduled, it’s not going to be able to pull all of the packets off of the ring buffer, and they get dropped. Relaxed coscheduling helps with this, but does not eliminate it. I can’t count the number of cases that were resolved by reducing the number of vCPU’s from 4 to 2. (Be aware of HAL/kernel compatibility when changing between 1 and 2 vCPUs). Another issue that can cause packet loss/VM pausing is the CDROM drive. If the iso that is mounted is not available (there can be multiple reasons for this) but the operating system keeps trying to access it, this will lead to small frequent pauses, often resulting in every other ping being dropped. A common cause of this on ESX(i) 5.0 and below is mounting an ISO on a VMFS datastore, and then vMotioning it to the 9th host to access that image. VMFS only supports 8 hosts accessing the same read only file at one time (This has been increased to 32 with VMFS5 on 5.5 and above).

Related information:

2016/12/31 00:03 · brandon

vsish, or the vmkernel system information shell, provides behind the curtain information on the running vmkernel, similar to the way /proc provides information on a running linux kernel. For more information, see What is VMware vsish? from William Lam over at VirtuallyGhetto.

First a word of caution, vsish is not supported unless directed to use it by VMware Support. Do not make uneducated changes to the vmkernel or you can significantly reduce performance, or cause a purple screen.

I am going to focus specifically on network nodes that can be helpful for retrieving information.

  • /vmkModules/cdp CDP information for vmnics
  • /net/pNics/vnmicX/stats Vmnic statistics
  • /net/pNics/vmnicX/properties Driver/firmware information, other properties
  • /net/tcpip/v4/neighbors/ Arp cache information
  • /net/portsets/vSwitchX/ports/#####/X will be the vSwitch number ##### is the port number from esxtop
    • status information on the port, including what device it is connected to
    • stats standard switch counters
    • clientStats counters from the vnic perspective
    • teamuplink what uplink this port is bound to
    • vmxnet3/rxSummary additional counters for the vmxnet3
    • vmxnet3/txSummary additional counters for the vmxnet3
  • /net/portsets/DvsPortset-#/ports/######/ Same as above, for the vDS
  • /system/heaps/NetPktHeap/###### Current NetPktHeap status, below 30% free of max size and problems may occur, check high and low, but low is more important. (this is usually only a problem in 4.0 with more than two 10GB Nics)
2016/12/30 23:42

Not quite a blog, but articles I've written at one point or another.

2016/12/30 23:22 · brandon
start.1487175272.txt.gz · Last modified: 2017/02/15 11:14 by brandon