VMware vCenter Inventory Service Filling Up Drive Space

Recently we started experiencing an issue with vCenter filling up the drive space, with continuous 4GB files every few minutes:

C:\Program Files\VMware\Infrastructure\Inventory Service\data\log\xhive_0000000001
...
C:\Program Files\VMware\Infrastructure\Inventory Service\data\log\xhive_0000000012

Each of these files being 4GB, it will fill the drive and take your vCenter down.

After 2 days of wrestling with VMware support, they found this KB.  If you are using vFRC (aka vFlash), you are likely dealing with this issue.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2072392

Basically, there is a bug that causes these log files to fill up with null data.

Symptoms

• Virual Flash (vFlash) is enabled in your vSphere Server environment.
• Most of the CPU and memory is consumed by vpxd.exe or java.exe.
• Disk space is consumed rapidly because the vCenter Inventory Service data folder increase in size (located at C:\Program Files\VMware\Infrastructure\Inventory Service ). 
• You are unable to display the whole inventory in the vSphere Web Client. 
• The ds.log (located at C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs ), contains entries similar to:

Disk space is consumed rapidly is an understatement - it uses it all!

Workaround

According to VMware, there is no patch until 5.5 U2.  However, the fix the KB does work. 

This is a known issue affected VMware vCenter Server 5.5. Currently, there is no resolution. 

Caution: Do not run these steps if the hosts are used for vCloud Director or VMware Horizion View, as removing the hosts will cause virtual machines to become invalid.  It is recommended to contact support for further troubleshooting.

To verify your environment has affected hosts, query the database:

1. Connect the SQL Server where the vCenter Server's database resides using SQL Management Studio.
2. Query the vCenter Server database, with the command:
   
   select distinct fm.ID, h.DNS_NAME
   from vpx_file_system_mount_info fm,
        VPX_HOST h
   where not exists
    (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi
     where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID) and fm.ID = h.ID

To work around this issue, remove the affected hosts modifying the vmConfigInfo.xml file. Then restart vCenter Server.

To remove the hosts and modify vmConfigInfo.xml:

1. Use the same query as above to list the affected ESXi hosts from vCenter Server:
   
   select distinct fm.ID, h.DNS_NAME
   from vpx_file_system_mount_info fm,
        VPX_HOST h
   where not exists
    (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi
     where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID) and fm.ID = h.ID
2. Remove the affected ESXi hosts from vCenter Server inventory.
3. Stop the VirtualCenter Server service. For more information, see Stopping, starting, or restarting vCenter services (1003895).
4. Navigate to:
   
   • In vCenter Server:
     
      C:\Program Files\VMware\Infrastructure\VirtualCenter Server\
   • In the VMware vCenter Server appliance:
     
     /usr/lib/vmware-vpx/
5. Caution: Take a backup of vmConfigInfo.xml. Do not skip this step.
6. Open vmConfigInfo.xml in a text editor.
7. Insert <Table name=...> after the table key for VPX_VMFS_VOLUME and before VPX_SCSI_DISK_PARTITION.
   
   
   <Table name="VPX_VMFS_VOLUME" vmodl_name="vim.host.VffsVolume" parentClass="vim.host.FileSystemVolume" parentAttr="fileSystemVolume.mountInfo.volume" >
           <Column name="VPX_FILE_SYSTEM_MOUNT_INFO_ID" isKey="1" type="int" option="NOT NULL" />
           <Column name="UUID" path="fileSystemVolume.mountInfo.volume.uuid" type="string" />
           <Column name="MAJOR_VERSION" path="fileSystemVolume.mountInfo.volume.majorVersion" type="int" />
           <Column name="VERSION" path="fileSystemVolume.mountInfo.volume.version" type="string" />
           <PrimaryKey name="PK_VPX_VMFS_VOLUME" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
           <ForeignKey name="FK_V_VM_VOL_REF_V_FI_SYS_MO_IN" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" ref="VPX_FILE_SYSTEM_MOUNT_INFO" refVal="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
   </Table>
   
   For example:
   
   
   <Table name="VPX_VMFS_VOLUME" vmodl_name="vim.host.VmfsVolume" parentClass="vim.host.FileSystemVolume" parentAttr="fileSystemVolume.mountInfo.volume" >
   ...
   </Table>
   <Table name="VPX_VMFS_VOLUME" vmodl_name="vim.host.VffsVolume" parentClass="vim.host.FileSystemVolume" parentAttr="fileSystemVolume.mountInfo.volume" >
           <Column name="VPX_FILE_SYSTEM_MOUNT_INFO_ID" isKey="1" type="int" option="NOT NULL" />
           <Column name="UUID" path="fileSystemVolume.mountInfo.volume.uuid" type="string" />
           <Column name="MAJOR_VERSION" path="fileSystemVolume.mountInfo.volume.majorVersion" type="int" />
           <Column name="VERSION" path="fileSystemVolume.mountInfo.volume.version" type="string" />
           <PrimaryKey name="PK_VPX_VMFS_VOLUME" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
           <ForeignKey name="FK_V_VM_VOL_REF_V_FI_SYS_MO_IN" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" ref="VPX_FILE_SYSTEM_MOUNT_INFO" refVal="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
   </Table><Table name="VPX_SCSI_DISK_PARTITION" vmodl_name="vim.host.ScsiDisk.Partition" parentClass="vim.host.VmfsVolume" parentAttr="fileSystemVolume.mountInfo.volume.extent" isArray="1" isUnkeyed="1" >
   ...
   </Table>
8. Save and close vmConfigInfo.xml.
9. Restart vCenter Server, to reflect the updates.
10. Once the vCenter Server system has come back up, re-add the affected hosts. Re-run the SQL Query from the Verify section above. 

-----------------------------------------

WARNING!

If you use dvSwitches, your dvSwitch will become ghosted after you re-add it to vCenter.  You will need to re-add it to the dvSwitch, but this may cause downtime for your VMs as the dvPort ID will likely change as the host is now a "new" host to vCenter (it is issued a new UUID)
-----------------------------------------

Resolution

As of September 9, 2014 VMWare released vCenter 5.5U2.  This issue is supposedly fixed this this release.

• The VMware VirtualCenter Server service shuts down when Virtual Flash is enabled on the ESXi host
  When you enable Virtual Flash on the ESXi host, the VMware VirtualCenter Server service shuts down as the vpxd.exe and java.exe use up all the CPU resources. The Inventory Service data folder increases rapidly in size and reduces the available disk space. Messages similar to the following example are logged in the vpxd.log file:
  
  [06236 error 'commonvpxVmomi' opID=3C35744C-001F520A-a8] [VpxVmomi] Exception while sending result: Not initialized: vim.host.FileSystemVolume volume
  YYYY-02-04T07:54:16.157+01:00 [06236 verbose 'commonvpxVmomi' opID=3C35744C-001F520A-a8] backtrace[00] rip 000000018018cd7a
  
  This issue is resolved in this release.

https://www.vmware.com/support/vsphere5/doc/vsphere-vcenter-server-55u2-release-notes.html

VMware Flash Read Cache (vFRC) Test Drive

VMware vFRC, or Flash Read Cache is a technology that lets you use host-side solid state storage or SSDs as a read cache for your virtual machines.  This allows for extremely low latency and very high performance for cached data, and has the corollary affect of reducing load on the SAN.  While it looks great on paper - how does it really perform?

A few weeks ago, I was able to get my hands on some 300GB eMLC SSD drives - seven of them.  I slapped one into each blade in the test cluster, and got to testing the performance of vFRC.  I am fairly impressed with the results.

To enable vFRC you need local SSD storage on your ESXi host.  This can be PCI-E flash, or a SAS or SATA SSD drive.  If you are going to put this in a production environment, it's highly recommended to use eMLC or SLC drives, as the wear and tear these drives will experience will be significant, especially when used as a cache.

It is also important to note that you will need ESXi 5.5 and vCenter - and you must use the vSphere web client to make all the changes.  An Enterprise Plus license is also required to take advantage of the features.

First, you need to enable the SSD on the host as a vFRC cache in the vSphere web client.  You may need to reboot the host in order to recognize the SSD storage.

You can verify the addition of the flash resource on the summary tab of the host screen.

Second, go onto the VM you wish to accelerate in the vSphere web client.  The VM hardware revision must be vmx-10.

Click OK, and you're done!  You will see the cache on the summary tab of the virtual machine screen.

Now, here's a random access performance test from HDTune.  This is the first run after enabling vFRC.

Here is the second run after enabling vFRC

I also ran the tests in IOMeter - 512B 100% Random Read

Not too shabby.  But, how is the SAN performing?

These are real numbers.  You'll have to trust me, but the second column is NFS ops/s.  And there is a decent load on this SAN (a couple hundred VMs).

But wait, there's more!  vFRC works with VMotion, so you can select to have it keep the migrate the cache contents.

To truly test this, I ran a benchmark test during a host to host vMotion of this VM.  IOMeter doesn't show this very well, so I ran a HDTune graph so you could visibly see the impact.

The red arrows show the start and finish of the vmotion operation.  This was using 8MB block sizes.

As you can see, vFRC is a highly useful feature of VMware.  While it doesn't have the same feature set as other vendors who have been offering SSD host-cache/acceleration, it works very well for a first iteration.

---

Update:

I chose not to set the block size for this post, as it doesn't really matter for this instance to say - "hey, it works!"

However, VMware has a great article if on vFRC performance and tuning.  If you're just getting started with vFRC, you should definitely read this:

http://www.vmware.com/files/pdf/techpaper/vfrc-perf-vsphere55.pdf

NetApp Flash Statistics - CLI Commands - Flash Pool/Hybrid Aggregate, Flash Cache

While the NetApp OnCommand System Manager has some pretty graphs, it doesn't give the best details.  The CLI gives substantially more information.

Flash Pool:

Default (5 second iteration)
stats show -p hybrid_aggr

1 second iteration
stats show -p hybrid_aggr -i 1


Flash Cache:

Default (5 second iteration)
stats show -p flexscale-access

1 second iteration
stats show -p flexscale-access -i 1