VMware vCenter Inventory Service Filling Up Drive Space

Recently we started experiencing an issue with vCenter filling up the drive space, with continuous 4GB files every few minutes:

C:\Program Files\VMware\Infrastructure\Inventory Service\data\log\xhive_0000000001
...
C:\Program Files\VMware\Infrastructure\Inventory Service\data\log\xhive_0000000012

Each of these files being 4GB, it will fill the drive and take your vCenter down.

After 2 days of wrestling with VMware support, they found this KB.  If you are using vFRC (aka vFlash), you are likely dealing with this issue.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2072392

Basically, there is a bug that causes these log files to fill up with null data.

Symptoms

• Virual Flash (vFlash) is enabled in your vSphere Server environment.
• Most of the CPU and memory is consumed by vpxd.exe or java.exe.
• Disk space is consumed rapidly because the vCenter Inventory Service data folder increase in size (located at C:\Program Files\VMware\Infrastructure\Inventory Service ). 
• You are unable to display the whole inventory in the vSphere Web Client. 
• The ds.log (located at C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs ), contains entries similar to:

Disk space is consumed rapidly is an understatement - it uses it all!

Workaround

According to VMware, there is no patch until 5.5 U2.  However, the fix the KB does work. 

This is a known issue affected VMware vCenter Server 5.5. Currently, there is no resolution. 

Caution: Do not run these steps if the hosts are used for vCloud Director or VMware Horizion View, as removing the hosts will cause virtual machines to become invalid.  It is recommended to contact support for further troubleshooting.

To verify your environment has affected hosts, query the database:

1. Connect the SQL Server where the vCenter Server's database resides using SQL Management Studio.
2. Query the vCenter Server database, with the command:
   
   select distinct fm.ID, h.DNS_NAME
   from vpx_file_system_mount_info fm,
        VPX_HOST h
   where not exists
    (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi
     where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID) and fm.ID = h.ID

To work around this issue, remove the affected hosts modifying the vmConfigInfo.xml file. Then restart vCenter Server.

To remove the hosts and modify vmConfigInfo.xml:

1. Use the same query as above to list the affected ESXi hosts from vCenter Server:
   
   select distinct fm.ID, h.DNS_NAME
   from vpx_file_system_mount_info fm,
        VPX_HOST h
   where not exists
    (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi
     where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID) and fm.ID = h.ID
2. Remove the affected ESXi hosts from vCenter Server inventory.
3. Stop the VirtualCenter Server service. For more information, see Stopping, starting, or restarting vCenter services (1003895).
4. Navigate to:
   
   • In vCenter Server:
     
      C:\Program Files\VMware\Infrastructure\VirtualCenter Server\
   • In the VMware vCenter Server appliance:
     
     /usr/lib/vmware-vpx/
5. Caution: Take a backup of vmConfigInfo.xml. Do not skip this step.
6. Open vmConfigInfo.xml in a text editor.
7. Insert <Table name=...> after the table key for VPX_VMFS_VOLUME and before VPX_SCSI_DISK_PARTITION.
   
   
   <Table name="VPX_VMFS_VOLUME" vmodl_name="vim.host.VffsVolume" parentClass="vim.host.FileSystemVolume" parentAttr="fileSystemVolume.mountInfo.volume" >
           <Column name="VPX_FILE_SYSTEM_MOUNT_INFO_ID" isKey="1" type="int" option="NOT NULL" />
           <Column name="UUID" path="fileSystemVolume.mountInfo.volume.uuid" type="string" />
           <Column name="MAJOR_VERSION" path="fileSystemVolume.mountInfo.volume.majorVersion" type="int" />
           <Column name="VERSION" path="fileSystemVolume.mountInfo.volume.version" type="string" />
           <PrimaryKey name="PK_VPX_VMFS_VOLUME" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
           <ForeignKey name="FK_V_VM_VOL_REF_V_FI_SYS_MO_IN" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" ref="VPX_FILE_SYSTEM_MOUNT_INFO" refVal="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
   </Table>
   
   For example:
   
   
   <Table name="VPX_VMFS_VOLUME" vmodl_name="vim.host.VmfsVolume" parentClass="vim.host.FileSystemVolume" parentAttr="fileSystemVolume.mountInfo.volume" >
   ...
   </Table>
   <Table name="VPX_VMFS_VOLUME" vmodl_name="vim.host.VffsVolume" parentClass="vim.host.FileSystemVolume" parentAttr="fileSystemVolume.mountInfo.volume" >
           <Column name="VPX_FILE_SYSTEM_MOUNT_INFO_ID" isKey="1" type="int" option="NOT NULL" />
           <Column name="UUID" path="fileSystemVolume.mountInfo.volume.uuid" type="string" />
           <Column name="MAJOR_VERSION" path="fileSystemVolume.mountInfo.volume.majorVersion" type="int" />
           <Column name="VERSION" path="fileSystemVolume.mountInfo.volume.version" type="string" />
           <PrimaryKey name="PK_VPX_VMFS_VOLUME" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
           <ForeignKey name="FK_V_VM_VOL_REF_V_FI_SYS_MO_IN" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" ref="VPX_FILE_SYSTEM_MOUNT_INFO" refVal="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
   </Table><Table name="VPX_SCSI_DISK_PARTITION" vmodl_name="vim.host.ScsiDisk.Partition" parentClass="vim.host.VmfsVolume" parentAttr="fileSystemVolume.mountInfo.volume.extent" isArray="1" isUnkeyed="1" >
   ...
   </Table>
8. Save and close vmConfigInfo.xml.
9. Restart vCenter Server, to reflect the updates.
10. Once the vCenter Server system has come back up, re-add the affected hosts. Re-run the SQL Query from the Verify section above. 

-----------------------------------------

WARNING!

If you use dvSwitches, your dvSwitch will become ghosted after you re-add it to vCenter.  You will need to re-add it to the dvSwitch, but this may cause downtime for your VMs as the dvPort ID will likely change as the host is now a "new" host to vCenter (it is issued a new UUID)
-----------------------------------------

Resolution

As of September 9, 2014 VMWare released vCenter 5.5U2.  This issue is supposedly fixed this this release.

• The VMware VirtualCenter Server service shuts down when Virtual Flash is enabled on the ESXi host
  When you enable Virtual Flash on the ESXi host, the VMware VirtualCenter Server service shuts down as the vpxd.exe and java.exe use up all the CPU resources. The Inventory Service data folder increases rapidly in size and reduces the available disk space. Messages similar to the following example are logged in the vpxd.log file:
  
  [06236 error 'commonvpxVmomi' opID=3C35744C-001F520A-a8] [VpxVmomi] Exception while sending result: Not initialized: vim.host.FileSystemVolume volume
  YYYY-02-04T07:54:16.157+01:00 [06236 verbose 'commonvpxVmomi' opID=3C35744C-001F520A-a8] backtrace[00] rip 000000018018cd7a
  
  This issue is resolved in this release.

https://www.vmware.com/support/vsphere5/doc/vsphere-vcenter-server-55u2-release-notes.html