Storage vMotion Failure: vFRC Bug

Recently I attempted to move a virtual machine from one datastore to another.  This is usually a very routine task, with no errors - ever.

However, I was greeted an unfriendly error.

"Failed to copy one or more of the virtual machine's disks.  See the virtual machine's log for more details. Failed to attach mirror devices for disk copy."

Usually when I get an error, I just google it first in quotes and then poke around the logs if anything turns up to match.

However, this phrase was "un-googleable".  The phrase "failed to attach mirror devices for disk copy" returned no results.  The results without quotes were useless, just the VMware administrators guide, etc.

VMware support can be almost worthless sometimes, as they will take days to get back to you and up to a week to do a log analysis.  I usually end up doing my own log analysis, unless I need third party confirmation.

From the /vmfs/volumes/datastore/vmfolder/vmware-0.log

vmx| I120: Vix: [7695417 mainDispatch.c:4003]: Error VIX_E_FAIL in VMAutomation_ReportPowerOpFinished(): Unknown error
vmx| I120: VmdbPipeStreamsOvlError Couldn't read: OVL_STATUS_EOF, (2) No such file or directory.

Great, unknown error.  Let's dig around a little more.

Let's check /var/log/vmkernel.log

cpu20:41182)VFC: VFCRW_ChangeState:620: VRW Cache state changed to DRAIN

cpu20:41182)VFC: VFCRW_ChangeState:617: VRW Cache state changed to OK

cpu20:41182)WARNING: VFC: VFC_DoIoctlNotification:1641: Cache in Invalid state. Cache detached failed

cpu20:41182)VFC: VFC_DoIoctlNotification:1877: VMK_VFLASH_NOTIFICATION: 6 Failure

cpu20:41182)WARNING: VFLASH: 1475: vflash detach notification to device vfc-1582082800-VMNAME_1 failed. Module vfc detach() function returned Failure.

Bingo.

Normally you can do a storage vMotion with a vFRC (VMware Flash Read Cache) enabled machine, but apparently not this one.

I manually turned off vFRC on just this virtual machine and the storage vMotion ran successfully.