VCAC Virtual Machine Deployment "In Progress" forever

If you're deploying a Windows VM using VCAC...and the VM gets stuck at "in progress" and OS customization never completes, this may be the fix for you.  I didn't find the fix by reading the documentation or googling.

Here are the criteria:

• Using the Clone method for the Blueprint
• Using an existing template
• Using an existing customization specification
• The virtual machine clones, and powers up in vsphere
• The virtual machine customization never happens
• The virtual machine status is "In Progress" for a long time
• The workflow finally finishes with "failed"
• You have the following errors in the workflow details:

Request Failed: Machine SERVERNAME: Timeout customizing machine

In the Home > Infrastructure > Monitoring > Log you see

[SERVERNAME] [CustomizeVM] Customization of SERVERNAME failed.: Timeout waiting for CustomizeVM completion event.

Error processing [CustomizeVM], error details: Customization of SERVERNAMEfailed.

Error resuming workflow d27437e3-3c20-41b1-9d47-5e5f803aec44 before delivering workitem response CustomizeVM b8d5063f-c3ca-4d6d-99c2-dc5294316b53, exception: Workflow with id "d27437e3-3c20-41b1-9d47-5e5f803aec44" not found in state persistence store.

Can not upgrade workflow d27437e3-3c20-41b1-9d47-5e5f803aec44

Error delivering workitem response CustomizeVM b8d5063f-c3ca-4d6d-99c2-dc5294316b53 to workflow d27437e3-3c20-41b1-9d47-5e5f803aec44, exception: Can not upgrade workflow d27437e3-3c20-41b1-9d47-5e5f803aec44 Inner Exception: Event "ReceiveWorkItemResponse" on interface type "DynamicOps.VMPS.Service.IWorkItemService" for instance id "d27437e3-3c20-41b1-9d47-5e5f803aec44" cannot be delivered.

If you're not new to VMware (who is these days) and you've been using it for a while, you probably have many templates and customization specs to streamline your day to day deployment of virtual machines.  And if you're like me, you probably have spent a lot of time tweaking those templates, patching those templates, and keeping them up to date.

In order to fix this issue, after re-reading the guides many times and experimenting, I did the following:

1. Removed all spaces in the name of the customization specification
2. Removed all spaces in the template name
3. Recreated the blueprint
4. After none of these fixed the issue, I created a new customization specification.  That did not help either.
5. Lastly, I created a brand new template.  I just installed VMware tools and shut it down.  I left it at vmx-8, and did not upgrade it to vmx-10 (as I am running 5.5)
6. I rescanned the inventory of the cluster compute resource
7. After creating the new template, new blueprint (and publishing it), new catalog service and new entitlement, I was able to successfully deploy a new VM using a customization spec from vCAC.