VMware: Consolidate snapshots not working (file locked error)

Written by Claudio Kuenzler - 0 comments

Published on April 2nd 2015 - Listed in VMware

---

I came across a VMware problem where several VM's were in a warning state with the following alert:

In English: "Virtual machine requires consolidation".

A right-click on the "VM -> Snapshot -> Consolidate" failed with the following error:

In English, the error message is "unable to access file since it is locked".
In the vmware.log file of this VM this could be followed and the virtual disk clearly failed to lock:

2015-04-02T07:01:00.608Z| vcpu-0| I120: DISK: Failed to open disk for consolidate '/vmfs/volumes/mydatastoreuuid/myvm/myvm-000009.vmdk' : Failed to lock the file (16392) 6873
2015-04-02T07:01:00.753Z| vcpu-0| I120: Vix: [173318 vigorCommands.c:548]: VigorSnapshotManagerConsolidateCallback: snapshotErr = Failed to lock the file (5:4008)

This happened on several VM's on several physical hosts. One thing was in common though: The vCenter server was recently updated and therefore there was a communication loss between vCenter server and the physical ESXi hosts.
The VMware KB entry 2003638 contains an interesting note:

    The snapshot commit fails due to locked files. For details on how to release the lock on locked files, see:
        Investigating virtual machine file locks on ESXi/ESX (10051)
        Unable to delete the virtual machine snapshot due to locked files (2017072)
    There is a temporary loss of communication between the vCenter Server and the ESXi/ESX host during snapshot commit. The most common cause of this is an environmental issue which can have a knock on affect on the ESXi/ESX host management agent hostd. For troubleshooting guidance and some possible causes of such an outage, see Troubleshooting vmware-hostd service if it fails or stops responding on an ESX/ESXi host (1002849).

Actually both hints (the lock and loss of communication) combined present the reason and the solution.
There was a loss of communication between the vCenter Server and the ESXi hosts (due to the vCenter update). Yet it is also very likely, that the backup software tried to create a snapshot during this communication loss.
The solution now is to force a re-connection of the ESXi server to the vCenter server. This can be done by restarting the hostd service:

/etc/init.d/hostd restart

After the hostd restart, wait 1 to 2 minutes and then launch "Consolidate" again.
This has worked fine on all VM's and solved the warnings.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.