[an error occurred while processing this directive] Prev Contents Last Next
Clustering Virtual Machines
Clustering Software in Virtual Machines
Network Load Balancing, Microsoft Clustering Service and Veritas Clustering Service run without modification in virtual machines on ESX Server 2.0.
Use of clustering services in virtual machines provides high availability with less hardware (such as machines and network adapters).
Clustering Scenarios
Several scenarios are possible for clustering in virtual machines.
Cluster in a Box —
This provides simple clustering to deal with software crashes or administrative errors. The cluster consists of multiple virtual machines on a single physical machine. It supports shared disks without any shared SCSI hardware. It supports heartbeat network without any extra network adapters.
A four-node cluster on a single physical machine; each node is running clustering software
Cluster across Boxes —
This type of cluster also consists of virtual machines. The virtual disks are stored on real shared disks, so all virtual machines can access them. Using this type of cluster, you can deal with the crash of a physical machine.
A four-node cluster using two physical machines; each node is running clustering software.
Consolidating Clusters —
This type of cluster combines features of the previous two types. For example, you can consolidate four clusters of two machines each to two physical machines with four virtual machines each. This provides protection from both hardware and software failures.
Four two-node clusters moved from eight physical machines to two.
Cost-effective Standby Host —
Provide a standby host for multiple physical machines on one standby box with multiple virtual machines.
A standby host using three virtual machines on a single physical machine; all are running clustering software.
Configuring Virtual Machine Clusters with Shared Disks
To create a set of clustered virtual machines, you need to configure each of them with the following:
•A primary virtual SCSI host adapter with one SCSI virtual disk
•At least two virtual network adapters
◦A public network adapter connected to vmnicx (that is, to vmnic0 or higher). A vmnic is a virtual machine device that uses a network adapter dedicated to the virtual machines.
◦A private network adapter connected to vmnicx (that is, to vmnic0 or higher) or to vmnet_x (that is, to vmnet_0 or higher). This device selection must match in all virtual machines in a cluster set. This is the network adapter that the clustering service will use to monitor the heartbeat between nodes.
•The remaining default virtual machine devices (such as the CD-ROM drive and the floppy disk drive).
In addition to the above devices, the following is required for shared storage:
•A secondary virtual SCSI host adapter
•One or more virtual disks that will be shared attached to the secondary SCSI host adapter
Important Notes
•Each virtual machine by default has five PCI slots available. In this configuration (two network adapters and two SCSI host bus adapters), four of these slots are used. This leaves one more PCI slot for a third network adapter if needed.
•VMware virtual machines currently emulate only the SCSI-2 disk reservation protocol and do not support applications using SCSI-3 disk reservations. However, all popular clustering software (including MSCS and VCS) currently uses SCSI-2 reservations.
Two Node Cluster with Microsoft Cluster Service on a Single ESX Server Machine
This procedure creates a two-node cluster using Microsoft Cluster Service on a single ESX Server machine and uses the following:
•Portsaid = host name of node 1 of the cluster
•Kena = host name of node 2 of the cluster
•Arish = public host name of the cluster
•sharedfs = VMFS volume label of the shared storage
•vms = VMFS volume label of the local storage
Note: Virtual disks stored on vms and sharedfs can also be stored on the same partition. In this case, use the partition label on which these virtual disks reside.
Creating the First Node's Base Virtual Machine
1.Access the VMware Management Interface at
https://<hostname>/ and log on as the user who will own the virtual machine.
2.Click Add Virtual Machine.
3.Keep the default Guest Operating System selection of Microsoft Windows 2000 Server.
Note: This example uses Microsoft Windows 2000 Server as the guest operating system. You may substitute another Windows operating system that supports Microsoft Cluster Service.
4.Change the Display Name field to describe the virtual machine — for example, MSCS Node 1 (Portsaid).
5.Change the Location of the virtual machine configuration file to
/home/<user>/vmware/cluster1/cluster1.vmx.
6.Click Next.
7.Select the number of processors you want the guest operating system to use, up to 2.
8.Change Memory to show the amount of RAM you want to allocate to this virtual machine.
9.Click Next.
10.Click Blank to create a new virtual disk.
11.Choose the VMFS volume on which you want to store the virtual disk.
12.Give the virtual disk image a unique name — for example, cluster1.dsk.
13.If you need a primary SCSI disk larger than 4GB, enter the appropriate value in the Capacity field.
14.Choose the virtual SCSI node to which you want to attach the virtual disk.
15.By default, the disk mode is set to persistent. Click Persistent to verify the disk mode.
16.Click Next.
You have successfully created the virtual machine.
The hardware tab for this virtual machine appears. From that tab, you now need to add additional hardware devices.
Virtual Disk Configuration —
You need a shared SCSI controller and shared SCSI disks for shared access to clustered services and data.
To add a shared SCSI controller and shared SCSI disks, click the Hardware tab, then take the following steps:
1.Click Add Device.
2.Click Hard disk.
3.Click Blank to create a new virtual disk.
4.Choose the VMFS volume on which you want to store the virtual disk.
5.Give the virtual disk image a unique name — for example, quorum.dsk.
6.Enter the appropriate value in the Capacity field.
7.Choose the virtual SCSI node to which you want to attach the virtual disk.
Note: Shared disks must be attached to a separate SCSI controller. Select SCSI 1:1
8.By default, the disk mode is set to persistent. Click Persistent to verify the disk mode.
9.Click OK.
Note: A new virtual disk and SCSI Controller 1 are now visible on the hardware tab.
10.Click Edit next to SCSI Controller 1 and change the bus sharing from none to virtual.
From the Bus Sharing drop-down list, select virtual, then Click OK.
Repeat step 1-step 9 to create an additional shared virtual disk using SCSI 1:2 with the filename shared2.dsk.
Network Device Configuration —
You need an additional virtual network adapter to be used by Microsoft Cluster Service to maintain the cluster heartbeat. To add this adapter, click the Hardware tab for this virtual machine, then take the following steps:
1.Click Add Device.
2.Click Network Adapter.
3.From the Device Binding drop-down list choose vmnet_0. This attaches the second Ethernet adapter to a private network between the cluster nodes.
4.Click OK.
You have created the first cluster node virtual machine.
Installing the Guest Operating System
Now you need to install Windows 2000 Advanced Server in the virtual machine you just created
1.Insert the Windows 2000 Advanced Server CD in the ESX Server machine's CD-ROM drive.
2.In the management interface, click the blue terminal icon next to the virtual machine's name to launch the remote console.
3.Log on as the user who created the virtual machine or as root.
4.Click Power On.
5.Install Windows 2000 Advanced Server on the disk connected to scsi0.
6.Accept all the default options during the installation. Do not install the clustering service at this time.
7.When the installation is completed, install VMware Tools in the guest operating system.
Cloning the Virtual Machine
Now that you have a virtual machine with Windows 2000 Advanced Server installed, you can save time by cloning this virtual machine as follows:
1.Run sysprep.exe, which is available on the Windows 2000 CD in the \support\tools\deploy.cab file. This strips the security ID assigned to the guest operating system and resets the machine information as well as the TCP/IP network configuration.
2.Shut down the guest operating system and power off the virtual machine.
3.Remove the Windows 2000 Advanced Server CD from the server's CD-ROM drive.
4.On the management interface's Overview page, click Manage Files.
5.Drill down to the vmfs folder, then the vms folder. This may take some time to refresh.
6.Select the check box next to the cluster1.dsk file.
7.Click Copy.
8.Click Paste.
9.When the copy process is complete, select the check box next to the file copy of cluster1.dsk.
10.Click the Edit Properties button.
11.Change the filename to cluster2.dsk.
12.Click OK.
13.Close the Manage Files window.
This concludes the cloning process. Now continue with creating the second node virtual machine
Creating the Second Node Virtual Machine
Create a new virtual machine as follows:
1.On the management interface's Overview page, click Add Virtual Machine.
2.Keep the default Guest Operating System selection of Microsoft Windows 2000 Server.
3.Change the Display Name field to describe the virtual machine — for example, SCS Node 2 (Kena).
4.Change the Location to
home/<user>/vmware/cluster2/cluster2.vmx
5.Click Next.
6.Select the number of processors you want the guest operating system to use, up to 2.
7.Change Memory to show the amount of RAM you want to allocate to this virtual machine.
8.Click Next.
9.Click Existing to attach an existing virtual disk to this virtual machine.
10.From the Virtual Disk Image drop-down list, choose cluster2.dsk.
11.Choose the virtual SCSI node to which you want to attach the virtual disk.
12.Click Next.
Virtual Disk Configuration —
You need a shared SCSI controller and shared SCSI disks for shared access to clustered services and data.
To add a shared SCSI controller and shared SCSI disks, click the Hardware tab for this virtual machine, then take the following steps:
1.Click Add Device.
2.2. Click Hard Disk.
3.Click Blank to create a new virtual disk.
4.4. Choose the VMFS Volume on which you want to store the virtual disk.
5.Give the virtual disk image a unique name — for example, quorum.dsk.
6.Enter the appropriate value in the Capacity field.
7.Choose the virtual SCSI node to which you want to attach the virtual disk.
Note: Shared disks must be attached to a separate SCSI controller. Select SCSI 1:1.
8.By default the disk mode is set to persistent. Click Persistent to verify the disk mode.
9.Click OK.
Note: A new virtual disk and SCSI Controller 1 are now visible on the hardware tab.
10.Click Edit next to SCSI Controller 1 to change the bus sharing from none to virtual.
11.From the Bus Sharing drop-down list select virtual, then click OK.
Repeat step 1-step 9 create an additional shared virtual disk using SCSI 1:2 with the filename shared2.dsk.
Network Device Configuration —
You need an additional virtual network adapter to be used by Microsoft Cluster Service to maintain the cluster heartbeat. To add this adapter, click the Hardware tab for this virtual machine, then take the following steps:
1.Click Add Device.
2.Click Network Adapter.
3.From the Device Binding drop-down list choose vmnet_0. This attaches the second Ethernet adapter to a private network between the cluster nodes.
4.Click OK.
You have created the second cluster node virtual machine.
Go to the management interface's Overview page. The management interface should list both virtual machines and show them powered off.
Installing Microsoft Cluster Service
1.Start the node 1 virtual machine.
2.Follow the Windows 2000 Advanced Server mini-setup prompts to enter Advanced Server's serial number, the host name (Portsaid) and the IP addresses. Note that you need to enter the addresses for both public and private network adapters.
For the public network adapter, enter an IP address that belongs to the physical network.
For the private IP address, you may use an address like 192.168.x.x with a class C subnet mask (255.255.255.0).
3.At the end of the process, Windows automatically reboots.
4.Start the Disk Administrator and change both shared disks to basic disks.
5.Format both shared virtual disks with NTFS if they are not already formatted.
6.Assign the first shared disk to Q: (quorum) and the second disk to R:.
If you have joined this virtual machine to an existing Active Directory domain, skip to step 11.
7.Run dcpromo.exe from the command prompt. This starts the Active Directory Wizard.
8.Set up the current machine as a domain controller. For the domain name, use something like vmcluster.domain.com where domain.com is your DNS domain and vmcluster is your Active Directory domain. This node may be setup as a new domain tree and also a new domain forest, or it may join existing ones.
9.Make sure the DNS server is installed.
10.Set the domain permissions as mixed mode unless you plan otherwise.
11.To add a cluster services account in the domain, go to Programs > Administrative Tools > Active Directory Users and Computers.
12.Add an account named cluster, check User cannot change password and Password never expires.
13.Insert the Windows 2000 Advanced Server CD in the server's CD-ROM drive.
14.Go to Control Panel > Add/Remove Programs.
15.Select Add/Remove Windows Components.
16.Check the Cluster Service component.
17.Click Next. Follow the prompts to install the service.
18.As you configure Cluster Service, choose Form a New Cluster.
19.Specify the cluster name (Arish)
20.Specify the cluster IP address. This is the address that will represent the cluster. It must be on the same network as that of the vmnic0.
21.Specify the cluster service account created above.
22.Specify that both shared disks should be managed by the cluster service.
23.Indicate the shared disk (Q:) to be the quorum disk.
24.Specify which network adapter is public and which is private.
25.Stop the cluster service on the local node (from Cluster Manager, right-click the node name), so the second virtual machine can access the shared disks.
26.Start the node 2 virtual machine.
27.Repeat step 2 and step 3 above.
28.Start the Disk Administrator and assign the first shared disk to Q: (quorum) and the second disk to R:.
29.Start dcpromo.exe and add this virtual machine as a domain controller in the same domain created in step 8 above or add it to an existing domain. You must match the setup done in step 8.
30.In the node 1 virtual machine, start the cluster service by reversing step 25 above.
31.In the node 2 virtual machine, repeat step 14-step 24 above with one exception: In step 18, select Join a Cluster.
This concludes the Microsoft Cluster Service installation and configuration.
Running Microsoft Cluster Service
Microsoft Cluster Service should operate normally in the virtual machine once it is installed.
Note: Some disk errors are recorded in the Windows event log in normal operation. These error messages have a format similar to
The driver detected a controller error on
\Device\Scsi\BusLogic3
They should be reported periodically only on the passive node of the cluster and should also be reported when the passive node is taking over during a failover. The errors are reported because the active node of the cluster has reserved the shared virtual disk(s). The passive node periodically probes the shared disk and receives a SCSI reservation conflict error. This is normal operation.
Two Nodes with Microsoft Cluster Service on Separate ESX Server Machines
This procedure creates a two node cluster in virtual machines that will run on two separate ESX Server machines. It uses the same naming conventions as in the previous procedure.
In addition, the physical shared storage is either:
•Shared SCSI
•A storage area network (SAN)
For this exercise the VMFS partition for the internal storage on each ESX Server computer is labeled vms. The VMFS partition for the shared storage is labeled sharedfs.
•The VMFS partition for the internal storage on each ESX Server machine is labeled vms.
•The VMFS partition for the shared storage is labeled sharedfs.
Each ESX Server machine must have an additional physical network adapter assigned to the virtual machines to use for the private network that monitors the heartbeat. The procedure assumes this network adapter uses the device named vmnic1. You should connect the private network adapter to a separate network from that used by the public network adapter.
Creating the First Node's Base Virtual Machine
Follow the procedure in Creating the First Node's Base Virtual Machine, with the following changes:
•In the Virtual Disk Configuration section, in step 10 click Edit next to SCSI Controller 1 to change the bus sharing from none to physical instead of virtual. From the Bus Sharing drop-down list select physical, then click OK.
•In the Network Device Configuration section, in step 3 use vmnic1 instead of vmnet_0 as the device used by Ethernet Adapter 1.
•Access the virtual machine menu by clicking the arrow to the right of the virtual machine icon. Choose Configure Options. Under Verbose Options, click the click here link.
Change the specifications of scsi1:1.name and scsi1:2.name to use the strict vmhba name (for example, vmhba0:1:0:1:shared1.dsk) for the VMFS partition, rather than the VMFS name (for example, sharedfs:shared1.dsk). The reason for this change is that if one ESX Server machine reboots while a virtual machine on the other physical machine is reserving the shared SCSI disk, ESX Server cannot read the VMFS name on the shared disk when it is loaded and initialized. If the shared virtual disk is not specified using the full vmhba name, ESX Server cannot determine the disk specified by the VMFS name and gives an error when restarting the virtual machine.
When you have made these changes, click OK.
In addition to these minor changes, you need to change the access rights of the VMFS partition where you store the shared virtual disks. By default VMFS partitions are configured for public access. In order to support clustering, the VMFS partition needs to be configured for shared access.
Take the following steps to change the access settings for the VMFS partition:
1.From the management interface click the Options tab
2.Click Storage Configuration.
3.Identify the disk volume that contains the VMFS partition where the shared virtual disks are stored. Click Edit for the disk volume.
4.From the VMFS Access drop-down list, choose Shared.
5.Click OK.
You have created the first cluster node virtual machine.
Installing the Guest Operating System
Follow the procedure in Installing the Guest Operating System.
Cloning the Virtual Machine
Now that you have a virtual machine with Windows 2000 Advanced Server installed, you can save time by cloning this virtual machine as follows:
1.Run sysprep.exe, which is available on the Windows 2000 CD in the \support\tools\deploy.cab file. This strips the Security ID assigned to the guest operating system and resets the machine information as well as the TCP/IP network configuration.
2.Shut down the guest operating system and power off the virtual machine.
3.Go to the console of the second ESX Server machine. This is where you will copy the virtual disk that resulted from creating the first node.
4.Log on as root.
5.Change directories: cd /vmfs/vms
This assumes that the internal storage for the second server is in a VMFS partition labeled vms.
6.Use the ftp command: ftp <server1-hostname>
7.Change directories: cd /vmfs/vms
This changes the current directory to the VMFS partition on the first server where you created the first node's virtual disk.
8.Set the type to binary: bin
This sets the transfer mode to binary. If you use text transfer mode, the virtual disk may not be usable on the target server.
9.Type: hash on
This turns on the display of a series of hash signs as a transfer progress indicator.
10.Retrieve the virtual disk file: get cluster1.dsk
This initiates the transfer of the virtual disk file to the current directory on the second ESX Server machine.
11.Quit the ftp session: bye
After the file transfer is completed, type the bye command to end the FTP session.
12.Rename the file: mv cluster1.dsk cluster2.dsk
This renames the virtual disk to cluster2.dsk.
This concludes the cloning process. Continue with creating the second node virtual machine.
Creating the Second Node Virtual Machine
Follow the procedure in Creating the First Node's Base Virtual Machine, noting the following differences:
•In the Virtual Disk Configuration section, step 10, click Edit next to SCSI Controller 1 to change the bus sharing from none to physical instead of virtual. From the Bus Sharing drop-down list, choose physical, then click OK.
•In the Network Device Configuration section, step 3, from the Device Binding drop-down list, choose vmnic1 instead of vmnet_0. This attaches the second Ethernet adapter to the second physical adapter designated for virtual machine use. This is used to create a private network between the cluster nodes.
•Change the specifications of scsi1:1.name and scsi1:2.name as you did when creating the first node's base virtual machine.
Clustering Using a Raw SCSI Disk
The shared disk used for clustering can also be a complete shared SCSI disk, rather than a VMFS file on a shared disk. Using a raw SCSI disk as a shared disk may simplify initial setup. It may be especially useful for importing an existing physical cluster that already has cluster data on a SCSI disk. In addition, using a raw SCSI disk as a shared disk allows a virtual machine to participate in a cluster with a physical machine. For example, the virtual machine can be used as the passive node for a physical machine that is the active node.
In order for the virtual machine to access a physical disk, the instructions in the Virtual Disk Configuration section on Virtual Disk Configuration — should be replaced with the following steps:
To add a physical SCSI controller and shared raw SCSI disks, go to the Hardware tab and take the following steps:
1.Click Add Device.
2.Click Hard disk.
3.Click System LUN/Disk to give your virtual machine direct access to a SAN or shared storage volume.
4.Choose the LUN/Partition you want to attach to this VM as a raw disk.
Note: In ESX Server, physical disks are identified by a vmhba number. For example, vmhba0:1:2:1 means physical adapter vmhba0, target 1, LUN 2, partition 1. When the final number is :0, that indicates you are specifying the entire disk, rather than a particular partition.
5.Choose the virtual SCSI node to which you want to attach the raw disk.
Note: Shared disks must be attached to a separate SCSI controller from the system disk. Select, SCSI 1:1
6.Click OK.
A new virtual disk and SCSI Controller 1 appear on the Hardware tab.
7.Click Edit next to SCSI Controller 1 to change the bus sharing from none to physical.
8.From the Bus Sharing drop-down list choose physical, then click OK.
Setting the bus sharing to physical makes sure that all the SCSI reserve and reset commands go through to the physical disk.
Repeat step 1-step 8 to create an additional shared raw disk using SCSI 1:2.
You have completed the virtual machine configuration.
Installing Microsoft Cluster Service
Follow the procedure in Installing Microsoft Cluster Service.
Additional Notes for Clustering Across Physical Machines
•For maximum flexibility, put each shared virtual disk on a separate VMFS on a separate SCSI disk (or LUN). The reason for this is that SCSI reservation can be done only at the granularity of a complete SCSI disk or LUN. If two shared virtual disks are on the same VMFS, reserving one disk will also reserve the other disk. If you choose to put more than one shared disk on the same VMFS (as in the example installation above), always put these virtual disks in the same MSCS resource group. That ensures the disks are always reserved by the same computer and are always failed over at the same time.
•Ensure that the swap file for each ESX Server machine is not on any of the physical shared disks. Since the shared disks are reserved by one host or another, the swap file may become inaccessible to one host or the other during the operation of the cluster.
•If you are accessing the shared disks on a storage network (SAN) using a QLogic host bus adapter, you must use particular values for some QLogic configuration settings. Reboot your physical machine and enter the QLogic configuration utility during bootup. Under Advanced Configuration Settings, ensure that
◦Enable Target Reset is set to Yes.
◦Full LIP Login is set to Yes.
◦Full LIP Reset is set to No.
•If you are accessing the shared disks on a SAN using an Emulex host bus adapter, you need to take the following extra steps:
a.On the Options > Advanced Settings page, Set DiskUseDeviceReset to 1.
b.Supply an extra parameter to the Emulex driver when it is loaded. You do this by editing the file /etc/vmware/hwconfig. First, identify the bus, slot and function holding the first (or only) Emulex card. You can find this information by looking at the Startup Profile page. Then add a line with the format
device.vmnix.6.14.0.options = "lpfc_delay_rsp_err=0"
to the end of /etc/vmware/hwconfig. Here, the numbers 6.14.0 specify the bus, slot and function where the Emulex card is located. If you have more than one Emulex card, you should have only a line referencing the first card.
•If you are doing multipathing and can access your shared cluster disk via multiple HBAs from a host, then you need to modify another configuration file. Go to the Options > Advanced Settings page and set DiskResetOnFailover to 1. If an HBA failover occurs on the host, this option causes a SCSI bus reset to be issued after the failover happens. This bus reset allows the second HBA to access a shared disk, even if the disk has been reserved by the first HBA. The clustering software then automatically reserves the disk again via the second HBA.
Running Microsoft Cluster Service
Microsoft Cluster Service should operate normally in the virtual machines once it is installed.
Note: Some disk errors are recorded in the Windows event log in normal operation. These error messages have a format similar to
The driver detected a controller error on
\Device\Scsi\BusLogic3
They should be reported periodically only on the passive node of the cluster and should also be reported when the passive node is taking over during a failover. The errors are reported because the active node of the cluster has reserved the shared virtual disk. The passive node periodically probes the shared disk and receives a SCSI reservation conflict error.
VMFS Locking and SCSI Reservation
For a shared SCSI disk that can be accessed by multiple ESX Server machines, two kinds of locking may be in use. These two kinds of locking are somewhat independent and can cause confusion. The shared SCSI disk may be on shared SCSI bus or, more likely, on a storage area network (SAN).
VMFS File System Locking
The first kind of locking is VMFS file system locking. ESX Server locks VMFS file systems on a server level when a VMFS file system is configured as a public or shared file system. This locking is done in order to ensure that there is no corruption caused by multiple accesses to the file system by different hosts.
If a VMFS-1 volume is configured in public mode, only one server can ever access that VMFS at a time. If one server is accessing the VMFS-1 volume, through a virtual machine or a file system command, then a file system operation by another host fails. For example, a vmkfstools command fails with a message that says:
vmkfstools: file system is locked by another server. Use 'vmkfstools --recover' to unlock file system if no other server is accessing
Typically, you should not run vmkfstools --recover at this point, since another host is actually using the file system. The error message simply indicates that this server cannot access the VMFS until the other server has finished accessing it. However, if a server fails while accessing the file system, the file system may stay in the locked state and you may need to run vmkfstools --recover.
In a public VMFS-2 volume, locking is at a per-file level, resulting in fewer locking issues. However, you may still encounter the preceding message and may need to use vmkfstools --recover, if a server fails.
If a VMFS is used to store a virtual disk that is accessed by multiple virtual machines on multiple physical servers for the purposes of failover clustering, the VMFS should be configured as a shared file system. Then, the locking protocol is slightly relaxed to allow multiple virtual machines on different servers to access the same VMFS file at the same time. However, file system commands do the same locking as with public file systems (that is, per-VMFS in VMFS-1 volumes and per-file in VMFS-2 volumes).
Additionally, when multiple virtual machines access the VMFS, the VMFS file system enters a read-only mode in which it is impossible to create, delete or change the size of files. However, the contents of the individual files can still be modified. If you later want to create or remove VMFS files, you must stop all virtual machines using the VMFS and re-enter writable mode by using this command:
vmkfstools --config writable vmhba0:1:0:0
Substitute the name of the appropriate disk or VMFS in place of vmhba0:1:0:0.
Locking at SCSI Disk Level
The second kind of locking is locking at the SCSI disk level, which is called SCSI disk reservation.
Any server connected to a SCSI disk can issue a SCSI command to reserve the disk. If no other server is already reserving the disk, the current server obtains a reservation on the disk. As long as that reservation exists, no other server can access the disk. All SCSI commands to that disk by other servers fail with an appropriate error code.
If a vmkfstools command is attempted on a VMFS on a disk that is reserved by another server, the vmkfstools command fails with a message that says:
vmkfstools: shared SCSI disk is reserved by another server. Use 'vmkfstools -L release/reset' to end reservation if no other server is using the SCSI reservation
Similarly, a virtual machine fails to start if its virtual boot disk is stored on a physical disk that is reserved by another host.
Most applications do not ever reserve a SCSI disk. However, failover clustering software reserves SCSI disks in order to ensure that only the active node is able to access the shared SCSI disk. Therefore, you should expect that the shared disk in a physical clustering setup is reserved when the cluster is active. Similarly, for a virtual machine cluster that is running across physical machines, reservations by the clustering software are transmitted through to the physical shared disk.
If you encounter a disk that is reserved unexpectedly, you should try to determine if some clustering software has explicitly reserved the disk. If not, you can release the reservation on the server that has the reservation by running a command in this format:
vmkfstools -L release vmhba0:1:0:0
Substitute the name of the appropriate disk or VMFS in place of vmhba0:1:0:0. If you cannot determine which server holds the reservation, you may be able to eliminate the reservation by issuing a SCSI bus reset on any server machine using a command in this format:
vmkfstools -L reset vmhba0:1:0:0
Using LUN Masking to Avoid Locking Issues
Locking issues are especially likely to happen on a SAN, where multiple users may be accessing some of the same disks or may mistakenly access a disk assigned to another user.
It is often helpful to use LUN masking or zoning to limit what disks are visible to each server in the system, and therefore reduce the ways in which one user can affect another user. In particular, the use of LUN masking or zoning can help prevent problems such as those described above in which one server unexpectedly locks or reserves the wrong SCSI disk.