SCSI: vm 1043: 5522: Sync CR at 64
SCSI: vm 1043: 5522: Sync CR at 48
SCSI: vm 1043: 5522: Sync CR at 32
SCSI: vm 1043: 5522: Sync CR at 16
SCSI: vm 1043: 5522: Sync CR at 0
WARNING: SCSI: 5532: Failing I/O due to too many reservation conflicts
WARNING: SCSI: 5628: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:7. residual R 919, CR 0, ER 3
WARNING: J3: 1970: Error committing txn to slot 0: SCSI reservation conflict
Solution
There are two main categories of operation under which VMFS makes use of SCSI reservations.
 
The first category is for VMFS data-store level operations. These include opening, creating, resignaturing, and expanding/extending of VMFS data-store.
 
The second category involves acquisition of locks. These are locks related to VMFS specific meta-data (called cluster locks) and locks related to files (including directories). Operations in the second category occur much more frequently than operations in the first category. The following are examples of VMFS operations that require locking metadata:
•Creating a VMFS datastore
•Expanding a VMFS datastore onto additional extents
•Powering on a virtual machine
•Acquiring a lock on a file
•Creating or deleting a file
•Creating a template
•Deploying a virtual machine from a template
•Creating a new virtual machine
•Migrating a virtual machine with VMotion
•Growing a file, for example, a Snapshot file or a thin provisioned Virtual Disk 
If the VMkernel log contains the messages described in the Details section, follow this procedure:
1.If the VMware ESX version is 3.0.1, install Patch ESX-1002960: Fix for SCSI Reservation Conflict Issue. For more information, see ESX Server 3.0.1, Patch ESX-1002960: Fix for SCSI Reservation Conflict Issue (1002960).

If the VMware ESX version is 3.0.2, install Patch ESX-1002974: Fix for SCSI Reservation Conflicts. For more information, see ESX Server 3.0.2, Patch ESX-1002974: Fixes for SCSI Reservation Conflicts; Support for EMC Invista (1002974).

2.If the log messages persist and the ESX host is running on an HP Server using Insight Manager Agents, see Insight Manager may cause SCSI reservation conflicts (1004771).
3.If the log messages persist, see the article that applies to your environment:

SCSI Reservation Failures on HDS USP and NSC Arrays (1005010)
SCSI Reservation Failures on HP XP Storage Arrays (1005011)
SCSI Reservation Failures on Hitachi USP and NSC Arrays (1006001)
SCSI Reservation Failures on SUN StorageTek 9985 and 9990 Arrays (1006002)
SCSI Reservation Failures on Nihon Unisys SANARENA 5200 and 5800 Arrays (1006003)
Virtual machines may experience I/O failures due to too many SCSI reservation conflicts on some 3PAR arrays (1020366
 

Note: The list of arrays is not exhaustive and will be revised when other arrays are identified reporting these errors.

4.Follow these steps to resolve potential sources of the reservation: 
a.Try to serialize the operations of the shared LUNs, if possible, limit the number of operations on different hosts that require SCSI reservation at the same time.
b.Increase the number of LUNs and try to limit the number of ESX hosts accessing the same LUN.
c.Reduce the number snapshots as they cause a lot of SCSI reservations.
d.Do not schedule backups (VCB or console based) in parallel from the same LUN.
e.Try to reduce the number of virtual machines per LUN. See vSphere 4.0 Configuration Maximums and ESX 3.5 Configuration Maximums.
f.What targets are being used to access LUNs?
g.Check if you have the latest HBA firmware across all ESX hosts.
h.Is the ESX running the latest BIOS (avoid conflict with HBA drivers)?
i.Contact your SAN vendor for information on SP timeout values and performance settings and storage array firmware.
j.Turn off 3rd party agents (storage agents) and rpms not certified for ESX.
k.MSCS rdms (active node holds permanent reservation). For more information, see ESX servers hosting passive MSCS nodes report reservation conflicts during storage operations (1009287).
l.Ensure correct Host Mode setting on the SAN array.
m.LUNs removed from the system without rescanning can appear as locked.
n.When SPs fail to release the reservation, either the request did not come through (hardware, firmware, pathing problems) or 3rd party apps running on the service console did not send the release. Busy virtual machine operations are still holding the lock.
Note: Use of SATA disks is not recommended in high I/O configuration or when the above changes do not resolve the problem while SATA disks are used.
 
If your array is not listed above and none of the above points eliminate the log messages, file a support request with VMware Support and note this KB Article ID in the problem description.  For more information, see How to Submit a Support Request.
Keywords

Clustering Virtual Machines

Clustering Software in Virtual Machines
Network Load Balancing, Microsoft Clustering Service and Veritas Clustering Service run without modification in virtual machines on ESX Server 2.0.

Use of clustering services in virtual machines provides high availability with less hardware (such as machines and network adapters).

Clustering Scenarios
Several scenarios are possible for clustering in virtual machines.

Cluster in a Box —
This provides simple clustering to deal with software crashes or administrative errors. The cluster consists of multiple virtual machines on a single physical machine. It supports shared disks without any shared SCSI hardware. It supports heartbeat network without any extra network adapters.

A four-node cluster on a single physical machine; each node is running clustering software

Cluster across Boxes —
This type of cluster also consists of virtual machines. The virtual disks are stored on real shared disks, so all virtual machines can access them. Using this type of cluster, you can deal with the crash of a physical machine.

A four-node cluster using two physical machines; each node is running clustering software.

Consolidating Clusters —
This type of cluster combines features of the previous two types. For example, you can consolidate four clusters of two machines each to two physical machines with four virtual machines each. This provides protection from both hardware and software failures.

Four two-node clusters moved from eight physical machines to two.

Cost-effective Standby Host —
Provide a standby host for multiple physical machines on one standby box with multiple virtual machines.

A standby host using three virtual machines on a single physical machine; all are running clustering software.

Configuring Virtual Machine Clusters with Shared Disks
To create a set of clustered virtual machines, you need to configure each of them with the following:

•A primary virtual SCSI host adapter with one SCSI virtual disk
•At least two virtual network adapters
◦A public network adapter connected to vmnicx (that is, to vmnic0 or higher). A vmnic is a virtual machine device that uses a network adapter dedicated to the virtual machines.
◦A private network adapter connected to vmnicx (that is, to vmnic0 or higher) or to vmnet_x (that is, to vmnet_0 or higher). This device selection must match in all virtual machines in a cluster set. This is the network adapter that the clustering service will use to monitor the heartbeat between nodes.
•The remaining default virtual machine devices (such as the CD-ROM drive and the floppy disk drive).
In addition to the above devices, the following is required for shared storage:

•A secondary virtual SCSI host adapter
•One or more virtual disks that will be shared attached to the secondary SCSI host adapter

Important Notes
•Each virtual machine by default has five PCI slots available. In this configuration (two network adapters and two SCSI host bus adapters), four of these slots are used. This leaves one more PCI slot for a third network adapter if needed.
•VMware virtual machines currently emulate only the SCSI-2 disk reservation protocol and do not support applications using SCSI-3 disk reservations. However, all popular clustering software (including MSCS and VCS) currently uses SCSI-2 reservations.

Two Node Cluster with Microsoft Cluster Service on a Single ESX Server Machine
This procedure creates a two-node cluster using Microsoft Cluster Service on a single ESX Server machine and uses the following:

•Portsaid = host name of node 1 of the cluster
•Kena = host name of node 2 of the cluster
•Arish = public host name of the cluster
•sharedfs = VMFS volume label of the shared storage
•vms = VMFS volume label of the local storage
Note: Virtual disks stored on vms and sharedfs can also be stored on the same partition. In this case, use the partition label on which these virtual disks reside.

Creating the First Node's Base Virtual Machine
1.Access the VMware Management Interface at
https://<hostname>/ and log on as the user who will own the virtual machine.
2.Click Add Virtual Machine.
3.Keep the default Guest Operating System selection of Microsoft Windows 2000 Server.
Note: This example uses Microsoft Windows 2000 Server as the guest operating system. You may substitute another Windows operating system that supports Microsoft Cluster Service.

4.Change the Display Name field to describe the virtual machine — for example, MSCS Node 1 (Portsaid).
5.Change the Location of the virtual machine configuration file to
/home/<user>/vmware/cluster1/cluster1.vmx.
6.Click Next.
7.Select the number of processors you want the guest operating system to use, up to 2.
8.Change Memory to show the amount of RAM you want to allocate to this virtual machine.
9.Click Next.
10.Click Blank to create a new virtual disk.
11.Choose the VMFS volume on which you want to store the virtual disk.
12.Give the virtual disk image a unique name — for example, cluster1.dsk.
13.If you need a primary SCSI disk larger than 4GB, enter the appropriate value in the Capacity field.
14.Choose the virtual SCSI node to which you want to attach the virtual disk.
15.By default, the disk mode is set to persistent. Click Persistent to verify the disk mode.
16.Click Next.
You have successfully created the virtual machine.

The hardware tab for this virtual machine appears. From that tab, you now need to add additional hardware devices.

Virtual Disk Configuration —
You need a shared SCSI controller and shared SCSI disks for shared access to clustered services and data.

To add a shared SCSI controller and shared SCSI disks, click the Hardware tab, then take the following steps:

1.Click Add Device.
2.Click Hard disk.
3.Click Blank to create a new virtual disk.
4.Choose the VMFS volume on which you want to store the virtual disk.
5.Give the virtual disk image a unique name — for example, quorum.dsk.
6.Enter the appropriate value in the Capacity field.
7.Choose the virtual SCSI node to which you want to attach the virtual disk.
Note: Shared disks must be attached to a separate SCSI controller. Select SCSI 1:1

8.By default, the disk mode is set to persistent. Click Persistent to verify the disk mode.
9.Click OK.
Note: A new virtual disk and SCSI Controller 1 are now visible on the hardware tab.

10.Click Edit next to SCSI Controller 1 and change the bus sharing from none to virtual.
From the Bus Sharing drop-down list, select virtual, then Click OK.

Repeat step 1-step 9 to create an additional shared virtual disk using SCSI 1:2 with the filename shared2.dsk.

Network Device Configuration —
You need an additional virtual network adapter to be used by Microsoft Cluster Service to maintain the cluster heartbeat. To add this adapter, click the Hardware tab for this virtual machine, then take the following steps:

1.Click Add Device.
2.Click Network Adapter.
3.From the Device Binding drop-down list choose vmnet_0. This attaches the second Ethernet adapter to a private network between the cluster nodes.
4.Click OK.
You have created the first cluster node virtual machine.

Installing the Guest Operating System
Now you need to install Windows 2000 Advanced Server in the virtual machine you just created

1.Insert the Windows 2000 Advanced Server CD in the ESX Server machine's CD-ROM drive.
2.In the management interface, click the blue terminal icon next to the virtual machine's name to launch the remote console.
3.Log on as the user who created the virtual machine or as root.
4.Click Power On.
5.Install Windows 2000 Advanced Server on the disk connected to scsi0.
6.Accept all the default options during the installation. Do not install the clustering service at this time.
7.When the installation is completed, install VMware Tools in the guest operating system.

Cloning the Virtual Machine
Now that you have a virtual machine with Windows 2000 Advanced Server installed, you can save time by cloning this virtual machine as follows:

1.Run sysprep.exe, which is available on the Windows 2000 CD in the \support\tools\deploy.cab file. This strips the security ID assigned to the guest operating system and resets the machine information as well as the TCP/IP network configuration.
2.Shut down the guest operating system and power off the virtual machine.
3.Remove the Windows 2000 Advanced Server CD from the server's CD-ROM drive.
4.On the management interface's Overview page, click Manage Files.
5.Drill down to the vmfs folder, then the vms folder. This may take some time to refresh.
6.Select the check box next to the cluster1.dsk file.
7.Click Copy.
8.Click Paste.
9.When the copy process is complete, select the check box next to the file copy of cluster1.dsk.
10.Click the Edit Properties button.
11.Change the filename to cluster2.dsk.
12.Click OK.
13.Close the Manage Files window.
This concludes the cloning process. Now continue with creating the second node virtual machine

Creating the Second Node Virtual Machine
Create a new virtual machine as follows:

1.On the management interface's Overview page, click Add Virtual Machine.
2.Keep the default Guest Operating System selection of Microsoft Windows 2000 Server.
3.Change the Display Name field to describe the virtual machine — for example, SCS Node 2 (Kena).
4.Change the Location to
home/<user>/vmware/cluster2/cluster2.vmx
5.Click Next.
6.Select the number of processors you want the guest operating system to use, up to 2.
7.Change Memory to show the amount of RAM you want to allocate to this virtual machine.
8.Click Next.
9.Click Existing to attach an existing virtual disk to this virtual machine.
10.From the Virtual Disk Image drop-down list, choose cluster2.dsk.
11.Choose the virtual SCSI node to which you want to attach the virtual disk.
12.Click Next.
Virtual Disk Configuration —
You need a shared SCSI controller and shared SCSI disks for shared access to clustered services and data.

To add a shared SCSI controller and shared SCSI disks, click the Hardware tab for this virtual machine, then take the following steps:

1.Click Add Device.
2.2. Click Hard Disk.
3.Click Blank to create a new virtual disk.
4.4. Choose the VMFS Volume on which you want to store the virtual disk.
5.Give the virtual disk image a unique name — for example, quorum.dsk.
6.Enter the appropriate value in the Capacity field.
7.Choose the virtual SCSI node to which you want to attach the virtual disk.
Note: Shared disks must be attached to a separate SCSI controller. Select SCSI 1:1.

8.By default the disk mode is set to persistent. Click Persistent to verify the disk mode.
9.Click OK.
Note: A new virtual disk and SCSI Controller 1 are now visible on the hardware tab.

10.Click Edit next to SCSI Controller 1 to change the bus sharing from none to virtual.
11.From the Bus Sharing drop-down list select virtual, then click OK.
Repeat step 1-step 9 create an additional shared virtual disk using SCSI 1:2 with the filename shared2.dsk.

Network Device Configuration —
You need an additional virtual network adapter to be used by Microsoft Cluster Service to maintain the cluster heartbeat. To add this adapter, click the Hardware tab for this virtual machine, then take the following steps:

1.Click Add Device.
2.Click Network Adapter.
3.From the Device Binding drop-down list choose vmnet_0. This attaches the second Ethernet adapter to a private network between the cluster nodes.
4.Click OK.
You have created the second cluster node virtual machine.

Go to the management interface's Overview page. The management interface should list both virtual machines and show them powered off.

Installing Microsoft Cluster Service
1.Start the node 1 virtual machine.
2.Follow the Windows 2000 Advanced Server mini-setup prompts to enter Advanced Server's serial number, the host name (Portsaid) and the IP addresses. Note that you need to enter the addresses for both public and private network adapters.
For the public network adapter, enter an IP address that belongs to the physical network.

For the private IP address, you may use an address like 192.168.x.x with a class C subnet mask (255.255.255.0).

3.At the end of the process, Windows automatically reboots.
4.Start the Disk Administrator and change both shared disks to basic disks.
5.Format both shared virtual disks with NTFS if they are not already formatted.
6.Assign the first shared disk to Q: (quorum) and the second disk to R:.
If you have joined this virtual machine to an existing Active Directory domain, skip to step 11.

7.Run dcpromo.exe from the command prompt. This starts the Active Directory Wizard.
8.Set up the current machine as a domain controller. For the domain name, use something like vmcluster.domain.com where domain.com is your DNS domain and vmcluster is your Active Directory domain. This node may be setup as a new domain tree and also a new domain forest, or it may join existing ones.
9.Make sure the DNS server is installed.
10.Set the domain permissions as mixed mode unless you plan otherwise.
11.To add a cluster services account in the domain, go to Programs > Administrative Tools > Active Directory Users and Computers.
12.Add an account named cluster, check User cannot change password and Password never expires.
13.Insert the Windows 2000 Advanced Server CD in the server's CD-ROM drive.
14.Go to Control Panel > Add/Remove Programs.
15.Select Add/Remove Windows Components.
16.Check the Cluster Service component.
17.Click Next. Follow the prompts to install the service.
18.As you configure Cluster Service, choose Form a New Cluster.
19.Specify the cluster name (Arish)
20.Specify the cluster IP address. This is the address that will represent the cluster. It must be on the same network as that of the vmnic0.
21.Specify the cluster service account created above.
22.Specify that both shared disks should be managed by the cluster service.
23.Indicate the shared disk (Q:) to be the quorum disk.
24.Specify which network adapter is public and which is private.
25.Stop the cluster service on the local node (from Cluster Manager, right-click the node name), so the second virtual machine can access the shared disks.
26.Start the node 2 virtual machine.
27.Repeat step 2 and step 3 above.
28.Start the Disk Administrator and assign the first shared disk to Q: (quorum) and the second disk to R:.
29.Start dcpromo.exe and add this virtual machine as a domain controller in the same domain created in step 8 above or add it to an existing domain. You must match the setup done in step 8.
30.In the node 1 virtual machine, start the cluster service by reversing step 25 above.
31.In the node 2 virtual machine, repeat step 14-step 24 above with one exception: In step 18, select Join a Cluster.
This concludes the Microsoft Cluster Service installation and configuration.

Running Microsoft Cluster Service
Microsoft Cluster Service should operate normally in the virtual machine once it is installed.

Note: Some disk errors are recorded in the Windows event log in normal operation. These error messages have a format similar to
The driver detected a controller error on
\Device\Scsi\BusLogic3
They should be reported periodically only on the passive node of the cluster and should also be reported when the passive node is taking over during a failover. The errors are reported because the active node of the cluster has reserved the shared virtual disk(s). The passive node periodically probes the shared disk and receives a SCSI reservation conflict error. This is normal operation.

Two Nodes with Microsoft Cluster Service on Separate ESX Server Machines
This procedure creates a two node cluster in virtual machines that will run on two separate ESX Server machines. It uses the same naming conventions as in the previous procedure.

In addition, the physical shared storage is either:

•Shared SCSI
•A storage area network (SAN)
For this exercise the VMFS partition for the internal storage on each ESX Server computer is labeled vms. The VMFS partition for the shared storage is labeled sharedfs.

•The VMFS partition for the internal storage on each ESX Server machine is labeled vms.
•The VMFS partition for the shared storage is labeled sharedfs.
Each ESX Server machine must have an additional physical network adapter assigned to the virtual machines to use for the private network that monitors the heartbeat. The procedure assumes this network adapter uses the device named vmnic1. You should connect the private network adapter to a separate network from that used by the public network adapter.

Creating the First Node's Base Virtual Machine
Follow the procedure in Creating the First Node's Base Virtual Machine, with the following changes:

•In the Virtual Disk Configuration section, in step 10 click Edit next to SCSI Controller 1 to change the bus sharing from none to physical instead of virtual. From the Bus Sharing drop-down list select physical, then click OK.
•In the Network Device Configuration section, in step 3 use vmnic1 instead of vmnet_0 as the device used by Ethernet Adapter 1.
•Access the virtual machine menu by clicking the arrow to the right of the virtual machine icon. Choose Configure Options. Under Verbose Options, click the click here link.
Change the specifications of scsi1:1.name and scsi1:2.name to use the strict vmhba name (for example, vmhba0:1:0:1:shared1.dsk) for the VMFS partition, rather than the VMFS name (for example, sharedfs:shared1.dsk). The reason for this change is that if one ESX Server machine reboots while a virtual machine on the other physical machine is reserving the shared SCSI disk, ESX Server cannot read the VMFS name on the shared disk when it is loaded and initialized. If the shared virtual disk is not specified using the full vmhba name, ESX Server cannot determine the disk specified by the VMFS name and gives an error when restarting the virtual machine.

When you have made these changes, click OK.

In addition to these minor changes, you need to change the access rights of the VMFS partition where you store the shared virtual disks. By default VMFS partitions are configured for public access. In order to support clustering, the VMFS partition needs to be configured for shared access.

Take the following steps to change the access settings for the VMFS partition:

1.From the management interface click the Options tab
2.Click Storage Configuration.
3.Identify the disk volume that contains the VMFS partition where the shared virtual disks are stored. Click Edit for the disk volume.
4.From the VMFS Access drop-down list, choose Shared.
5.Click OK.
You have created the first cluster node virtual machine.

Installing the Guest Operating System
Follow the procedure in Installing the Guest Operating System.

Cloning the Virtual Machine
Now that you have a virtual machine with Windows 2000 Advanced Server installed, you can save time by cloning this virtual machine as follows:

1.Run sysprep.exe, which is available on the Windows 2000 CD in the \support\tools\deploy.cab file. This strips the Security ID assigned to the guest operating system and resets the machine information as well as the TCP/IP network configuration.
2.Shut down the guest operating system and power off the virtual machine.
3.Go to the console of the second ESX Server machine. This is where you will copy the virtual disk that resulted from creating the first node.
4.Log on as root.
5.Change directories: cd /vmfs/vms
This assumes that the internal storage for the second server is in a VMFS partition labeled vms.

6.Use the ftp command: ftp <server1-hostname>
7.Change directories: cd /vmfs/vms
This changes the current directory to the VMFS partition on the first server where you created the first node's virtual disk.

8.Set the type to binary: bin
This sets the transfer mode to binary. If you use text transfer mode, the virtual disk may not be usable on the target server.

9.Type: hash on
This turns on the display of a series of hash signs as a transfer progress indicator.

10.Retrieve the virtual disk file: get cluster1.dsk
This initiates the transfer of the virtual disk file to the current directory on the second ESX Server machine.

11.Quit the ftp session: bye
After the file transfer is completed, type the bye command to end the FTP session.

12.Rename the file: mv cluster1.dsk cluster2.dsk
This renames the virtual disk to cluster2.dsk.

This concludes the cloning process. Continue with creating the second node virtual machine.

Creating the Second Node Virtual Machine
Follow the procedure in Creating the First Node's Base Virtual Machine, noting the following differences:

•In the Virtual Disk Configuration section, step 10, click Edit next to SCSI Controller 1 to change the bus sharing from none to physical instead of virtual. From the Bus Sharing drop-down list, choose physical, then click OK.
•In the Network Device Configuration section, step 3, from the Device Binding drop-down list, choose vmnic1 instead of vmnet_0. This attaches the second Ethernet adapter to the second physical adapter designated for virtual machine use. This is used to create a private network between the cluster nodes.
•Change the specifications of scsi1:1.name and scsi1:2.name as you did when creating the first node's base virtual machine.

Clustering Using a Raw SCSI Disk
The shared disk used for clustering can also be a complete shared SCSI disk, rather than a VMFS file on a shared disk. Using a raw SCSI disk as a shared disk may simplify initial setup. It may be especially useful for importing an existing physical cluster that already has cluster data on a SCSI disk. In addition, using a raw SCSI disk as a shared disk allows a virtual machine to participate in a cluster with a physical machine. For example, the virtual machine can be used as the passive node for a physical machine that is the active node.

In order for the virtual machine to access a physical disk, the instructions in the Virtual Disk Configuration section on Virtual Disk Configuration — should be replaced with the following steps:

To add a physical SCSI controller and shared raw SCSI disks, go to the Hardware tab and take the following steps:

1.Click Add Device.
2.Click Hard disk.
3.Click System LUN/Disk to give your virtual machine direct access to a SAN or shared storage volume.
4.Choose the LUN/Partition you want to attach to this VM as a raw disk.
Note: In ESX Server, physical disks are identified by a vmhba number. For example, vmhba0:1:2:1 means physical adapter vmhba0, target 1, LUN 2, partition 1. When the final number is :0, that indicates you are specifying the entire disk, rather than a particular partition.

5.Choose the virtual SCSI node to which you want to attach the raw disk.
Note: Shared disks must be attached to a separate SCSI controller from the system disk. Select, SCSI 1:1

6.Click OK.
A new virtual disk and SCSI Controller 1 appear on the Hardware tab.

7.Click Edit next to SCSI Controller 1 to change the bus sharing from none to physical.
8.From the Bus Sharing drop-down list choose physical, then click OK.
Setting the bus sharing to physical makes sure that all the SCSI reserve and reset commands go through to the physical disk.

Repeat step 1-step 8 to create an additional shared raw disk using SCSI 1:2.

You have completed the virtual machine configuration.

Installing Microsoft Cluster Service
Follow the procedure in Installing Microsoft Cluster Service.

Additional Notes for Clustering Across Physical Machines
•For maximum flexibility, put each shared virtual disk on a separate VMFS on a separate SCSI disk (or LUN). The reason for this is that SCSI reservation can be done only at the granularity of a complete SCSI disk or LUN. If two shared virtual disks are on the same VMFS, reserving one disk will also reserve the other disk. If you choose to put more than one shared disk on the same VMFS (as in the example installation above), always put these virtual disks in the same MSCS resource group. That ensures the disks are always reserved by the same computer and are always failed over at the same time.
•Ensure that the swap file for each ESX Server machine is not on any of the physical shared disks. Since the shared disks are reserved by one host or another, the swap file may become inaccessible to one host or the other during the operation of the cluster.
•If you are accessing the shared disks on a storage network (SAN) using a QLogic host bus adapter, you must use particular values for some QLogic configuration settings. Reboot your physical machine and enter the QLogic configuration utility during bootup. Under Advanced Configuration Settings, ensure that
◦Enable Target Reset is set to Yes.
◦Full LIP Login is set to Yes.
◦Full LIP Reset is set to No.
•If you are accessing the shared disks on a SAN using an Emulex host bus adapter, you need to take the following extra steps:
a.On the Options > Advanced Settings page, Set DiskUseDeviceReset to 1.
b.Supply an extra parameter to the Emulex driver when it is loaded. You do this by editing the file /etc/vmware/hwconfig. First, identify the bus, slot and function holding the first (or only) Emulex card. You can find this information by looking at the Startup Profile page. Then add a line with the format
device.vmnix.6.14.0.options = "lpfc_delay_rsp_err=0"

to the end of /etc/vmware/hwconfig. Here, the numbers 6.14.0 specify the bus, slot and function where the Emulex card is located. If you have more than one Emulex card, you should have only a line referencing the first card.

•If you are doing multipathing and can access your shared cluster disk via multiple HBAs from a host, then you need to modify another configuration file. Go to the Options > Advanced Settings page and set DiskResetOnFailover to 1. If an HBA failover occurs on the host, this option causes a SCSI bus reset to be issued after the failover happens. This bus reset allows the second HBA to access a shared disk, even if the disk has been reserved by the first HBA. The clustering software then automatically reserves the disk again via the second HBA.

Running Microsoft Cluster Service
Microsoft Cluster Service should operate normally in the virtual machines once it is installed.

Note: Some disk errors are recorded in the Windows event log in normal operation. These error messages have a format similar to
The driver detected a controller error on
\Device\Scsi\BusLogic3
They should be reported periodically only on the passive node of the cluster and should also be reported when the passive node is taking over during a failover. The errors are reported because the active node of the cluster has reserved the shared virtual disk. The passive node periodically probes the shared disk and receives a SCSI reservation conflict error.

VMFS Locking and SCSI Reservation
For a shared SCSI disk that can be accessed by multiple ESX Server machines, two kinds of locking may be in use. These two kinds of locking are somewhat independent and can cause confusion. The shared SCSI disk may be on shared SCSI bus or, more likely, on a storage area network (SAN).

VMFS File System Locking
The first kind of locking is VMFS file system locking. ESX Server locks VMFS file systems on a server level when a VMFS file system is configured as a public or shared file system. This locking is done in order to ensure that there is no corruption caused by multiple accesses to the file system by different hosts.

If a VMFS-1 volume is configured in public mode, only one server can ever access that VMFS at a time. If one server is accessing the VMFS-1 volume, through a virtual machine or a file system command, then a file system operation by another host fails. For example, a vmkfstools command fails with a message that says:
vmkfstools: file system is locked by another server. Use 'vmkfstools --recover' to unlock file system if no other server is accessing

Typically, you should not run vmkfstools --recover at this point, since another host is actually using the file system. The error message simply indicates that this server cannot access the VMFS until the other server has finished accessing it. However, if a server fails while accessing the file system, the file system may stay in the locked state and you may need to run vmkfstools --recover.

In a public VMFS-2 volume, locking is at a per-file level, resulting in fewer locking issues. However, you may still encounter the preceding message and may need to use vmkfstools --recover, if a server fails.

If a VMFS is used to store a virtual disk that is accessed by multiple virtual machines on multiple physical servers for the purposes of failover clustering, the VMFS should be configured as a shared file system. Then, the locking protocol is slightly relaxed to allow multiple virtual machines on different servers to access the same VMFS file at the same time. However, file system commands do the same locking as with public file systems (that is, per-VMFS in VMFS-1 volumes and per-file in VMFS-2 volumes).

Additionally, when multiple virtual machines access the VMFS, the VMFS file system enters a read-only mode in which it is impossible to create, delete or change the size of files. However, the contents of the individual files can still be modified. If you later want to create or remove VMFS files, you must stop all virtual machines using the VMFS and re-enter writable mode by using this command:
vmkfstools --config writable vmhba0:1:0:0

Substitute the name of the appropriate disk or VMFS in place of vmhba0:1:0:0.

Locking at SCSI Disk Level
The second kind of locking is locking at the SCSI disk level, which is called SCSI disk reservation.

Any server connected to a SCSI disk can issue a SCSI command to reserve the disk. If no other server is already reserving the disk, the current server obtains a reservation on the disk. As long as that reservation exists, no other server can access the disk. All SCSI commands to that disk by other servers fail with an appropriate error code.

If a vmkfstools command is attempted on a VMFS on a disk that is reserved by another server, the vmkfstools command fails with a message that says:
vmkfstools: shared SCSI disk is reserved by another server. Use 'vmkfstools -L release/reset' to end reservation if no other server is using the SCSI reservation

Similarly, a virtual machine fails to start if its virtual boot disk is stored on a physical disk that is reserved by another host.

Most applications do not ever reserve a SCSI disk. However, failover clustering software reserves SCSI disks in order to ensure that only the active node is able to access the shared SCSI disk. Therefore, you should expect that the shared disk in a physical clustering setup is reserved when the cluster is active. Similarly, for a virtual machine cluster that is running across physical machines, reservations by the clustering software are transmitted through to the physical shared disk.

If you encounter a disk that is reserved unexpectedly, you should try to determine if some clustering software has explicitly reserved the disk. If not, you can release the reservation on the server that has the reservation by running a command in this format:
vmkfstools -L release vmhba0:1:0:0

Substitute the name of the appropriate disk or VMFS in place of vmhba0:1:0:0. If you cannot determine which server holds the reservation, you may be able to eliminate the reservation by issuing a SCSI bus reset on any server machine using a command in this format:
vmkfstools -L reset vmhba0:1:0:0

Using LUN Masking to Avoid Locking Issues
Locking issues are especially likely to happen on a SAN, where multiple users may be accessing some of the same disks or may mistakenly access a disk assigned to another user.

It is often helpful to use LUN masking or zoning to limit what disks are visible to each server in the system, and therefore reduce the ways in which one user can affect another user. In particular, the use of LUN masking or zoning can help prevent problems such as those described above in which one server unexpectedly locks or reserves the wrong SCSI disk.
 

微软对instsrv.exe的官方说明如下：

Installs and uninstalls executable services and assigns names to them. 也就是个加载services的小东东。

其用法如下：instsrv <service name> <srvany path> （这里的srvany path也就是工具srvany.exe的路径了）

解释下，srvany.exe是微软出的用于将一个程序注册为一个服务的小程序。它可以实现讲任何程序设置成服务启动。具体用法可以自己想下哦。

解释完毕，现在来实际的操作了吧。

1.讲所需要的工具instsrv.exe和srvany.exe放在一个文件夹内，在这我放在d:\tools中吧。

2.需要了解vmware.exe的安装路径，以本机为例：D:\VMware Workstation\vmware.exe
要启动的虚 拟机配置文件路径，我的2003虚拟机的配置文件windows2003.vmx的路径是D:\vmare\windows2003\windows server 2003.vmx

3.新建服务

假设服务名为vm_autostart，所以我的命令行是：

instsrv VM_AutoStart  D:\tools\srvany.exe

4.注册服务

在注册表中，定位到HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services \vm_autostart
新建项："Parameters"
在"Parameters"项里面，新建字符 串"Application"，
字符串的值：
"D:\VMware Workstation\vmware.exe" -x "D:\vmare\windows2003\windows server 2003.vmx"

(如果你想要开机加载两个虚拟机系统，只需要在其后再添加一个虚拟机配置文件路径即可）

5.设置虚拟机启动状态

管理工具－服务，选择vm_autostart的属性－登录，选中“本地系统帐户”，并勾选“允许服务与桌面交互”，这样，你的电脑开机后就会出现 vmware的启动界面了。

VMware VirtualCenter service does not start automatically after reboot

Symptoms

Purpose

Resolution

1. 在ESXi的控制台（物理机器面前）里输入Alt+F1。
2. 在控制台输入 unsupported 并按下回车（你看不到输入的内容）。
3. 如果你正确的输入了“unsupported”，就可以看到Tech Support模式的警告和下方的密码提示。输入root密码即可登陆。
4. 现在你可以看到命令提示符“~ #” ，编辑inetd.conf文件（输入命令  vi /etc/inetd.conf ）。
5. 找到开头是#ssh的那一行，删除注释符#，保存退出。
6. 退出vi编辑器后，输入命令  /sbin/services.sh restart  来重启管理服务。然后你就可以使用SSH客户端连接ESXi主机了。

ESXi 3.5 Update 2 以上版本  - 在Update 2 以后的版本中，services.sh已经不能重启控制SSH的inetd进程。你需要重启服务器或者使用下面的方法：执行  ps | grep inetd  来获取inetd的进程号，然后执行  kill -HUP <进程号>  使其重新读取配置文件，这时你就可以通过SSH连接服务器了。

Go to the Configuration tab of your ESX Server, then click on Virtual Machine Startup/Shutdown.

By default (I’m pretty sure) automatic startup is disabled. To enable it, click on “Properties…” on the far upper right of the window.

You’ll now see this window:

Check/Tick “Allow virtual machines to start and stop automatically with the system”.

Now, this is the bit where I nearly cried…

You know you want to “enable” your Guest OSes to automatically boot, but how? I tried clicking and dragging, right clicking for a context menu to enable “Automatic start up” and gave up.

Turns out, you need to click on the Guest OS you’d like to enable, and then click “Move Up” until it sits underneath the “Automatic startup” title. Argh!

这篇文章的背景问题在上一篇文章(“LVM.DisallowSnapshotLun” Problem when Migrating a Datastore on VMWare ESX Server)里提到了——当时不知道取什么题目好，当然现在还不是很清楚，不过会明确一点了，就是将Datastore的LUN调整成其它值(unpresent&&present)了以后，ESX就没法把这个识别到的LUN作为Datastore了。

系统日志vmkernel里有显示一些经常出现的信息：

Dec 25 13:21:55 vm0 vmkernel: 229:02:33:19.877 cpu7:1037)WARNING: LinSCSI: 4524: The physical media represented by vmhba2:0:2 has changed. The device has been re-synchronized with the system.

Dec 25 14:53:57 vm0 vmkernel: 0:00:01:56.956 cpu6:1038)LVM: ValidateDevice: [vmhba3:0:1:1] Mismatch between stored disk ID and actual disk ID
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:56.966 cpu6:1038)LVM: CheckDevSnapshot: Device vmhba3:0:1:1 is a snapshot:
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:56.973 cpu6:1038)LVM: CheckDevSnapshot: disk ID: <type 2, len 22, lun 1, devType 0, scsi 6, h(id) 16169674206390955907>
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:56.984 cpu6:1038)LVM: CheckDevSnapshot: m/d disk ID: <type 2, len 22, lun 2, devType 0, scsi 6, h(id) 16169674206390955907>
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:56.997 cpu6:1038)LVM: ValidateDevice: [vmhba3:0:1:1] Mismatch between stored disk ID and actual disk ID
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:57.008 cpu6:1038)LVM: ProbeDeviceInt: vmhba3:0:1:1 => Operation should be retried due to temporary error condition
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:57.019 cpu6:1038)LVM: ValidateDevice: [vmhba3:0:1:1] Mismatch between stored disk ID and actual disk ID
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:57.029 cpu6:1038)LVM: CheckDevSnapshot: Device vmhba3:0:1:1 is a snapshot:
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:57.037 cpu6:1038)LVM: CheckDevSnapshot: disk ID: <type 2, len 22, lun 1, devType 0, scsi 6, h(id) 16169674206390955907>
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:57.048 cpu6:1038)LVM: CheckDevSnapshot: m/d disk ID: <type 2, len 22, lun 2, devType 0, scsi 6, h(id) 16169674206390955907>
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:57.061 cpu6:1038)LVM: ValidateDevice: [vmhba3:0:1:1] Mismatch between stored disk ID and actual disk ID
Dec 25 14:53:57 vm0 vmkernel: 0:00:01:57.070 cpu6:1038)WARNING: LVM: ProbeDeviceInt: [vmhba3:0:1:1] detected as a snapshot device. Disallowing access to the LUN since resignaturing is turned off.
Dec 25 15:53:05 vm0 vmkernel: 0:00:01:11.546 cpu7:1037)LVM: 5670: Device vmhba2:0:1:1 is a snapshot :
Dec 25 15:53:05 vm0 vmkernel: 0:00:01:11.546 cpu7:1037)LVM: 5676: disk ID: <type 2, len 22, lun 1, devType 0, scsi 6, h(id) 16169674206390955907>
Dec 25 15:53:05 vm0 vmkernel: 0:00:01:11.546 cpu7:1037)LVM: 5678: m/d disk ID: <type 2, len 22, lun 2, devType 0, scsi 6, h(id) 16169674206390955907>
Dec 25 15:53:05 vm0 vmkernel: 0:00:01:11.549 cpu7:1037)LVM: 5670: Device vmhba2:0:1:1 is a snapshot:
Dec 25 15:53:05 vm0 vmkernel: 0:00:01:11.549 cpu7:1037)LVM: 5676: disk ID: <type 2, len 22, lun 1, devType 0, scsi 6, h(id) 16169674206390955907>
Dec 25 15:53:05 vm0 vmkernel: 0:00:01:11.549 cpu7:1037)LVM: 5678: m/d disk ID: <type 2, len 22, lun 2, devType 0, scsi 6, h(id) 16169674206390955907>
Dec 25 15:53:06 vm0 vmkernel: 0:00:01:11.549 cpu7:1037)WARNING: LVM: 4844: [vmhba2:0:1:1] detected as a snapshot device. Disallowing access to the LUN since resignaturing is turned off .

按照我最后的理解，这部分信息应该是最重要的。其中被强调显示标记的说明了，volume的LUN更改以后Disk ID也发生了变化，这个和ESX里存储的Disk ID出现了不一致。由于Configuration-Advanced Settings里LVM.EnableResignaturing 这个开关被设置为0(turned off)，ESX就把这个volume视作一个“新”的volume，而且是现有Datastore的snapshot而禁止加载。也正因此，可以通过把 LVM.DisallowSnapshotLun开关关闭，将这个可能是现有Datastore的snapshot的volume加载为一个新的 Datastore。这就是我对这个问题的理解。

至于什么是resignaturing呢？正常状态下， ESX的/vmfs/volumes/下面显示一些目录，这是现有的Datastore所对应的volume的挂载点(mount point)，目录名就是volume在系统里的标志（我猜测就是Disk ID），例如：44b0d972-e8adad27-adee-00132165b7cb、44b65ab5-928b798d- fbb7-00132165b7cb。这些volumes可能是位于本地磁盘的VMFS的分区，也可能是位于SAN上的磁盘分区。对于添加的空白 volume，ESX会自动signature这个volume的一些信息，成为这个标识（即目录名），并挂载到/vmfs/volumns里；对于那些 非空白的，可能是其它Datastore的snapshot的volume，除非系统允许RE-signature，即重新signature信息，重新 挂载到ESX中。但是resignature后那个标识信息将不是原来的，而是新的了。换句话说要访问这个“新”的Datastore就要通过新的目录名 来访问了。

在/vmfs/volumes/下面还有一些以storage1之类的链接名，这些链接名就是VIC里面所看到的Datastore的名称。如果通过VIC将名称改了，链接名也马上发上改变。

知道问题的原理以后，也就知道该怎么解决了。具体详细的过程可以参考这篇文章（Resignaturing VMFS3 Volumes That Are Not Snapshots）。

在操作之前有几点要注意的：

• VIC连接的最好是ESX本身的地址，而不是Virtual Center的地址；
• 把所有的虚拟机停掉。把所有ESX启动后会自动开机的虚拟机改成不要自动开机，因为后面最好是重启一下；
• 重启服务器的方法：通过VIC 右键点击具体的ESX Host选择reboot或者在登录服务器控制台执行reboot命令，而不要在SSH远程登录终端里halt（这样不能完全关闭，控制台显示已经halt了，但是SSH远程还是可以登录，VIC也还可以连接）

由于我之前用了DisallowSnapshotLun的办法，所以我认为的操作方法和文章提到的有点差别，具体是这样的：

1. 将虚拟机停掉
2. VIC连上ESX， 将LVM.DisallowSnapshotLun开关设置为1，将LVM.EnableResignaturing开关设置为1，Rescan，将 LVM.EnableResignaturing设置回0；（由于我实际上是连到VirtualCenter操作，并且rescan之前没有停掉虚拟机， 所以rescan之后原先的Datastore出现错误，同时VirtualCenter和ESX断开连接，只好重启一遍）
3. ESX重启后，用VIC连上，在Configuration-Storage(SCSI, SAN and NFS)就可以看到以snap开头的一个新的DataStore了，可以将这个DataStore的名字改成自己喜欢的；
4. 由于旧的DataStore已经失效或者说找不到了，虚拟主机列表里所有主机都显示为不可用，而且不能显示原先的名字；
5. 重启ESX服务器，在操作系统内核选择菜单选择进入Console模式；
6. 以root登录，查看/vmfs/volumes/下新的目录名（类似44b65ab5-928b798d-fbb7-00132165b7cb的磁盘信息），记下来；
7. 修改/etc/vmware/hostd/vmInventory.xml文件（这个文件保存的是所有虚拟主机的配置文件路径），将位于已经失效的DataStore的volume上的虚拟机的配置路径(<<vmxCfgPath>)修改成现在的；
8. 重启ESX服务器，进入正常状态，用VIC连上Virtual Center，可以看到现在虚拟机都回来了；
9. 使用文章介绍的方法把失效的DataStore删除，如果无法删除可能是Virtual Machines里还有Template，就先把Template Remove from Inventory ；
10. 将每台虚拟机启动一下，会出现关于UUID是要keep，还是renew的的时候，选择keep就好了；
11. 最后就是将原先修改的自动开机的设置改回去。

目前这样就算是基本完成了，最后再补充几点：

• 操作步骤里5,6,7所产生的效果就是可以将虚拟主机清单指向新的DataStore，而不用像文章里所说的一样，将虚拟主机Remove from Inventory，然后再从DataStore中浏览出来，Add to Inventory；
• 步骤3,4其实可以不要，即直接重启到Console，修改好了以后再重启到正常状态（服务器重启一次要好久，大部分的时间都是浪费在等待重启上了）。

还需要解决的问题：

• 如果能知道ESX是如果进行signature，以及signature后的结果是存放到哪里的，就可以直接在Console模式下去编辑配置文件修改原先的DataStore的Disk ID，就可以减少Add && Remove的操作了；
• 如果能知道Template添加到Inventory后是保存到那个配置文件的话，就可以向修改vmInventory一样，直接修改Template的路径，而不用remove以后再Add，然后还要Convert to template。