Hello fellow vmers,
I have an intriguing issue. Last week i performed a full power cycle test for my hardware.(10 hp bl460c g7 blades) all running vsphere 5 on a vcenter server.
After the PowerOn event and things settled down i observed that some of my vms have their network in a disconnected state. I am talking here about the connected at power on mark checked but connected not checked. Upon further investigation i couldn't find any pattern regarding from which dvPortGroup or which vSphere host did the vms with the errors came from. I did a quick powershell script that connected all the missing networks and that was it.
I now wander what happened and most importantly why? Let me explain a bit more about my infrastructure:
1. 10 vSphere hosts. v5.0
2. 100% of the vms are on local storage. i only use a small iscsi shared datastore for when i need to migrate something live from one host to another.
3. before my power cycle i unmonted the shared datastore to be shure that i don't enconunter any issues.
4. digging through the logs i found a lot of errors like: [2B3C5B90 warning 'NetworkProvider'] Error saving dvport 1d 23 14 50 4e 2f 08 97-60 47 78 ac c2 f2 91 2d-1717 to file /vmfs/volumes/4ee5dc5e-518c5294-78fe-0010185df286/.dvsData/1d 23 14 50 4e 2f 08 97-60 47 78 ac c2 f2 91 2d/1717 repeated hundreads of times and the dvPortID matched the vms that didn't have the network cards connected.
5. The /vmfs/volumes/4ee5dc5e-518c5294-78fe-0010185df286/ matched the ID for the disconnected iscsi shared datastore
a. What does really happens when i power on a vm connected to the dvs? Do all the hosts from the dvs update on disk the dvs config and if one of them fails to update the port doesn't get connected?
b. Why does it try to update the dvs config from an unmounted datastore?
c. When i powered on the infrastructure i've set up the delay between the vms to 5seconds. could this affect the lock on the disks to the point of failing to update the dvs and getting the port not connected? (i have ~500 vms / ~50/ server that powered at 5 sec each)
Thank you for your attention
Ion