Hello
We are in the process of migrating from vsphere 4.1 to vsphere 6.0U2 .The new 6.0U2 platform is located in another physical datacenter. The two data centers are connected with MPLS VPLS (over 2 x 1Gbps physical link) - that is more like transparent or stretched LAN. So a VM in the old datacenter in the same vlan and subnet as a VM in the New datacenter appears to be on the same lan segment. latency between the two sites is < 3 milliseconds at 1500 MTU non fragmented packets. Client is not willing to enable jumbo frames across its core network.
We have set up a temporary 5.5U3 esxi hosts and associated vcenter5 accordingly. As per agreed procedures and policies our client wants to maintain attachment or management of the existing 4.1 esxi hosts to the vcenter4.1. What we have done then is to attach the 5.5U3 esxi hosts to the same storage as the 4.1 hosts. So our client powers off scheduled VMs to be migrated removes it from the 4.1 inventory and registers it in the 5.5U3 inventory. We then update the network settings - (client was using standard switches on all 50+ blades: we have setup 2 dvswitches each with 2 uplinks in the 5.5U3 to consolidate all the port groups that were on the 4.1), trim off old snapshots as well upgrade the VM hardware compatibility. All these have to be done in 10mins if the VM was powered on. The VM is then powered on a 5.5U3 hosts in old datacenter and subsequently migrated to a 5.5U3 hosts in the new datacenter using vMotion.
vMotion works to our expectation and we are able to migrate live VM - about 500GB in 3hrs 35mins at transfer rates peaking at 650Mbps with spikes to 850Mbps. Each 5.5U3 has 4 physical nics which we have assigned to 2 distributed switch Uplinks (nic0 and nic1 for dvSwitch01 and nic2 and nic3 for dvSwitch02). vmk for management is located on dvSwitch01. vmk for vMotion is located on dvSwitch02.
The issue we are facing is cold migration - powered off VM migration is VERY VERY SLOW with maximum transfer rate of 90Mbps with spikes to 200Mbps. I understand cold migration traffic traverses through management network. We have done test to see if there are bottle necks on that network but our tests shows no such issues exist as we can push 800Mbps traffic consistently.
Any assistance, suggestion and pointers will be highly appreciated.
cheers.