UPDATE: Problem has been fixed, but I wouldn't say the mystery has been solved. Please see my entry near the end of this thread for further details.
summary at bottom
Current setup:
H/W: 4x Cisco UCS B200 blades with VIC (Palo)
S/W: vSphere 5 (ESXi 5.0u1, vCenter 5.0)
vmKernel setup:
vmk0 - Management Network (mgmt traffic only) - vSwitch0 - vmnic0, vmnic5; no vlan tagging (on Network 2); IP: 10.x.x.101...104
vmk1 - vMotion (vMotion traffic only) - vSwitch1 - vmnic4 (primary), vmnic9 (stand-by); no vlan tagging (on Network 3); IP: 10.a.a.1...4/24
vmk2 - FaultTol (FT log traffic only) - vSwitch1 - vmnic9 (primary), vmnic4 (stand-by); no vlan tagging (on Network 3); IP: 10.a.a.64...67/24
*note: 10.a.a.x/24 is not a routed network
Problem and steps already taken:
vMotion is not (no longer) working. Any vMotion task fails at 9% with "The VMotion failed because ESX hosts were not able to connect over the VMotion network". Initially, vMotion worked (as evident by the auto-migrations it did the first night). What changed was that Host1 was on 5.0.0 and the other 3 were on 5.0u1. I later blew away Host1 and recreated it from scratch on 5.0u1. I doubt this caused the issue; I just thought it should be put out there.
This cluster is not really in production yet, but I've been vetting it to make sure everything is good before I give it the green light. Of the 4 hosts currently in the cluster, I can not vMotion between any of them. On the tech support console, vmkping can hit the local host vMotion IP but is unable to hit any other host (this applies with the FT log ports as well). I went through the network config once again to establish that all was as it should be. In order to do additional testing, I made a VM Network on the hosts, using as similar config as I could to the vMotion ports (vS1, vmnic4 PRI, vmnic9 SB) and created two VMs on separate hosts). The VMs, assigned IPs 10.a.a.51 and 10.a.a.61 were able to ping the vMotion and FT IPs of their hosts (as I would have expected). The odd thing: The VMs are also able to ping one-another just fine. This seems to establish in my mind that network connectivity is solid...
What am I missing? Any ideas out there?
Summary: vmkping is only able to hit local host's vMotion vmk-IP. VMs on same network as vMotion vmk port can ping one another from separate hosts, but can only still ping their local host's vmk-IPs. I'm confused, since this seems to validate the physical network setup.