We are investigating an issue where a SQL-based report is taking excessively long on a sporadic basis.
At first glance, I noticed bursts of high latency at the VM-level. Checking the host and array, these latency numbers do not seem to appear, falling on average below 10ms, even during reporting with a burst here and there to 15-50ms but not lasting more than a second.
In addition to disk, I noticed significant memory ballooning. The DBA team reported that they did not tune SQL appropriately and that they were forcing swap activity very aggressively. I thought that since their pagefile.sys lives on the array with the database (this is an XIV, I don't control what spindles the .vmdk's live on) that perhaps simulataneous paging and database I/O were conflicting and causing the latency bursts.
However...I spun up IOMETER when they weren't doing any SQL processing. I threw 7000 IOPS at a new NTFS-formatted .vmdk on the same VM & Datastore, and lo and behold I'm still seeing latency spikes at the VM. When I check the host->array, they see the expected IOPS but report 0ms latency.
Are these numbers as reported by vCenter correct? I ran perfmon during this exercise and saw <10 pages/sec, and 0 read/write disk queue lengths. Plenty of RAM and CPU cycles were available to the guest during this exercise, 0% pagefile usage. No other VM's were figthing for disk I/O on this volume.
The test ran from around 3:45PM to 3:55PM in the attached image. Has anyone else seen these latency bursts? I noticed today that the guy who built the machine chose paravirtulization for the SCSI adapter. I've never used that, never needed to, is it suspect? Also discovered this is on an 8TB VMFS5 volume. It's "new" VMFS5, 1MB block size. 2008R2 guest.
ESX 5.0.0 469512
Thanks!
D