Hi all
Over the last month or two I have been seeing a strange performance issue on some Linux hosts (running the latest kernel from Debian 6.0 (squeeze) and 7.0 (wheezy)).
It manifests itself as very slow logins, either via the console or ssh (no, it's not an SSH and DNS issue, it happens on the console too). For example, logging in at the console, the password prompt appears instantly, as does /etc/motd, but the shell prompt takes anything from 20 seconds to 2 minutes to appear. It also shows up as very slow appearance of the SMTP banner on the mail server - it can take 30 seconds or more for the Postfix banner to show up if you connect to port 25 on the mail server.
Running various tests with the 'dd' command shows acceptable I/O performance (on the order of 100MB+/sec)
Running a short program that does a lot of floating point calculations in a tight loop to stress the CPU shows normal performance.
The 'uptime' command reports a load average of 0.00, free reports there's still free memory, top shows that there are no processes hogging the resources.
From the host, you can see this is indeed true from the performance graphs - CPU usage is almost zero, I/O is almost zero, and there are no memory issues. (Running the 'dd' command as described above, and cpustress tool as above shows a corresponding peak in disk I/O and CPU).
The host itself is using less than 5% CPU for all guests and there is still free memory, and disk usage for the entire host is similarly low, so another guest using a lot of resources can be ruled out.
Kernel command line parameters include "noapic acpi=off" and the host has been set into "performance" in the power profile.
Rebooting the guest seems to solve the problem for about 4 to 8 hours or so, and then the problem reappears. The same problem does not happen at all with physical hosts running the same operating system or another instance of Debian running using a different virtualization product.
Any ideas?