Virtual Interrupt Coalescing


The performance of high-IOPS workloads can be improved, in terms of higher throughput and lower CPU utilization, by turning on interrupt coalescing for Fibre Channel HBAs. This optimization has been in place since ESX 3.5.

Analogous to this, in order to further reduce the CPU cost of I/O in high-IOPS workloads, ESX 4.0 implements virtual interrupt coalescing to allow for some batching of I/O completions to happen at the guest level. In order to avoid the undesirable side-effect of increasing latency, this coalescing only occurs when I/O levels exceed a threshold. This threshold is set high enough that performance does not degrade under trickle I/O or latency-bound conditions.

Another optimization, related to inter-processor interrupt (IPI) coalescing, addresses a problem specific to virtual environments. Hardware interrupts that signal I/O completions are routed by the ESX storage stack to the virtual machine that issued the I/O. If the interrupt is not received on the physical CPU where the relevant VMM is executing, an IPI must be sent to the pCPU executing that VMM. As processor core density increases, it becomes more likely that hardware interrupts will be received on processors not running the target virtual machine, thus increasing the number of times the IPIs need to be issued. In ESX 4.0, we provide a mechanism to reduce the number of IPIs issued using the time-stamp of the last interrupt that was sent to the guest operating system. Before sending an IPI, we check the current time against this timestamp. If the time difference is greater than a threshold, 100μs by default, the IPI is posted. Otherwise, the IPI is delayed and the VMM has the opportunity to batch it with other I/O completions at a later time. This reduces the overall number of IPIs while bounding the latency of notifying the guest operating system about an I/O completion. As with other interrupts, lowering the rate of IPIs reduces the CPU cost for this benchmark.

For a detailed discussion of interrupt coalescing improvements, see the following publication:

Irfan Ahmad, Ajay Gulati, and Ali Mashtizadeh, “Improving Performance with Interrupt Coalescing for Virtual Machine Disk I/O in VMware ESX Server,” Second International Workshop on Virtualization Performance: Analysis, Characterization, and Tools (VPACT’09), held with IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2009.

Advertisements

About PiroNet

Didier Pironet is an independent blogger and freelancer with +15 years of IT industry experience. Didier is also a former VMware inc. employee where he specialised in Datacenter and Cloud Infrastructure products as well as Infrastructure, Operations and IT Business Management products. Didier is passionate about technologies and he is found to be a creative and a visionary thinker, expressing with passion and excitement, hopefully inspiring and enrolling people to innovation and change.
This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Virtual Interrupt Coalescing

  1. Irfan says:

    Couple new things to mention 🙂

    First, the full conference paper for this work has been published at USENIX ATC ’11. Available at http://www.usenix.org/events/atc11/stream/tech

    The video is also now up. Available at: http://www.usenix.org/events/atc11/stream/ahmad/index.html

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s