“BUG: soft lockup detected on CPU” in a Linux Virtual Machine: Cause and Fix

The Symptom

When a Linux kernel runs in a symmetric multiprocessing (SMP) virtual machine, you may see messages like BUG: soft lockup detected on CPU#1! in the system log. The exact wording varies by kernel, and the message may include a kernel stack backtrace.

What Causes It

Many Linux kernels run a soft lockup watchdog thread and log this message if that thread isn’t scheduled for more than 10 seconds. On a physical server, a soft lockup usually points to a kernel or hardware bug. In a virtual machine, it more often means the host is heavily overcommitted (especially on memory) or there’s other virtualization overhead.

The Fix

Soft lockup messages are not kernel panics and can generally be ignored. Some kernels let you raise the threshold, where time is the number of seconds before a soft lockup is reported (default is usually 10):

echo time > /proc/sys/kernel/softlockup_thresh

If they’re frequent, look at memory and CPU overcommitment on the host.

What the Error Report Looked Like

Here’s the start of the ABRT report from an Oracle Linux (UEK) VM where I saw this (memory maps and limits trimmed):

abrt_version:    2.0.8
cmdline:         /usr/bin/ksh ./update_archive.ksh RMUAT
executable:      /bin/ksh93
kernel:          2.6.39-400.212.1.el6uek.x86_64
last_occurrence: 1403155682
pid:             25607
pwd:             /oracle/scripts/update_archive
time:            Thu 19 Jun 2014 12:28:02 AM CDT
uid:             502
username:        oracle
sosreport.tar.xz: Binary file, 3959288 bytes

environ:
:SHELL=/bin/sh
:USER=oracle
:LD_LIBRARY_PATH=/lib:/usr/lib:/oracle/product/11.2.0/dbhome_1/lib
:PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin
:PWD=/oracle/scripts/update_archive
:JAVA_HOME=/usr/java
:LANG=en_US.UTF-8
:HOME=/home/oracle
:'NLS_DATE_FORMAT=MM-DD-YYYY HH24:MI:SS'
...