Soft lockup messages from Linux kernel running in an SMP-enabled virtual machine

Details:

When running a Linux kernel in a symmetric multiprocessing (SMP) enabled virtual machine, messages similar to BUG: soft lockup detected on CPU#1! are written to the message log file. The exact format of these messages vary from kernel to kernel, and might be accompanied by a kernel stack backtrace.

Many Linux kernels have a soft lockup watchdog thread, and report soft lockup messages if that watchdog thread does not get scheduled for more than 10 seconds. On a physical host, a soft lockup message generally indicates a kernel bug or hardware bug. When running in a virtual machine, this might instead indicate high levels of overcommitment (especially memory overcommitment) or other virtualization overheads.

Solution:

The soft lockup messages are not kernel panics, and can be safely ignored.Some kernels allow you to adjust the soft lockup threshold by running the command:echo time > /proc/sys/kernel/softlockup_thresh

Where time is the number of seconds after which a soft lockup is reported. The default is generally 10 seconds.

What the Error Looks Like:

abrt_version: 2.0.8

cgroup:

cmdline: /usr/bin/ksh ./update_archive.ksh RMUAT

executable: /bin/ksh93

kernel: 2.6.39-400.212.1.el6uek.x86_64

last_occurrence: 1403155682

pid: 25607

pwd: /oracle/scripts/update_archive

time: Thu 19 Jun 2014 12:28:02 AM CDT

uid: 502

username: oracle

sosreport.tar.xz: Binary file, 3959288 bytes

environ:

:SHELL=/bin/sh

:OLDPWD=/home/oracle

:USER=oracle

:LD_LIBRARY_PATH=/lib:/usr/lib:/oracle/product/11.2.0/dbhome_1/lib

:PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin

:PWD=/oracle/scripts/update_archive

:JAVA_HOME=/usr/java

:LANG=en_US.UTF-8

:HOME=/home/oracle

:SHLVL=2

:LOGNAME=oracle

:’NLS_DATE_FORMAT=MM-DD-YYYY HH24:MI:SS’

:_=./update_archive.ksh

limits:

:Limit Soft Limit Hard Limit Units

:Max cpu time unlimited unlimited seconds

:Max file size unlimited unlimited bytes

:Max data size unlimited unlimited bytes

:Max stack size 10485760 33554432 bytes

:Max core file size 0 unlimited bytes

:Max resident set unlimited unlimited bytes

:Max processes 16384 16384 processes

:Max open files 1024 65536 files

:Max locked memory 65536 65536 bytes

:Max address space unlimited unlimited bytes

:Max file locks unlimited unlimited locks

:Max pending signals 387052 387052 signals

:Max msgqueue size 819200 819200 bytes

:Max nice priority 0 0

:Max realtime priority 0 0

:Max realtime timeout unlimited unlimited us

maps:

:00400000-0055b000 r-xp 00000000 fc:00 261718 /bin/ksh93

:0075a000-0076d000 rw-p 0015a000 fc:00 261718 /bin/ksh93

:0076d000-00773000 rw-p 00000000 00:00 0

:0096c000-0096e000 rw-p 0016c000 fc:00 261718 /bin/ksh93

:3e8dc00000-3e8dc20000 r-xp 00000000 fc:00 1177356 /lib64/ld-2.12.so

:3e8de1f000-3e8de20000 r–p 0001f000 fc:00 1177356 /lib64/ld-2.12.so

:3e8de20000-3e8de21000 rw-p 00020000 fc:00 1177356 /lib64/ld-2.12.so

:3e8de21000-3e8de22000 rw-p 00000000 00:00 0

:3e8e000000-3e8e002000 r-xp 00000000 fc:00 1177390 /lib64/libdl-2.12.so

:3e8e002000-3e8e202000 —p 00002000 fc:00 1177390 /lib64/libdl-2.12.so

:3e8e202000-3e8e203000 r–p 00002000 fc:00 1177390 /lib64/libdl-2.12.so

:3e8e203000-3e8e204000 rw-p 00003000 fc:00 1177390 /lib64/libdl-2.12.so

:3e8e400000-3e8e58b000 r-xp 00000000 fc:00 1177360 /lib64/libc-2.12.so

:3e8e58b000-3e8e78a000 —p 0018b000 fc:00 1177360 /lib64/libc-2.12.so

:3e8e78a000-3e8e78e000 r–p 0018a000 fc:00 1177360 /lib64/libc-2.12.so

:3e8e78e000-3e8e78f000 rw-p 0018e000 fc:00 1177360 /lib64/libc-2.12.so

:3e8e78f000-3e8e794000 rw-p 00000000 00:00 0

:3e8f000000-3e8f083000 r-xp 00000000 fc:00 1177398 /lib64/libm-2.12.so

:3e8f083000-3e8f282000 —p 00083000 fc:00 1177398 /lib64/libm-2.12.so

:3e8f282000-3e8f283000 r–p 00082000 fc:00 1177398 /lib64/libm-2.12.so

:3e8f283000-3e8f284000 rw-p 00083000 fc:00 1177398 /lib64/libm-2.12.so

:3e98400000-3e98402000 r-xp 00000000 fc:00 1177423 /lib64/libutil-2.12.so

:3e98402000-3e98601000 —p 00002000 fc:00 1177423 /lib64/libutil-2.12.so

:3e98601000-3e98602000 r–p 00001000 fc:00 1177423 /lib64/libutil-2.12.so

:3e98602000-3e98603000 rw-p 00002000 fc:00 1177423 /lib64/libutil-2.12.so

:7fea21b5d000-7fea21bcd000 rw-p 00000000 00:00 0

:7fea21bcd000-7fea27a5e000 r–p 00000000 fc:00 1441433 /usr/lib/locale/locale-archive

:7fea27a5e000-7fea27aaa000 rw-p 00000000 00:00 0

:7fea27ab8000-7fea27ab9000 rw-p 00000000 00:00 0

:7fffdfc56000-7fffdfc77000 rw-p 00000000 00:00 0 [stack]

:7fffdfde7000-7fffdfde8000 r-xp 00000000 00:00 0 [vdso]

:ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

open_fds:

:0:pipe:[122311558]

:pos: 0

:flags: 00

:1:/oracle/log/RMUAT_update_archive_140619002801.log

:pos: 199

:flags: 0100001

:2:/oracle/log/RMUAT_update_archive_140619002801.log