How to Mount a Samba (CIFS) Share with a Username and Password in Linux

Many Samba shares on Linux, and shares on Windows Vista and Windows 7, require a username and password before they’ll let you in. Those credentials are set on the server:

  • Windows Vista/7: the login username and password of the share owner on that machine.
  • Linux Samba: any member of the server’s Samba user database who is allowed by the share’s definition.

Mount the Share with Credentials

Instead of the guest option, pass username= and password=:

mount -t cifs -o username=server_user,password=secret //192.168.44.100/share /path_to/mount

To mount it at boot, or to fix write permissions for non-root users, see fixing CIFS mount permissions with file_mode and dir_mode.

How to Forward CentOS Syslog to Splunk with rsyslog (TCP + Disk Queue)

To send a CentOS server’s logs to Splunk reliably, configure rsyslog to forward over TCP with an on-disk queue. If the Splunk server goes down, messages are spooled to disk and delivered once it’s back up, so nothing is lost.

rsyslog Forwarding Rule

Add this block to /etc/rsyslog.conf (or a file in /etc/rsyslog.d/). Replace splunk:514 with your Splunk host and listening port:

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@splunk:514
# ### end of the forwarding rule ###

The double @@ means TCP; a single @ would use UDP. Restart rsyslog to apply the change, and make sure Splunk has a matching TCP data input on that port.

Still on the old syslogd? See replacing syslogd with rsyslog and forwarding to Splunk.

How to Extend an LVM Logical Volume and Filesystem on CentOS

Logical Volume Management (LVM) makes Linux storage flexible. By default, CentOS installs the root filesystem on an LVM logical volume instead of a fixed-size partition, so you can grow it later, even across multiple physical disks. This guide explains how the LVM layers fit together and walks through extending a logical volume and its filesystem.

How LVM Works

LVM is built in layers, each on top of the last:

Physical Volumes (PV) → Volume Groups (VG) → Logical Volumes (LV) → Filesystems

A logical volume can only be allocated or extended within the free space of its volume group, which acts as the storage pool. For example, the root filesystem here lives on /dev/vg_prd00/lv_root inside the vg_prd00 volume group (lvdisplay output):

--- Logical volume ---
  LV Name                /dev/vg_prd00/lv_root
  VG Name                vg_prd00
  LV UUID                ----uuid----
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                50 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Check Free Space in the Volume Group

Out of the box, the volume group may not have enough free space:

# vgdisplay
  --- Volume group ---
  VG Name               vg_prd00
  Format                lvm2
  VG Access             read/write
  VG Status             resizable
  Cur LV                3
  Open LV               3
  Cur PV                1
  Act PV                1
  VG Size               63.51 GiB
  PE Size               4.00 MiB
  Total PE              16258
  Alloc PE / Size       16258 / 63.51 GiB
  Free  PE / Size       0 / 0
  VG UUID               ----uuid----

The Free PE / Size line tells you how much you can allocate. If it reads, say, 165 / 660.00 MiB, you only have 660 MB to give to logical volumes. To grow the root LV beyond that, you first have to extend the volume group.

Volume groups are made of physical volumes. This shows which physical volumes LVM knows about and which group each belongs to:

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               vg_prd00
  PV Size               18.15 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              4645
  Free PE               165
  Allocated PE          4480
  PV UUID               ----uuid----

Step 1: Add a Physical Volume to the Volume Group

If the volume group is short on space, add another physical volume. In VMware you can either add a new virtual disk or expand the existing one and create a new partition:

# Example 1: you added a new virtual hard disk, /dev/sdb
pvcreate /dev/sdb
vgextend vg_prd00 /dev/sdb

# Example 2: you expanded the existing sda disk
cfdisk /dev/sda   # create /dev/sda3 (reboot before it shows up)
pvcreate /dev/sda3
vgextend vg_prd00 /dev/sda3

Step 2: Extend the Logical Volume

# lvextend -L+10G /dev/vg_prd00/lv_root
Extending logical volume lv_root to XXXX GiB
Logical volume lv_root successfully resized

Step 3: Grow the Filesystem

Finally, resize the filesystem so it can use the extra 10 GB. For ext3/ext4:

# resize2fs /dev/vg_prd00/lv_root
resize2fs 1.41.11 (14-Mar-2010)
Filesystem at /dev/vg_prd00/lv_root is mounted on /; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 2
Performing an on-line resize of /dev/vg_prd00/lv_root to XXXX (4k) blocks.
The filesystem on /dev/vg_prd00/lv_root is now XXXX blocks long.

For XFS (the default on CentOS 7 and later):

xfs_growfs /dev/vg_name/lv_name

Also see adding a disk to CentOS LVM.

How to Install VMware Tools on CentOS from the Command Line

Here are the steps I use to install VMware Tools on a CentOS server VM with no GUI, using the vSphere Client and the command line.

  1. In the vSphere Client, choose Guest → Install/Upgrade VMware Tools. This attaches the VMware Tools installer as a virtual CD-ROM.
  2. On the VM console, mount the virtual CD-ROM, extract the installer, and run it (press Tab to auto-complete the file names):
mkdir /cdrom
mount /dev/cdrom /cdrom
cd /tmp
tar -xvf /cdrom/VMwareTools*.tar.gz
cd vmware-tools-distrib
./vmware-install.pl
umount /cdrom
  1. Accept the defaults in the installer. You may need to install Perl first (yum install perl).
  2. Back in the vSphere Client, choose Guest → Install/Upgrade VMware Tools → OK to finish and disconnect the installer.
  3. Confirm the install on the VM’s Summary tab: VMware Tools should show OK.

If VMware Tools stops starting after kernel updates, see fixing VMware Tools not starting after reboot on CentOS.

Install and Tune the Dell EqualLogic MEM (Multipathing Extension Module) on ESXi

The Dell EqualLogic Multipathing Extension Module (MEM) gives VMware ESXi hosts smarter iSCSI multipathing to EqualLogic SANs. Here’s how I installed it and raised the session limits to get more iSCSI paths per volume.

1. Install the MEM VIB

Copy the zipped VIB (for example, dell-eql-mem-esx5-1.2.0.365964.zip) to a datastore your host can reach using SCP. Then SSH to the host and install it, adjusting the path to match your datastore:

esxcli software vib install --depot /vmfs/volumes/*****/dell-eql-mem-esx5-1.2.0.365964.zip

You’ll see a result similar to this (from an older version):

Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: Dell_bootbank_dell-eql-host-connection-mgr_1.1.1-268843, Dell_bootbank_dell-eql-hostprofile_1.1.0-212190, Dell_bootbank_dell-eql-routed-psp_1.1.1-262227

2. Check the Current MEM Parameters

esxcli equallogic param list

Name            Value  Max    Min  Description
--------------- ------ ------ ---- ---------------------------------------------
TotalSessions   512    1024   64   Max number of sessions per host.
VolumeSessions  12     12     1    Max number of sessions per volume.
MemberSessions  4      4      1    Max number of sessions per member per volume.
MinAdapterSpeed 1000   10000  10   Minimum adapter speed for iSCSI multipathing.

3. Raise Volume and Member Sessions

By default, member sessions are probably set to 2 and volume sessions to 4 or 6. Set them to the maximums:

esxcli equallogic param set -n=VolumeSessions -v=12
esxcli equallogic param set -n=MemberSessions -v=4

Then rescan the software iSCSI adapter. You should see about three times as many paths to each volume.

Locked Out of Sophos UTM 9.2 WebAdmin by OTP? Disable It from the Shell

While testing Sophos UTM 9.2’s one-time password (OTP) feature with Google Authenticator, I locked myself out of WebAdmin. Here’s how to turn OTP off for WebAdmin from the console using the cc configuration tool.

Steps to Disable WebAdmin OTP

1. Log in to the firewall console as root and start the configuration tool:

cc

2. Type auth and press Enter. It lists:

ad_sso
auto_add_to_facility@
auto_add_users$
block
cache_lifetime$
edir_sso
otp
servers@

3. Type otp and press Enter. It lists:

auto_create_token$
default_timestep$
facilities@
require_all_users$
required_users@
status$

4. Type facilities@ and press Enter. It shows which services require OTP:

0 webadmin
1 portal
2 ipsec
3 openvpn

5. Type -0 and press Enter to remove webadmin from the list. It now shows:

0 portal
1 ipsec
3 openvpn

6. Log in to WebAdmin as admin with just your password. Once you’re back in, you can reconfigure OTP properly.

“BUG: soft lockup detected on CPU” in a Linux Virtual Machine: Cause and Fix

The Symptom

When a Linux kernel runs in a symmetric multiprocessing (SMP) virtual machine, you may see messages like BUG: soft lockup detected on CPU#1! in the system log. The exact wording varies by kernel, and the message may include a kernel stack backtrace.

What Causes It

Many Linux kernels run a soft lockup watchdog thread and log this message if that thread isn’t scheduled for more than 10 seconds. On a physical server, a soft lockup usually points to a kernel or hardware bug. In a virtual machine, it more often means the host is heavily overcommitted (especially on memory) or there’s other virtualization overhead.

The Fix

Soft lockup messages are not kernel panics and can generally be ignored. Some kernels let you raise the threshold, where time is the number of seconds before a soft lockup is reported (default is usually 10):

echo time > /proc/sys/kernel/softlockup_thresh

If they’re frequent, look at memory and CPU overcommitment on the host.

What the Error Report Looked Like

Here’s the start of the ABRT report from an Oracle Linux (UEK) VM where I saw this (memory maps and limits trimmed):

abrt_version:    2.0.8
cmdline:         /usr/bin/ksh ./update_archive.ksh RMUAT
executable:      /bin/ksh93
kernel:          2.6.39-400.212.1.el6uek.x86_64
last_occurrence: 1403155682
pid:             25607
pwd:             /oracle/scripts/update_archive
time:            Thu 19 Jun 2014 12:28:02 AM CDT
uid:             502
username:        oracle
sosreport.tar.xz: Binary file, 3959288 bytes

environ:
:SHELL=/bin/sh
:USER=oracle
:LD_LIBRARY_PATH=/lib:/usr/lib:/oracle/product/11.2.0/dbhome_1/lib
:PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin
:PWD=/oracle/scripts/update_archive
:JAVA_HOME=/usr/java
:LANG=en_US.UTF-8
:HOME=/home/oracle
:'NLS_DATE_FORMAT=MM-DD-YYYY HH24:MI:SS'
...

Fix Non-Root Write Permissions on a CIFS Mount in Oracle Linux

I was working on a system recently migrated from a very old Solaris server to Oracle Linux 6.6. It mounts a CIFS (SMB) share from a Windows Server 2012 R2 host, just like the old system did, but non-root users could no longer write to it or create files.

The Symptom

The empty mount point has 777 permissions:

[root@localhost ~]# ls -ld /datastore/
drwxrwxrwx 2 root root 4096 Jan 6 09:50 /datastore/

But once the share is mounted, they drop to 755:

[root@localhost ~]# ls -ld /datastore/
drwxr-xr-x 1 root root 634564 Jan 6 09:50 /datastore/

On the old server, files and subdirectories on the share had the setuid bit, which the new system doesn’t. The original /etc/fstab entry was:

//cifshost/datastore /datastore cifs username=user,password=password,domain=mydomain.local 0 0

The Fix: Add file_mode and dir_mode

Edit /etc/fstab and add the file_mode=0666,dir_mode=0777 mount options:

//cifshost/share/datastore /datastore cifs user=user,pass=password,file_mode=0666,dir_mode=0777 0 0

Remount the share, and non-root users can read and write again.

How to Update EqualLogic Disk Drive Firmware (No Downtime)

Updating controller firmware on a Dell EqualLogic PS Series array is easy: open Group Manager, upload a file, click a few buttons, and you’re done. But what about when SAN HQ starts emailing you about outdated disk drive firmware? That takes a different process, and the good news is that it can be done hot, with no outage.

Firmware Downloads

Controller firmware kits:

Disk drive firmware kits:

SAN HQ: SANHQSetup32And64_v3.2.1

Steps to Update Disk Firmware

  1. Open your FTP client (I use WinSCP) and connect to the array’s group IP or management IP.
  2. Upload the drive firmware .tgz package. I used kit_V8.0_DriveFw_2285700222.tgz. Don’t unzip it; it needs no further prep.
  3. SSH (PuTTY) to the same group or management IP and run one command: update. That’s it.

Example Output

Here’s what the update looked like on my array (repetitive drive lines trimmed):

Welcome to Group Manager
Copyright 2001-2014 Dell Inc.

EQLSAN> update
13:24:23 Updating from kit file "kit_V8.0_DriveFw_2285700222.tgz"

This command will install the update kit file that was
copied to the array.

If you choose to proceed, you will be shown the current firmware version
and the version to which you will update. You will then be given the
choice to proceed again.

Do you want to proceed (y/n) [y]: y

13:24:39 Verifying kit integrity.
Starting Disk Firmware update... V8.0
...Initializing support libraries...

Identifying drives that qualify for firmware upgrades... Please wait.
22 drive(s) in this array qualify for a firmware upgrade.
The process will now update the 22 drives that qualify for a firmware upgrade.

If you proceed, please do not power off or restart the array, or remove
any drives until the update process completes.
Do you want to continue at this time (Y/N)? y
SKIPPING DriveID 0 No firmware is available for Model: HUS723030ALS640
SKIPPING DriveID 1 No firmware is available for Model: HUS723030ALS640
SKIPPING DriveID 2 No firmware is available for Model: HUS723030ALS640
Checking health status of Array
UPGRADING DriveID 3 firmware from RN08 to RE12 using ./fwbin/RE12.lod
Drive firmware successfully updated : on drive 3
...
Checking health status of Array
UPGRADING DriveID 46 firmware from RN08 to RE12 using ./fwbin/RE12.lod
Drive firmware successfully updated : on drive 46
SKIPPING DriveID 47 No firmware is available for Model: HUS723030ALS640

Logging post run disk information
Entering cleanup phase... Please Wait...
The update was successful.
Would you like to email the results to Dell (Y/N)? n
Done cleanup ... quitting with an exit status of 0

EQLSAN>

The array checks its health before each drive, updates only the drives that qualify, and stays online the whole time.

Fix VMware Tools Not Starting After Reboot on CentOS 6/7

The Problem

On vSphere 5.x with VMware Tools installed on a CentOS 6.x x64 guest, VMware Tools doesn’t start automatically after an OS update that requires a reboot.

Starting with version 8.6.11.20852 (build-1015158), the installer script vmware-install.pl no longer creates a start script in /etc/init.d because it now uses Upstart. On top of that, the option to automatically build and install kernel modules at boot was not enabled during installation, so a new kernel leaves the tools without matching modules.

Three Ways to Fix It

Option 1: Re-run /usr/bin/vmware-config-tools.pl after every OS update that changes the kernel.

Option 2: Run /usr/bin/vmware-config-tools.pl and enable automatic building and installation of kernel modules at boot. You’ll need to remember to enable this again on future VMware Tools upgrades.

Option 3: Create your own init script from the services.sh script in /etc/vmware-tools:

cp /etc/vmware-tools/services.sh /etc/init.d/vmware-tools
vim /etc/init.d/vmware-tools

Paste this line just below ##VMWARE_INIT_INFO## and save:

# chkconfig: 235 03 99

Register the service and enable it at boot:

chkconfig --add vmware-tools
chkconfig vmware-tools on

Verify that it works:

service vmware-tools restart