How to Forward CentOS Syslog to Splunk with rsyslog (TCP + Disk Queue)

To send a CentOS server’s logs to Splunk reliably, configure rsyslog to forward over TCP with an on-disk queue. If the Splunk server goes down, messages are spooled to disk and delivered once it’s back up, so nothing is lost.

rsyslog Forwarding Rule

Add this block to /etc/rsyslog.conf (or a file in /etc/rsyslog.d/). Replace splunk:514 with your Splunk host and listening port:

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@splunk:514
# ### end of the forwarding rule ###

The double @@ means TCP; a single @ would use UDP. Restart rsyslog to apply the change, and make sure Splunk has a matching TCP data input on that port.

Still on the old syslogd? See replacing syslogd with rsyslog and forwarding to Splunk.

WSUS Low Disk Space? Clean Up and Move the Content Folder and SUSDB Database

WSUS (Windows Server Update Services) can quickly fill a drive. Here are three ways to fix low disk space: run the Server Cleanup Wizard, move the content directory, and move the SUSDB database to another drive.

1. Run the WSUS Server Cleanup Wizard

  1. In the WSUS administration console, select Options → Server Cleanup Wizard.
  2. By default it removes unneeded content and computers that haven’t contacted the server in 30 days or more. Select all options and click Next.
  3. When it finishes, review the summary and click Finish.

2. Move the WSUS Content Directory

  1. Create a new content folder on another drive.
  2. Open a Command Prompt and go to the WSUS Tools folder, typically C:\Program Files\Update Services\Tools.
  3. Run the command below, where NewContentPath is the new folder and MoveLog is the path and file name for the log.
wsusutil movecontent NewContentPath MoveLog

For more on WSUSUtil, see “Managing WSUS 3.0 from the Command Line” in the WSUS Operations Guide.

3. Move the SUSDB Database

  1. Open a Command Prompt and stop the WSUS service: net stop wsusservice
  2. Detach the SUSDB database.
  3. Copy SUSDB.mdf and SUSDB_log.ldf to the new location.
  4. Attach SUSDB from the new location.
  5. Start the service again: net start wsusservice
  6. Once everything is working, delete the old SUSDB.mdf and SUSDB_log.ldf.

How to Extend an LVM Logical Volume and Filesystem on CentOS

Logical Volume Management (LVM) makes Linux storage flexible. By default, CentOS installs the root filesystem on an LVM logical volume instead of a fixed-size partition, so you can grow it later, even across multiple physical disks. This guide explains how the LVM layers fit together and walks through extending a logical volume and its filesystem.

How LVM Works

LVM is built in layers, each on top of the last:

Physical Volumes (PV) → Volume Groups (VG) → Logical Volumes (LV) → Filesystems

A logical volume can only be allocated or extended within the free space of its volume group, which acts as the storage pool. For example, the root filesystem here lives on /dev/vg_prd00/lv_root inside the vg_prd00 volume group (lvdisplay output):

--- Logical volume ---
  LV Name                /dev/vg_prd00/lv_root
  VG Name                vg_prd00
  LV UUID                ----uuid----
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                50 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Check Free Space in the Volume Group

Out of the box, the volume group may not have enough free space:

# vgdisplay
  --- Volume group ---
  VG Name               vg_prd00
  Format                lvm2
  VG Access             read/write
  VG Status             resizable
  Cur LV                3
  Open LV               3
  Cur PV                1
  Act PV                1
  VG Size               63.51 GiB
  PE Size               4.00 MiB
  Total PE              16258
  Alloc PE / Size       16258 / 63.51 GiB
  Free  PE / Size       0 / 0
  VG UUID               ----uuid----

The Free PE / Size line tells you how much you can allocate. If it reads, say, 165 / 660.00 MiB, you only have 660 MB to give to logical volumes. To grow the root LV beyond that, you first have to extend the volume group.

Volume groups are made of physical volumes. This shows which physical volumes LVM knows about and which group each belongs to:

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               vg_prd00
  PV Size               18.15 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              4645
  Free PE               165
  Allocated PE          4480
  PV UUID               ----uuid----

Step 1: Add a Physical Volume to the Volume Group

If the volume group is short on space, add another physical volume. In VMware you can either add a new virtual disk or expand the existing one and create a new partition:

# Example 1: you added a new virtual hard disk, /dev/sdb
pvcreate /dev/sdb
vgextend vg_prd00 /dev/sdb

# Example 2: you expanded the existing sda disk
cfdisk /dev/sda   # create /dev/sda3 (reboot before it shows up)
pvcreate /dev/sda3
vgextend vg_prd00 /dev/sda3

Step 2: Extend the Logical Volume

# lvextend -L+10G /dev/vg_prd00/lv_root
Extending logical volume lv_root to XXXX GiB
Logical volume lv_root successfully resized

Step 3: Grow the Filesystem

Finally, resize the filesystem so it can use the extra 10 GB. For ext3/ext4:

# resize2fs /dev/vg_prd00/lv_root
resize2fs 1.41.11 (14-Mar-2010)
Filesystem at /dev/vg_prd00/lv_root is mounted on /; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 2
Performing an on-line resize of /dev/vg_prd00/lv_root to XXXX (4k) blocks.
The filesystem on /dev/vg_prd00/lv_root is now XXXX blocks long.

For XFS (the default on CentOS 7 and later):

xfs_growfs /dev/vg_name/lv_name

Also see adding a disk to CentOS LVM.

How to Install VMware Tools on CentOS from the Command Line

Here are the steps I use to install VMware Tools on a CentOS server VM with no GUI, using the vSphere Client and the command line.

  1. In the vSphere Client, choose Guest → Install/Upgrade VMware Tools. This attaches the VMware Tools installer as a virtual CD-ROM.
  2. On the VM console, mount the virtual CD-ROM, extract the installer, and run it (press Tab to auto-complete the file names):
mkdir /cdrom
mount /dev/cdrom /cdrom
cd /tmp
tar -xvf /cdrom/VMwareTools*.tar.gz
cd vmware-tools-distrib
./vmware-install.pl
umount /cdrom
  1. Accept the defaults in the installer. You may need to install Perl first (yum install perl).
  2. Back in the vSphere Client, choose Guest → Install/Upgrade VMware Tools → OK to finish and disconnect the installer.
  3. Confirm the install on the VM’s Summary tab: VMware Tools should show OK.

If VMware Tools stops starting after kernel updates, see fixing VMware Tools not starting after reboot on CentOS.

How to Configure Port Mirroring (SPAN) on a Dell Force10 S50 Switch

Port mirroring copies the traffic from one switch port to another so you can capture it with a network analyzer like Wireshark. Here I mirrored all traffic sent and received on port 0/48 of a Dell Force10 S50V switch to port 0/44 on the same switch.

Prepare the Destination Port

The destination port (0/44) doesn’t need to belong to any VLAN. It only needs no ip address and no shutdown configured.

Create the Monitor Session

enable
config
monitor session 0
source gigabitethernet 0/48 destination gigabitethernet 0/44 direction both

Mirroring a Port-Channel (LAG)

Once a link aggregation group is set up, you can mirror the whole port-channel instead by changing monitor session 0 to:

source port-channel 128 destination gigabitethernet 0/44 direction both

Install and Tune the Dell EqualLogic MEM (Multipathing Extension Module) on ESXi

The Dell EqualLogic Multipathing Extension Module (MEM) gives VMware ESXi hosts smarter iSCSI multipathing to EqualLogic SANs. Here’s how I installed it and raised the session limits to get more iSCSI paths per volume.

1. Install the MEM VIB

Copy the zipped VIB (for example, dell-eql-mem-esx5-1.2.0.365964.zip) to a datastore your host can reach using SCP. Then SSH to the host and install it, adjusting the path to match your datastore:

esxcli software vib install --depot /vmfs/volumes/*****/dell-eql-mem-esx5-1.2.0.365964.zip

You’ll see a result similar to this (from an older version):

Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: Dell_bootbank_dell-eql-host-connection-mgr_1.1.1-268843, Dell_bootbank_dell-eql-hostprofile_1.1.0-212190, Dell_bootbank_dell-eql-routed-psp_1.1.1-262227

2. Check the Current MEM Parameters

esxcli equallogic param list

Name            Value  Max    Min  Description
--------------- ------ ------ ---- ---------------------------------------------
TotalSessions   512    1024   64   Max number of sessions per host.
VolumeSessions  12     12     1    Max number of sessions per volume.
MemberSessions  4      4      1    Max number of sessions per member per volume.
MinAdapterSpeed 1000   10000  10   Minimum adapter speed for iSCSI multipathing.

3. Raise Volume and Member Sessions

By default, member sessions are probably set to 2 and volume sessions to 4 or 6. Set them to the maximums:

esxcli equallogic param set -n=VolumeSessions -v=12
esxcli equallogic param set -n=MemberSessions -v=4

Then rescan the software iSCSI adapter. You should see about three times as many paths to each volume.

How to Shadow a VMware View (Horizon) PCoIP Desktop Session from vSphere

Unlike Citrix XenApp, VMware View has no central way to shadow a user’s desktop session, for example from the View Connection Server. It’s one of the most common questions from customers running View. You can, however, view a PCoIP session from the vSphere console by enabling one Group Policy setting.

Requirements

  • VMware vSphere 5 and VMware View 5 or later (vSphere 4 needs a registry hack instead).
  • Windows 7 desktops must use virtual hardware version 8. Windows XP and Vista can use any hardware version.

Steps to Enable PCoIP Session Shadowing

  1. Create a new Group Policy Object (GPO).
  2. Add the pcoip.adm template to Computer Configuration. You’ll find it on the View Connection Server in C:\Program Files\VMware\VMware View\Server\extras\GroupPolicyFiles.
  3. Enable Enable access to PCoIP session from a vSphere console under Computer Configuration.
  4. Link the GPO to the OU where the View desktops live.
  5. Let the domain controllers sync, then restart the View desktop.

You can now watch the PCoIP session from the vSphere console. Keyboard and mouse activity appears on both screens at the same time, with a slight delay.

Security note: the user isn’t asked to approve the shadow session, so consider the privacy implications before enabling this.

Locked Out of WordPress by Google Authenticator? How to Get Back In

I recently locked myself out of my WordPress site while experimenting with the Google Authenticator two-factor plugin. The fix is to disable the plugin over FTP, log in, and then turn it back on.

Steps to Disable the Plugin and Log In

  1. Connect to your site with an FTP or SFTP client and open /wp-content/plugins/google-authenticator.
  2. Rename the folder from google-authenticator to _google-authenticator. WordPress can no longer find it, so the plugin is disabled.
  3. Log in to WordPress as normal and open the Plugins page.
  4. Back in FTP, rename _google-authenticator back to google-authenticator.
  5. Refresh the Plugins page. The plugin now shows as deactivated. Reactivate it and reconfigure it in your user profile.

Avoid Locking Yourself Out Next Time

Before you enable two-factor authentication, create a second admin account without it. Test everything with your main account, and once it all works, delete the backup admin account. There’s a helpful video walkthrough of this approach here (video 3 on the page): Installation Essentials for WordPress.

Locked Out of Sophos UTM 9.2 WebAdmin by OTP? Disable It from the Shell

While testing Sophos UTM 9.2’s one-time password (OTP) feature with Google Authenticator, I locked myself out of WebAdmin. Here’s how to turn OTP off for WebAdmin from the console using the cc configuration tool.

Steps to Disable WebAdmin OTP

1. Log in to the firewall console as root and start the configuration tool:

cc

2. Type auth and press Enter. It lists:

ad_sso
auto_add_to_facility@
auto_add_users$
block
cache_lifetime$
edir_sso
otp
servers@

3. Type otp and press Enter. It lists:

auto_create_token$
default_timestep$
facilities@
require_all_users$
required_users@
status$

4. Type facilities@ and press Enter. It shows which services require OTP:

0 webadmin
1 portal
2 ipsec
3 openvpn

5. Type -0 and press Enter to remove webadmin from the list. It now shows:

0 portal
1 ipsec
3 openvpn

6. Log in to WebAdmin as admin with just your password. Once you’re back in, you can reconfigure OTP properly.

How to Change the Remote Desktop (RDP) Listening Port in Windows

Remote Desktop listens on TCP port 3389 by default. You can move it to a different port with a quick registry change, which is handy for running several hosts behind one public IP or cutting down on automated scans.

Steps to Change the RDP Port

  1. Open Registry Editor (regedit).
  2. Go to this key and select the PortNumber value:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp\PortNumber
  1. On the Edit menu, click Modify, then select Decimal.
  2. Type the new port number and click OK.
  3. Close Registry Editor and restart the computer.

Before You Connect

Allow the new port through Windows Firewall (and any network firewall) first, or you’ll lock yourself out. When connecting with Remote Desktop Connection, add the port to the address, for example server.example.com:3390.