Syslog over a SSH Reverse Tunnel

Create syslog users

Create a user on both the syslog client and the loghost. This user will be used for the ssh tunnel between the two systems. We'll create the user rsyslog-remote

rsyslog-remote user's /etc/passwd entry:

rsyslog-remote:x:50514:50514::/home/rsyslog-remote:/bin/rbash

rsyslog-remote user's /etc/group entry:

rsyslog-remote:x:50514:

Make sure that the rsyslog-remote user's password is very hard to guess with tons of entropy....or better yet lock the account as we'll be using ssh keys for authentication.

Configuring ssh public key authentication

This assumes that both systems sshd is configured to allow authtication with public keys.

On both the syslog client and the loghost, create a public and private keypair for the rsyslog-remote user.

cd ~rsyslog-remote
sudo -u rsyslog-remote ssh-keygen

Copy the rsyslog-remote user's public key from the .ssh directory on the loghost to the syslog client

root@loghost:~# scp /home/rsyslog-remote/.ssh/id_rsa.pub user@syslog-client.blackhole-networks.com:

On the syslog client, copy the rsyslog-remote user's loghost public key to the authorized_keys file for the rsyslog-remote user

cat /home/user/id_rsa.pub >> /home/rsyslog-remote/.ssh/authorized_keys
chown rsyslog-remote:rsyslog-remote /home/rsyslog-remote/.ssh/authorized_keys
chmod 600 /home/rsyslog-remote/.ssh/authorized_keys

Configuring ssh Rreverse Tunnel

We'll originate a reverse ssh tunnel on the loghost that listens on port 50514 on the loopback interface (127.0.0.1 and ::1 ) on the client and empties out on our loghost on port 1514. So, any packet that is sent to 127.0.0.1:50514 on the syslog client, will be encrypted by the reverse ssh session and be available to be read on port 1514 on the loghost.

On the loghost, create the reverse tunnel as the rsyslog-remote user

sudo -u rsyslog-remote ssh -nN -R 50514:loghost.blackhole-networks.com:1514 syslog-client.blackhole-networks.com

If the ssh public key authentication was setup properly, it should prompt you to accept the syslog clients key, and then just sit there quietly.

Configuring rsyslog

On the syslog client, configure rsyslogd to send logs to a remote host over a tcp port.

This will be the port that correseponds with the reverse tunnel that was opened up on the syslog client
Append the following line to /etc/rsyslog.d/50-default.conf or place it in a new file like /etc/rsyslog.d/60-remote-ssh-tunnel.conf
This line sends all syslog messages across the tunnel -- *.*
The @@> specifies to use TCP transport vice UDP

*.* @@127.0.0.1:50514

Restart rsyslog on the syslog client

service rsyslog restart

On the loghost, configure rsyslog to open up a TCP listener on the loghost side of our ssh reverse tunnel

Add the following lines to the /etc/rsyslog.d/50-default.conf file

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPMaxSessions 500
$InputTCPServerRun 1514

Restart rsyslog on the loghost

service rsyslog restart

Testing

On the loghost, start a tail on the syslog file (or other file if you've configured logging to go elsewhere).

user@loghost-1:~$ tail -f /var/log/syslog

On the syslog client, send a test message to the syslog facility with the logger command

The default facility and level for the logger command is user notice. You can change this with the -p switch.

user@syslog-client:~$ logger "This is a Stupid Test\!"

In a second or two, the message should appear in the syslog file being tailed on the loghost

user@loghost:~$ tail -f /var/log/syslog
Jan  8 23:21:07 syslog-client user: This is a Stupid Test\!

Automatic Tunnels with Upstart

Tested with Ubuntu 12.04

On the loghost, setup an upstart config file in the /etc/init directory called /etc/init/rsyslog-rssh-syslog-client.conf

This will contain our reverse ssh command, and be setup to be respawned automatically if the process should ever terminate

# ttyS0 - getty
#
# This service maintains a remote ssh reverse tunnel for syslog-ing 

start on stopped rc or RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec  sudo -u rsyslog-remote ssh -nN -R 50514:loghost.blackhole-networks.com:1514 syslog-client.blackhole-networks.com

Start the upstart job. Make sure the old reverse ssh session has terminated beforehand

root@loghost:~# start rsyslog-rssh-syslog-client
rsyslog-rssh-syslog-client start/running, process 9816

Now if the ssh tunnels are killed, or the system is rebooted the tunnels should start back up automatically.

Notes

This can also be accomplished by using the autossh command instead of an upstart job.

Cheat Sheet: Syslog over a SSH Reverse Tunnel

Topics