Debugging apache2 shared module:

During my time on a fantastic site: hackthebox a machine ctf by Ippsec was made available which required debugging a known rootkit that is loaded as a module into apache2 : this allows a get command to load a root shell on the machine but creates no logging in the access logs.

Merely viewing strings on the binary was not enough as the source code which can be seen here: had been modified to add an extra function called ‘darkarmy’, thus using the key “get root” or “get HackTheBox” which is a value seen via the strings command plus xxd both return a 404 error and fails to execute the shell, clearly more debugging was needed which lead to the creation of this post.

It was time to copy the binary to my local Virtual machine and disassemble the binary to find out more about the function ‘darkarmy’ that was seen via strings.

Copy over and enable mod_rootme

The simplest way to copy over the binary was to use old and faithful base64:

[email protected]:/$ base64 -w 0 /usr/lib/apache2/modules/

We can then copy the b64 over and cat this out to a file via:

cat modrootme.b64 |base64 -d >

The next few commands are to copy the module into the apache modules directory and set the module into apache as an available and loaded module which is explained in the github readme from the link above.

[email protected]:/$cp
[email protected]:/$sh -c \
'echo "LoadModule rootme_module \
/usr/lib/apache2/modules/" | sudo tee > \

[email protected]:/$a2enmod rootme

Now it’s important at this point to leave the apache2 service stopped as we wish to debug the the binary rather than attach to the service.

Setup Apache2 for Debug in gdb.

First things first let us back up the running apache2 configs:

cp /etc/apache2/conf-enabled/other-vhosts-access-log.conf /etc/apache2/conf-enabled/other-vhosts-access-log.conf.bak
cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/000-default.conf.bak
cp /etc/apache2/sites-enabled/000-default.conf /etc/apache2/sites-enabled/000-default.conf.bak
cp /etc/apache2/apache2.conf /etc/apache2/apache2.conf.bak

Now we have backed everything up we can edit all 4 files, this is purely so we can do the following:

  1. Set the variables in the conf files to be static, this then lets us save this for later use so I can switch configs at a later date to make getting into debug quicker and easier.
  2. Enable the following flag in the apache.conf to enable the to load correctly: HttpProtocolOptions Unsafe

To save time you can download the edited config from here: apache_gdb_configs

Now thats all setup lets get down to debugging the module.

Debugging with GDB

As mentioned before ensure apache2 is not running as we are going to call the binary not the process:

systemctl stop apache2

Next fire up gdb with the apache2 binary:

run gdb /usr/sbin/apache2

Next we run the binary with the -X flag which enables debug and all being well we successfully enter debug mode as shown in the image, once run successfully I hitCtrl + c then set a break point on the darkarmy function: “b *darkarmy”

In order to debug the module I want to add a few more commands to the step function by defining ‘s’ as below, this is probably overkill as I could most likely get away with setting just ‘step’ and ‘info locals’ but I want to be sure nothing is missed:

(gdb) def s
Type commands for definition of "s".
End with a line saying just "end".
>info args
>info locals

finally we are ready to test and enter ‘c’ to continue and then open a new terminal to test netcat access to the port and enter our get value of HackTheBox.

Sending the get key to the mod_rootme module:

As the get HackTheBox call via netcat has now triggered the breakpoint into the mod_rootme module we can now hit ‘s’ which was defined previously and hopefully we can begin to understand what is being executed inside the ‘darkarmy’ function.

The following screenshot shows the step through process which has been reduced to show the final result. Essentially the word HackTheBox is used as a key to an Xor method inside the darkarmy function, this is compared to a byte array stored at address “0x1bf2” we will look at this address shortly as a secondary method to get the correct key.

First Stepping through the darkarmy function shows us that it enters a loop and reads in a byte array from the above mentioned address, this is a for loop of 10 iterations so I have only shown the last 2 iterations to show the conversion process.

As we see the key we need to gain a root shell from the mod_rootme rootkit is actually FunSociety 

We now understand the following:

  1. HackTheBox is used as a key to encrypt an internal string.
  2. Calling “get keyword” enters the darkarmy function and is then used as the keyword to compare byte by byte against the byte array stored @ address “0x1bf2”
  3. If the byte comparison fails the function exits and drops the connection.
  4. If the keyword is equal to “FunSociety” then a shell is opened and root access is gained.

Obviously seeing this in context with the challenge and the Mr Robot theme has me conducting a real life facepalm ?

Method 2 Obtaining the key using GDB and Python:

The second method involves less debugging and a touch of disassembly with a bit of python.

All we need to do this time is mark the in our test dir as executable, load it into gdb then type disassemble darkarmy in order to dump the function;  the byte array location is listed beside the lea (load effective address) function:

 lea 0x11d(%rip),%rdi 

the following image shows this approach:

Next we take our byte array and copy over to a text editor, then we create a small python script to parse the byte array and xor the HackTheBox key against the encrypted string.
The following shows the python code and the byte array, note we stop at byte 0x01 as the next byte in the array is a 00 which is a null byte:

0x1bf2:	0x0e	0x14	0x0d	0x38	0x3b	0x0b	0x0c	0x27
0x1bfa:	0x1b	0x01	0x00	0x48	0x61	0x63	0x6b	0x54
0x1c02:	0x68	0x65	0x42	0x6f

encKey=['\x0e', '\x14', '\x0d', '\x38', '\x3b', '\x0b', '\x0c', '\x27', '\x1b', '\x01']

getKey=[ chr(ord(secret) ^ ord(htb)) for (secret,htb) in zip(encKey, key) ]

Running in a python3 terminal shows us the same key of FunSociety as a char array:

And there we go 2 ways to grab the key from the same function.

Thanks to Ippsec for this challenge I have learned so much in regards to debugging during this challenge and would seriously recommend anyone interested in CTFS/Infosec to watch his videos:

Ippsec Youtube Channel

Putting Things back:

Obviously we do not want to leave a root kit on a system so in order to clean up we disable and remove the module plus restore our backup configs as follows:

Restore configs:

cp /etc/apache2/conf-enabled/other-vhosts-access-log.conf.bak /etc/apache2/conf-enabled/other-vhosts-access-log.conf
cp /etc/apache2/sites-available/000-default.conf.bak /etc/apache2/sites-available/000-default.conf
cp /etc/apache2/sites-enabled/000-default.conf.bak /etc/apache2/sites-enabled/000-default.conf
cp /etc/apache2/apache2.conf.bak /etc/apache2/apache2.conf

Remove rootkit from modules and disable from apache2 available/loaded modules:

a2dismod rootme
rm /usr/lib/apache2/modules/