SSH Botnet Honeypot - Pt.3 - Analysis

What

Let’s see what the cat dragged in. With the Honeypot Qemu machine working we can analyse the logs.

Contents

When

I’ve run the honeypot for 2-3 hours, but already within 20 minutes the log was full of login attempts.

Even though the server is wide open and will accept any password for root there are login attempts for different user names that fail, because they don’t exist.

Why

I’m really curious to see what the state of Linux malware is and what kind of discoveries we will make.

Background

The Qemu virtual machine is using a *.qcow file as it’s virtual disk. We will need to mount that on our host machine to read the files in it.

Additionally modern linux uses journald as its logging service. To read a journald log we need to use a journalctl. Unfortunately there have been breaking changes in the journalctl that will likely require the use of chroot to be able to enter the same environment and use the journalctl from the honeypot machine as described below.

How

Accessing the Logs

Accessing the logs from the Qemu machine takes some steps that can easily be scripted.

Backup the Disk

In order to analyse the logs offline with the Qemu honeypot still running, we can simply copy the disk image.

cp hda.qcow hda-original.qcow

Compare with virt-diff

I didn’t have much luck with virt-diff, it should have worked as follows:

sudo apt -y install guestfs-tools
sudo virt-diff -a hda-original.qcow -A hda.qcow

Access by mounting the virtual disk

However you can mount the original and infiltrated qcow image as mentioned above and do a regular diff on the directories.

For convenience I’ve written a small shell script. It makes a copy of the running qemu disk and mounts it.

#!/bin/bash

sudo umount /mnt/tmp
sudo qemu-nbd --disconnect /dev/nbd0

cp hda.qcow hda-snapshot.qcow

sudo modprobe nbd max_part=8
sudo qemu-nbd --connect=/dev/nbd0 hda-snapshot.qcow
sudo mount /dev/nbd0p1 /mnt/tmp

and to remove

#!/bin/bash
sudo umount /mnt/tmp
sudo qemu-nbd --disconnect /dev/nbd0

Access log files

With the disk mounted we can access the log files.

journalctl incompatibilities and chroot

The Debian 12 I’m using

cat /mnt/tmp/etc/*release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"

uses journald for logging. Logs are found under:

/mnt/tmp/var/log/journal/534c2f67387847c396690747c845ab99/system.journal

This is a more advanced logging mechanism than syslog.

Normally you would read the logs like so

journalctl --file /mnt/tmp/var/log/journal/534c2f67387847c396690747c845ab99/system.journal

For instance, with journalctl from

PRETTY_NAME="Ubuntu 22.04.3 LTS"

I receive an error message

Failed to open files: Protocol not supported

The logfiles are compressed and there are incompatibilities between different versions of journalctl.

I can’t install a newer journalctl on my machine as it’s linked to systemd and that would update the entire system. I thought about compiling a newer journalctl from source, but it’s tied to systemd and requires dependencies I can’t easily install.

But as we have a full system-image from the Qemu image the easiest way to get around this is to chroot and use the journalctl version provided by Debian 12.

cd /mnt/tmp
sudo chroot /mnt/tmp /bin/bash
journalctl -m

View log files

To check for ssh login attempts, inside the chroot environment, the “-u ssh” option can be used.

journalctl -u ssh

Analysis

With that we can have a look at the log files from journald.

Failed due to invalid usernames

Even though anyone can log-in with the user “jason” and “root” with any password they please I couldn’t add all usernames the bots attempt to use and so we still get failed login attempts.

journalctl -u ssh | grep "Failed"

Mar 25 17:30:33 raspberrypi sshd[421]: Failed password for invalid user faridehnadimi from 117.72.44.30 port 45184 ssh2
Mar 25 17:34:12 raspberrypi sshd[449]: Failed password for invalid user omkar from 117.72.44.30 port 50276 ssh2
Mar 25 17:35:57 raspberrypi sshd[456]: Failed password for invalid user dan from 117.72.44.30 port 52598 ssh2

Accepted root logins

But, as we’ve allowed any password for our root user, we get loads of accepted root logins from various IP-Addresses

journalctl -u ssh | grep "Accepted"

Mar 26 11:35:02 raspberrypi sshd[2344]: Accepted password for root from 43.156.6.131 port 41394 ssh2
Mar 26 11:35:03 raspberrypi sshd[2346]: Accepted password for root from 190.0.63.226 port 47516 ssh2
Mar 26 11:35:08 raspberrypi sshd[2358]: Accepted password for root from 190.0.63.226 port 49374 ssh2
Mar 26 11:35:27 raspberrypi sshd[2365]: Accepted password for root from 165.154.131.4 port 35120 ssh2

journalctl -u ssh | grep "Accepted password for root " | awk '{print($11)}' | sort | uniq -c | sort -r -k 1

43.156.6.131
159.203.104.187
165.154.131.4
190.0.63.226
43.134.164.71
50.114.64.139
34.41.17.26
119.91.139.84
31.19.59.52
127.0.0.1

Attempted user names

Attempted user names within 1-2 hours

journalctl -u ssh | grep "Invalid user " | awk '{print($8)}' | sort -k 2 | uniq -c
    133 345gs5662d34
      1 administrator
      1 ahmadafsari
      4 ahmaddd

Expand for more

      2 alib
      1 alimokari
      3 alis
      1 amirf
      3 anid
      2 auraj
      1 az
      2 bakhtiari
      1 charlie
      2 daedong
      1 dai
      1 dan
      3 daniel
      5 developer
      1 dsceltic
      1 dspace
      1 f-tomizawa
      1 faridehnadimi
      4 feisal
      1 ghaderi
      3 git
      2 gpappas
      1 gracesb
      2 hezujiao
      2 hyundai
      2 jenkins
      3 kingabra
      1 lazer
      2 liujiahao
      2 liweiliang
      1 luiscarlos
      1 lvm
      2 m7
      2 marina
      1 mehrnoosh
      3 minecraft
      3 monitirtha
      3 nani
      1 olga
      1 omkar
      1 ork
      1 pippo
      2 r00t
      1 royafallahi
      2 rpandey
      1 sato
      2 sekreto
      1 senkhxsi
      3 shinestar
      3 student3
      3 sysadmin
      4 test
      4 tester
      4 testwww
      1 tom
      9 ubuntu
      1 unclem
      1 upload
      4 user
      1 www
      1 xsp
      1 youlog
      3 zhinoos
      1 zhouwei

Mysterious username

The username

345gs5662d34

was attempted hundreds of times.

That username has also been observed by others [8, 9]

On deencode [1] it decodes to a japanese sequence that Google translate translates to “Let’s meet each other”, but I’m not sure if that is just a faulty translation.

Mysterious SSH-key

An SSH key was added to my “authorized_keys”. The “mdrfckr” part of it immediatly caught my eye.

cat ./root/.ssh/authorized_keys 

ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8
dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/
75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengv
S8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hW
CwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWY
YjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr

Results

I’ve made some observations from running the honeypot for a short while.

“mdrfckr” SSH-key

The most interesting find was that SSH-key that got automatically added to my .ssh/authorized_hosts.

Looks like this has also been going on for a while [11,12,13,14,15] and seems to be related to a crypto-mining bot, either a modified Mirai, NoaBot, Dota or similar running the xmrig crypo-miner.

Many Intrusions, but not much happend

Interestingly, even though the system is wide open and there are logins, I see nothing special happening on the machine or in my Wireshark log.

Perhaps the successful logins are being stored in some database and, at another time, an actual attack may occur.

Honeypot being detected

Perhaps the honeypot is being detected. If we accept all passwords and the attackers are clever, they will realize that this system allows any password and is likely a honeypot.

Another indication of this being a honeypot is the cpuinfo showing Qemu.

My honeypot is not well hidden. From what I’ve read it seems that many botnets check cpuinfo. For a crypto-mining bot the performance of the device is interesting and CPU details may also reveal honeypots.

cat /proc/cpuinfo | grep "model name"
model name	: QEMU Virtual CPU version 2.5+

So perhaps a further infection does not occur due to this.

State of Linux Malware

During my research of that SSH-Key and Botnets in general I’ve found some information on the current state of Linux malware.

Intrusion

It seems the use of security vulnerabilities are rather rare, though easy to implement attacks against old vulnerabilities like “shellshock” [14] are still being used. Most attacks we see are dumb brute-force attacks where lists of user/password combinations are attempted.

Many bots are based on the “Mirai” Bot [3]. It originally only attacked Telnet-servers and was open-sourced revealing the hardcoded password list [4].

Goals

The goals of most of these linux botnets are to

maintain access to a system, allowing download, upload and execution of files and sometimes shell access
allow for coordinated DDOS attackes. Usually SYN/ACK/GRE (generic routung encapsulation) [9] packet flooding
install cryptominers, often “XMRig” is used.

Hiding

Some bots run only on memory and don’t write anything to disk. They don’t persist after reboots, but weaken security by adding malicious SSH-keys to authorized_hosts.

Some bots install trojans like XORDDOS [5], which have sophisticated rootkits that hook into kernel functions to hide from task lists, network port lists and file browsers.

Based on Open-Source

The XORDDOS apparently uses portions of “suterusu” [6] an open-source linux kernel module rootkit.

There is also mirai [7, 16] which was open-sourced. Process injections proof-of-concepts like “injectso64” [8]. The “kaiten” IRC bot [11] and its rewrite [12]. Or scripts for brute forcing for priveledge escalation [10]. All of these are available as open-source online and are being used by malware creators.

Use of programming languages

Malware creators seem to use a variety of programming languages. With some written in perl [13], others in C, go or simple shell scripts. The langauge “go” seems to be popular as it’s a bit more difficult to reverse engineer.

Common Linux Malware Names

bots

“Mirai”
“Noabot”
“Kaiten”
“Ziggy”
“Hand of Thief”
“Tsunami”

rootkits

“suterusu”

tools

“MIG Logcleaner v2.0”
“Shadow Log Cleaner”

cryptominers

“XMRig”

trojans

“XOR DDoS”

Sophistication

Most of the bots are poorly written, contain cobbled together code and just try passwords. But by far the most sophisticated malware on Linux I’ve seen so far is the XORDDOS trojan with its built-in rootkit

State actors

It truely seems there is a cyberwar with state actors going on. Groups such as the ‘Outlaw Hacking Group’ [15] are trying to gain access to western systems. These groups seem to be particularly blunt as they need not fear repercussions.

Progress

Conclusion

It feels like I’ve found a new hobby. Next time I’ll try to hide honeypot a bit better and see if the bots will then behave differently.

1] https://dencode.com/en/
3] https://en.wikipedia.org/wiki/Mirai_(malware)
4] https://github.com/jgamblin/Mirai-Source-Code/blob/3273043e1ef9c0bb41bd9fcdc5317f7b797a2a94/mirai/bot/scanner.c#L124
5] https://unit42.paloaltonetworks.com/new-linux-xorddos-trojan-campaign-delivers-malware/
6] https://github.com/mncoppola/suterusu
7] https://github.com/jgamblin/Mirai-Source-Code
8] https://github.com/ice799/injectso64
9] https://en.wikipedia.org/wiki/Generic_Routing_Encapsulation
10] https://github.com/carlospolop/PEASS-ng/tree/master/linPEAS
11] https://github.com/brianredbeard/toorcon16/blob/master/kaiten/kaiten.c
12] https://github.com/isdrupter/ziggystartux
13] https://www.trendmicro.com/en_us/research/18/k/perl-based-shellbot-looks-to-target-organizations-via-cc.html
14] https://securityintelligence.com/articles/shellshock-vulnerability-in-depth/
15] https://hackernoon.com/outlaw-hacking-group-resurfaces
16] https://github.com/techgaun/mirai/tree/master

Dennis Salzner - SSH Botnet Honeypot - Pt.3 - Analysis