Tales From the Honeypot: WatchDog Evolves With a New Multi-Stage Cryptojacking Attack

Summary

• Cado Labs’ honeypot infrastructure was recently compromised by a complex and multi-stage cryptojacking attack
• Although the attack utilised many TeamTNT TTPs, we assess with high confidence that the group WatchDog is continuing to repurpose TeamTNT payloads - as they’ve done in the past
• The attack targets exposed Docker Engine API endpoints and Redis servers, and can propagate in a worm-like fashion 
• Several sophisticated techniques were employed, including timestomping, process hiding and exploitation of a misconfigured Redis database that leaves it vulnerable to remote code execution

Introduction

Cado Labs regularly analyses attacks targeting services running within our honeypot infrastructure. One recent attack caught our eye as it involved a complex, multi-stage  lifecycle, with several payloads leveraged at our honeypot machine. As we’ll discuss, this new attack involved some familiar and distinctive TTPs, but attempts by the attacker to foil attribution were obvious.

A Note on Attribution

Back in October 2021, Palo Alto’s Unit42 security research team documented a cryptojacking malware campaign that they originally attributed to TeamTNT. Upon further research, they discovered that the attack was not actually carried out by TeamTNT, but instead by a rival cryptojacking group named WatchDog. 

The Unit42 article goes into further detail about their reasoning in regards to why they believe WatchDog, and not TeamTNT, were behind that particular campaign. We’ve included a table below summarising their points and whether or not they appear in this campaign, which we believe is an evolution of the one Unit42 discovered in 2021.

As can be seen from the above, several characteristics from the October 2021 campaign also appear in this most recent campaign, with some key differences. 

Throughout the attack lifecycle, the oracle.zzhreceive[.]top is utilised heavily for delivery of additional payloads. This domain was previously attributed to TeamTNT due to the presence of the zzhreceive string. This domain is now associated with WatchDog, based on Unit42’s research. We also observed the b2f628 directory being used in payload URLs throughout the attack. However, we also noticed an additional URL path, with s3f815 being used in place of b2f628.

This suggests that WatchDog have updated their C2 infrastructure since 2021, which is plausible given this is likely a new campaign.

Our analysis also observed the 43Xbg mining address embedded in the payloads leveraged at our honeypot infrastructure. This address was attributed to WatchDog back in 2021 and is a clear indicator that they’re behind this campaign - despite heavy use of TeamTNT ASCII art throughout. 

We also observed the 1.0.4.tar.gz payload (C code for the scanning utility masscan) being downloaded and compiled on the machine. Although Unit42 mentioned that Zgrab hadn’t been used in the campaign they analysed, we did observe it in use in this campaign. We suspect that WatchDog have decided to utilise it as part of their toolkit for propagation, due to its performant and open source nature.

Overview of Attack Lifecycle

Since this particular attack involved several payloads, some executed depending on whether certain conditions were met, a brief overview of the execution flow is included in the following diagram:

Initial Access

Initial access for this campaign was via misconfigured Docker Engine API endpoints, a common method of initial access for cryptojacking malware. The attackers used a scanning tool to enumerate large portions of the internet, looking for servers with port 2375 open. In the Docker Engine’s default state, this port provides unauthenticated access to the Docker daemon, allowing an attacker to list, create and delete running containers, while also allowing arbitrary shell commands to be run on the containers themselves.

After receiving notification that the honeypot machine had been compromised, we retrieved the relevant logs and noticed some unusual activity from the IP 218.76.246[.]69.

As we can see from the first line of the snippet above, the attacker likely encountered our honeypot via an internet scan using the scanner Zgrab. A subsequent Docker ping command was issued to check if the API endpoint was responsive, after which the attackers created a new container based on the Alpine Linux image and ran a malicious command on it to kickstart the infection chain.

As the above demonstrates, this malicious command first ensures cURL is installed on the Alpine container. It then goes on to register a cron job under root, which will ensure that the first payload is kept running, by executing a command that retrieves it from a remote server and pipes it through bash.

Stage 1 - cronb.sh 

This shell script was the first payload implanted by the attacker. The script is fairly straightforward, it runs a check to see if an executable file exists at the path /bin/ps.lanigiro or /usr/bin/ps.lanigiro. If this file exists, an environment variable (envar) $PS_CMD is set to the path of the file. If it doesn’t, the envar is set to /bin/ps. Incidentally, the ps.lanigiro file turned out to be a modified version of the UNIX ps utility - more on this later. 

The script then goes on to enumerate running processes, searching for a process named ddns. If this process is running, a log statement is echoed and the script exits. This appears to be a check to see whether the machine has been previously compromised.

In a similar vein, the latter half of the script checks for the existence of renamed versions of the cURL data transfer utility. Cloud-focused malware groups, such as TeamTNT, often rename the cURL binary to obfuscate its usage. If a renamed version of cURL is found, it’s a fairly reliable indicator that the machine (or container) has already been compromised in a similar campaign.

A function named clmo(), which is dedicated to removing files and killing processes associated with Alibaba Cloud Security agents, is defined and invoked. The function also logs whether the machine is running under Alibaba Cloud, if these agents are found. 

The clmo() function is interesting as it seems unlikely that the Alibaba Security agents would be running within a newly-created container built from the Alpine Linux image. This suggests that the code has been repurposed from another, potentially more generic, campaign - one that targets Linux servers rather than containers specifically. The checks for whether the machine has been previously compromised are also indicative of code reuse, and make little sense given the method of initial access.

Finally, the script creates a directory at the path “/var/tmp/…” and retrieves a second stage payload using a renamed version of cURL (if it exists). The payload differs depending on whether the current user is root. 

Stage 2 - ar.sh (retrieved if current user is root)

Since the attack covered by this blog relies on compromised Docker containers, we’ll focus on the stage 2 payload that’s retrieved if the current user is root. Both of the potential Stage 2 payloads are similar, with some differences in persistence mechanisms for the payload run as root. The ar.sh script is particularly long, so we’ve summarised notable techniques from our analysis of it in this section.

Timestomping and Process Hiding

In the previous section, we mentioned that the cronb.sh script began by checking for the existence of a modified version of the ps utility. Analysis of this second stage payload revealed how this modified version was implanted and gave us some insight into how it works.

Shortly after the script begins, a check to determine the size of the ps binary is performed and the output is saved to the variable pssize. If the value of pssize is less than or equal to 8000 bytes, another check is performed to determine the name of the ps binary. If it’s not named ps.lanigiro then whatever is located at /bin/ps is renamed ps.lanigiro. 

The script then goes on to create a simple shell script and saves it as /bin/ps. This script is actually a very rudimentary (albeit effective) process hider. When a user issues the ps command, instead of the ps binary being executed, the process hiding shell script is instead executed. This shell script calls the actual ps binary (now named ps.lanigiro) and pipes the output through grep, which is then used to filter out lines containing the words ddns or scan. These are both associated with malicious processes run by the attacker at a later stage. 

The script also uses the touch command to modify the file access and modification times of the shell script at /bin/ps, setting them to 25-08-2016. This method of timestamp manipulation is known as timestomping and is typically used as an anti-forensics measure. We’ve observed timestomping in use by other cloud-focused malware campaigns.

The attacker uses the same procedure to modify the top and pstree utilities, hiding malicious processes from their outputs.

Alibaba Cloud Agent Removal

Continuing from the process hiding procedure described above, the script goes on to determine whether it’s running on Gentoo or CoreOS. After the Linux distribution is identified, subsequent functions are dedicated to removing monitoring tools typically found in Alibaba Cloud.

This is interesting as it suggests targeting of Alibaba Cloud Linux servers rather than Docker containers, something which doesn’t add up given the method of initial access discussed previously. This was another indicator that the payload in question wasn’t developed with this particular campaign in mind.

Remainder of script

The rest of the script focuses on removing processes from competing miners and downloading then persisting the XMRig payload. Persistence is achieved via a systemd service unit, requiring the user to be root in order to register it.

Finally, the script ends by printing out some TeamTNT ASCII art before proceeding to propagate a copy of itself by enumerating known_hosts and connecting to each host in turn. This is a technique that we’ve seen in prior cryptojacking campaigns, including from TeamTNT.

Before finishing, the script performs a check to see whether the machine has been previously targeted by this malware campaign and outputs a log statement if it has. The third stage payload is then retrieved via cURL before the terminal output is cleared.

Stage 3 - b.sh

With the main objectives of implanting XMRig and establishing persistence complete, the attackers proceed to download and install the internet scanning utilities: zgrab, masscan and pnscan. Since these utilities are being compiled on the target machine, the script begins by ensuring that various compilation tools and dependencies are installed. The most interesting of which is Redis. It seems unlikely that Redis is a dependency of any of the scanning tools so we assumed that it must be used at a later stage.

The script then finishes by downloading two additional payloads and persisting them as service units.

One interesting technique seen throughout this attack chain is the creation of directories with “...” as their name. For example, in the screenshot above we see the path /etc/.httpd/…/httpd referenced. We suspect that this is an attempt to disguise the directory, as any files or directories starting with a full stop are hidden from directory listings.

Not only that, but when you do a directory listing with a command such as ls -la, a file or directory with an ellipsis as its name looks very similar to the alias for the parent directory, which is represented by “..”. This makes it more likely to be overlooked by a user and demonstrates the attacker’s knowledge of the Linux command line.

Stage 4 - c.sh & d.sh

c.sh

This payload begins to answer some of the questions raised from analysis of the earlier stages. Bearing in mind that the script is persisted by systemd and is configured to automatically restart if it’s killed, it first begins by disabling SELinux before going on to configure ulimit and iptables. For the latter, a rule is added to drop external ingress traffic to port 6379 - the default port used by Redis. Another rule is added to accept traffic to this same port, if the source address is 127.0.0.1. 

This suggests that the attacker intends to interact with the Redis server locally (presumably via redis-cli), but wishes to prevent external access - i.e. from a panicked admin attempting to kill the service after discovering what’s going on.

Our suspicions about Redis are confirmed in the subsequent lines, where the attacker writes various Redis and cron commands to a file named .dat. 

As can be seen from the above, these commands are used to register persistence via cron. The base64 encoded data includes a payload URL for the shell script used in the first stage of the campaign and each line attempts to use a different data transfer utility to retrieve it. These findings are consistent with those of Intezer, who covered a similar campaign.

You might be wondering how the shell commands within the .dat file are actually executed. For quite some time misconfigured deployments of Redis have been vulnerable to remote command execution. Redis themselves are quite open about the risks of exposing the data store to an untrusted network, yet it seems to be an effective method of achieving initial access and persistence for this campaign.

Scanning and Propagation

The c.sh payload next performs an existence check for the pnscan binary and proceeds to use the seq and sort utilities to build a random list of IP subnets to scan. The scanning is then conducted in a loop and a hex-encoded response string of “os:Linux” is searched for. The attacker also uses the -W flag of pnscan to send a hex-encoded request string of:

*1

$4

info

This effectively runs the info CLI command against the Redis server, probing it to determine if it’s accepting commands from remote clients. IP and port information for machines running Linux and responding to the Redis probing are then extracted and the commands saved in the .dat file mentioned previously are remotely run on them via redis-cli.

The script concludes by using a similar scanning process with the masscan utility before removing the files created at this stage.

d.sh

This payload performs a similar function to the previous in that it’s responsible for propagating the malware - this time via exposed Docker Engine API endpoints. The script begins by printing out some base64-encoded ASCII art which says “TeamTNT likes the Borg”. This is not the first time we’ve seen Borg mentioned in TeamTNT scripts.

In a function named dAPIpwn(), the masscan utility is used to build up an IP address range and save it as a file named .bat. Zgrab is then used to determine if any of these IPs are hosting an instance of Docker Engine. The list of hosts that are is then saved to a variable aabbcc and each host is used by the docker run command to spin up an Alpine Linux container and run the initial access script we mentioned at the beginning.

The script ends by building up a list of random subnets and calling dAPIpwn() in a loop so that the docker run command is attempted on each Docker Engine API endpoint in those subnets.

This is likely the method used to compromise our honeypot and it ensures that the malware is propagated in a worm-like fashion.

Conclusion

This is a sophisticated and extensive cloud-focused cryptojacking attack from a group that is well-established in this area. The profitability and ease of conducting cryptojacking at scale makes this type of attack low-hanging fruit for groups such as WatchDog, and this will likely continue for as long as users continue to expose services such as Docker and Redis to untrusted networks. 

Despite many attempts to foil attribution, it’s clear that this attack was conducted by the WatchDog group and not TeamTNT - since their infrastructure and Monero wallets were used. TeamTNT have publicly declared their retirement for some time now, so perhaps there’s some truth in that. We’ll continue to be sceptical about any future campaigns we analyse which make use of TeamTNT TTPs.

Cado Response Analysis

Indicators of Compromise (IoCs)