As the adoption of containerized environments like Docker continues to rise, so does the interest of cybercriminals in targeting these platforms. In recent years, Docker has become a popular attack vector due to its widespread usage in production environments, and the ease with which attackers can exploit misconfigurations. In this two-part series, we dive deep into an investigation of Docker malware, focusing on how threat actors utilize packed ELF binaries to spread their malicious payloads within containerized systems.
In Part 1, we explored how disk-based artifacts and automated tools can be used to identify suspicious activity within a Docker host. Now, in Part 2, we shift our attention to a full malware analysis of two ELF binaries discovered during our investigation. These binaries, packed using UPX (a common packer seen in malware), presented a unique challenge as the attacker had stripped the UPX header, preventing traditional unpacking methods. Through manual unpacking and static analysis, we will unravel the attacker’s methods, exposing the malware’s inner workings, including its techniques for persistence and spreading through Docker environments.
Let’s dive into the details of how we manually unpacked these binaries, reverse-engineered their functionality, and identified their role in a larger container compromise.
During the investigation, we discovered two interesting files, which we identified as ELF binaries. In this section, we will show how to analyze these files.
Firstly, let’s drop one of the binaries into detect-it-easy to see what we’re working with.
This shows the binary is packed with UPX, an open-source software packer seen frequently in malware. The UPX command line utility has a decode flag (-d) to unpack UPX binaries. However, when we try to run it on these samples, we get a “NotPackedException: not packed by UPX” error. This is because the attacker has stripped the UPX header from the file, making it not possible to unpack using the UPX tool. As such, we’ll need to manually unpack it by hand. We have done this before in a previous blog.
First, we need to understand how UPX actually works. When a file is packed, the contents of the ELF are encoded and compressed. A new ELF file is then created with a UPX code stub at the entry point and the compressed original code in the file as a data blob that the UPX stub can read. When the new ELF is run, the UPX stub decompresses the original code and creates a memory region for the original code to be written into. Once this is complete, the stub jumps into the entry point of the original ELF, handing control over to the payload.
Let’s run an strace on the nginx binary to highlight this:
$ strace ./docker-nginx
execve("./docker-nginx", ["./docker-nginx"], 0x7ffeb29056d0 /* 23 vars */) = 0
--- Suspected UPX unpacking stub ---
open("/proc/self/exe", O_RDONLY) = 3
mmap(NULL, 3093, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f29be464000
mprotect(0x7f29be464000, 3093, PROT_READ|PROT_EXEC) = 0
readlink("/proc/self/exe", "/home/ubuntu/docker-nginx", 4095) = 25
mmap(0x400000, 2091024, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x400000
mmap(0x400000, 917514, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x400000
mprotect(0x400000, 917514, PROT_READ|PROT_EXEC) = 0
mmap(0x4e1000, 853656, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0xe1000) = 0x4e1000
mprotect(0x4e1000, 853656, PROT_READ) = 0
mmap(0x5b2000, 104768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0x1b2000) = 0x5b2000
mprotect(0x5b2000, 104768, PROT_READ|PROT_WRITE) = 0
mmap(0x5cc000, 206864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x5cc000
brk(0x5ff000) = 0x1d5e000
munmap(0x5ff000, 706435) = 0
mmap(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f29be463000
close(3) = 0
munmap(0x7f29be464000, 3093) = 0
--- Suspected start of attacker code ---
arch_prctl(ARCH_SET_FS, 0x5cc830) = 0
sched_getaffinity(0, 8192, [0]) = 8
openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
read(3, "2097152\n", 20) = 8
We can see after running the binary, it opens a stream to its own code using /proc/self/exe so it can read the file’s contents. It then allocates a number of memory regions for the unpacked binary to occupy and sets flags on them to make it executable. It also allocates a couple of ranges that are used for temporary work, such as for handling decompression. We can take an educated guess that after it closes the stream to /proc/self/exe, and unmaps the working memory sections it allocated, the unpacking process is finished. We can see after this point it makes a few calls, such as checking hugepages, which is likely attacker code.
Using GDB, we can set a catchpoint on any syscalls, and step through the binary until we hit the final call to munmap.
We can see that after the syscall, the current function returns and then jumps to a code stub that moves the original argv and argc into rsi and rdi respectively. It then jumps into a go-style prologue that grows the stack by subbing from rsp. We can speculate that this is the entry point of the original binary.
Now that the program appears to be unpacked, we can run info proc mappings to see the memory regions created by UPX.
So let’s dump the memory and see what’s inside. To do this we can use the dump memory command, which takes in a file name and the start & end addresses as parameters to dump.
Quitting out of GDB, we can now concatenate these together into a singular ELF file, and voila, we now have a shiny new unpacked Go binary. Running strace on the binary, we can confirm that our theory about where the UPX stub ended was correct.
$ cat 1.dmp 2.dmp 3.dmp > combined
$ file combined
combined: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=WxSIvcsqMz0xQ8wotRgQ/H7-ppPqeODKfbkLgrmiL/IJI0u6QHVIuF0QLMGGTB/lAKt0fHOBu0nioeTavjO, not stripped
$ strace ./combined
execve("./combined", ["./combined"], 0x7ffe30cefe80 /* 23 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x5cc830) = 0
sched_getaffinity(0, 8192, [0]) = 8
openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
read(3, "2097152\n", 20) = 8
And running the binary in GDB, we can see the entry point is where we expected, just before argc and argv are loaded before the Go prologue we previously noted.
Now we can continue to analyze the binary, without the pesky packing getting in the way.
Let’s start by opening the binary in IDA. IDA is a powerful disassembler that will allow us to statically analyze the source assembly of the binary. IDA by default will drop us into the entry point of the application, and we can navigate the flow from here by double clicking symbols.
We can recognize the entry point code from the debugging earlier - although this time we have more contextual information such as symbols available.
We can see this chain of three functions line up with the snippet we saw in debugging - with a jump, loading argc and argv, and then the Go prologue. Now we can see these are part of the Go internal runtime.
This binary is not stripped, meaning it has symbols for the function names. This lets us see what the attacker named each function, making it easier to determine the functionality. Looking through the function listing, we can eliminate most functions as belonging to internal or third-party Go libraries, and realize that the attacker code is entirely in the main module. We can also verify the main_main is in fact the start of the attacker code by examining XREFs back until we reach the entry point, confirming the only code that runs before is internal Go runtime code.
From here, we can take an educated guess that this binary is responsible for spreading. Through static analysis, we can confirm our hypothesis, by reading through the assembly of each function to determine how they operate.
Looking through the main function, we can see it initializes a random number generator and a logger (with path set to /var/log/nginx.log), and launches a goroutine. When looking at the assembly, this can be simple to miss among the noise, however it is a very important feature to analyze.
Let’s break down how the goroutine pattern works. First, some memory is allocated using runtime_newobject for the struct at 4D7AA0, which is a runtime_type struct. This will store information needed for the goroutine, and the pointer to it is stored in rax. The function pointer to runtime_cgocallbackg1_dwrap_2_10 is then loaded into rcx, which is then added to the struct by moving it into the memory address pointed to by rax. The logger created earlier, which will act as the first argument, is then added to the struct after the function pointer. The Go internal function runtime_newproc is then invoked, which takes in the struct as an argument in rax and uses it to start the new goroutine.
You may have noticed that there are actually two branches. As Go is a garbage collected language, there are some additional checks it needs to make for the concurrent garbage collector to work properly. The logic here relates to the “mark” phase of the garbage collector, where the garbage collector is taking account of the allocated heap objects to determine which can be deallocated. As concurrent changes to objects may affect the result of this, it enables the write barrier to block direct writes.
As the struct is heap allocated and in scope of the garbage collector, the code must perform a check to see if the write barrier is enabled. If it is, it invokes the runtime_gcWriteBarrierCX function. This function performs the same assignment as the other branch, but allows the garbage collector to be notified of the change and take it into account for marking. The garbage collector can be ignored during malware analysis in most cases.
Navigating to the callback function defined in the goroutine, we can see it restores the argument and calls the original attacker-controlled function.
In this case, it is a simple loop that will run pgrep to check for the cloud process (the other dropped binary), and start it if not found.
After this, the main thread then continues to a thread that infinitely loops, selecting random subnets to scan.
Interestingly, it appears to scan two at a time. Looking at the logic for the generateRandomSubnet function, it generates two random numbers and string formats them into a /16 subnet in CIDR notation. In this case, we can use the IDA decompiler view to get a higher-level overview, however, the decompiler is not perfect and should not be blindly trusted, nor used as an alternative for reading the assembly. We can also use the n keyboard shortcut, which allows us to rename symbols to make the logic of the code more apparent.
Let’s move on to the scan function and look at it in the decompiler.
You may be confused reading the decompiled code as to how exactly the parameters end up getting assembled into the array, and why the subnet doesn’t appear to be included as an argument at all. This is because you have been deceived by the decompiler.
In Go, a common pattern is to use an empty interface as a collection to store data of multiple types in an array-like manner, which when combined with heavy compiler optimization results in the above. So while IDA is displaying each param as its own stack variable, they are actually members of the param_interface collection. This is why assembled_array is created from the pointer to the start of the interface object.
As IDA’s decompiler doesn’t understand the pattern, it also misses out on the subnet assignment. If we look at the actual assembly, we can see that the first element of the interface is actually set to rax and rbx. The decompilation correctly shows that rbx is the subnet string length, but not that rax is the subnet string itself. In Go, strings are almost always handled as a pointer to the start of the string, and the length. The strings are not null-terminated like C-strings are, so the length is needed to know how far to read from the pointer.
The rest of the binary can be analyzed in a similar way, and ultimately just wraps several commands in succession, making it a breeze to analyze by reading the strings and pseudocode. To surmise, after masscan finds a host, it logs the IP address, and then performs the following commands to infect it:
This matches with the behavior we saw during the earlier analysis of the network and disk image.
The “cloud” binary is also an unstripped ELF binary and we can use the same method to find the main function for the attacker code. However, we can quickly spot a number of interesting functions in the function listing.
Based on the functions mineblock, getwork, and threadaffinity which would be seen in a crypto miner, this binary looks to be a crypto miner. Investigating the main method further, we can see some symbols that stand out, such as symbols referencing github_com_deroproject_derohe. Some Google detective work shows us this is a cryptocurrency called Dero, with an open-source implementation of a miner named “dero-stratum-miner” that can be found here.
Comparing the symbols present with the Go Miner implementation, they line up exactly. This leads us to believe that the cloud binary is likely directly pulled from this git repository, with minimal changes. We can use a tool like bindiff to find any deviations between the official binary and the attacker’s payload.
|
File name |
Hash |
|
cloud |
558746c715eae6f6edd7854a469293bd1047c23fcb748105d183b706181059d9 |
|
nginx |
ae1fb47c96264f1db1fbe76004206fc4c598012a7cb2e94b753ae768e98a8859 |
As both files are UPX packed, generic UPX rules will match against them.
rule ELF_UPX_Packed {
meta:
description = "Detects ELF binaries packed with UPX"
author = "nbill@cadosecurity.com"
date = "2024-09-25"
strings:
$upx_sig = "UPX!"
condition:
uint32(0) == 0x464c457f and $upx_sig in (0..1024)
}
In this blog series, we've walked through the crucial steps required to analyze Docker-based malware, from initial detection to manually unpacking UPX-packed ELF binaries. As Docker remains a prime target for cyber attackers, it's essential to stay ahead of evolving threats, which include increasingly sophisticated payloads and techniques designed to evade traditional detection methods.
Our investigation highlighted several key points:
As threat actors continuously adapt, so too must the defenders. Staying informed on these evolving techniques ensures you're equipped to identify, analyze, and mitigate threats before they cause damage to your systems.
Want to see how Cado Security can help you combat advanced container-based threats? Contact us today to schedule a demo and discover how our platform can give you the insights and tools needed to detect and respond to sophisticated malware targeting Docker environments. Our team is ready to help you protect your containerized systems.