This is the second of a three-part blog series written by Cado and Invictus Incident Response, where we are investigating an incident that was discovered during an account audit of an Amazon environment.
In part one, we walked through the analysis of AWS logs for account DevOps1-bpvbp - which had been used to create an unknown account DevOps3 on 26 Oct 2022 at 21:29:03 UTC. Additionally the DevOps1-bpvbp account had initiated a StartSession which provided a connection to EC2 instance i-08e56d6b6c0439daf using Session Manager, at 21:29:53 UTC.
From here, we will continue the investigation analysis to answer the following questions:
As mentioned in the previous blog, we are focusing on the phases Acquisition and Processing & Analysis steps. Using the Cado platform, those phases look more like Acquisition & Processing and Analysis.
To start off using the Cado platform, investigations are compartmentalized as Projects, using the UI for the platform we created a project and acquired EC2 instance i-08e56d6b6c0439daf.
A disk image is created from a snapshot of the volume attached to the instance i-08e56d6b6c0439daf and automatically processed giving the analyst access to a super timeline and file content.
From previous analysis, we know that the AWS Session Manager initiated session took place at 2022-10-26 21:29:03, so that will be the pivot point.
As this is a Linux system I will start with the secure log file, the file shows that between Oct 26 21:29:53 and Oct 26 21:36:03 there was activity for the users ssm-user and ec2-user. ssm-user is the user account associated with SSM Session Manager and ec2-user is the default user for Amazon Linux systems and the time of the account being accessed lines up with the log analysis.
secure log shows session start and session close timestamps:
Oct 26 21:29:53 ip-10-0-2-54 useradd[32611]: new group: name=ssm-user, GID=1001
Oct 26 21:29:53 ip-10-0-2-54 useradd[32611]: new user: name=ssm-user, UID=1001, GID=1001, home=/home/ssm-user, shell=/bin/bash
Oct 26 21:30:08 ip-10-0-2-54 sudo: ssm-user : TTY=pts/0 ; PWD=/usr/bin ; USER=ec2-user ; COMMAND=/bin/bash
Oct 26 21:30:08 ip-10-0-2-54 sudo: pam_unix(sudo-i:session): session opened for user ec2-user by (uid=0)
Oct 26 21:36:03 ip-10-0-2-54 sudo: pam_unix(sudo-i:session): session closed for user ec2-user
Now we have a time window of interest.
Using the search function in the Cado platform, we want to see what activity took place on i-08e56d6b6c0439daf between Oct 26 21:29:53 and Oct 26 21:36:03 so let’s add a timestamp range 2022-10-26 21:28:00 to 2022-10-26 21:38:00 to the search:
AWS SSM generates quite a lot of log events which results in a lot of noise so we removed those entries from the timeline search, reducing the number of events from 227 to 57:
Along with the secure log, the audit log tracks user logon activity, here we can see the ssm-user is linked to audit entry USER_ACCT which is triggered when a user-space user account is modified.
Our Investigative Questions are:
Timeline Analysis and file content can help answer these questions.
Between 21:29:53 and 21:36:04 we have events that show an AWS SSM interactive session had started; the associated ssm-user switched user to the default Amazon Linux2 account ec2-user which initiated a new bash session. All of this takes place under the process ID 32627 directly linking the ssm-user and ec2-user accounts to the attacker activity. The ec2-user known-hosts file was accessed and a file named staff.txt was created in the ec2-user’s default directory.
Next step is to review the bash_history files for ssm-user and ec2-user to identify any commands the attacker has executed.
The screenshot above is from using the Cado platform to view the contents of ssm-user’s bash_history file. The command seen within the file doesn’t have an associated timestamp, however, in our timeline there is log evidence that indicates that sudo command was executed to switch user to ec2-user during the attacker’s session.
The screenshot above shows the contents of ec2-user’s bash_history file. Again, the commands seen within the file don’t have timestamps, however in our timeline we did see the ec2-user’s known_hosts file being accessed, so there is a good indication the ssh commands are linked to the attacker’s session. Also in our timeline we can see the file staff.txt was created during the attacker’s session which is likely to be the result of the scp command.
The other commands seen at the beginning and end of of the bash_history file are also interesting:
macid=$(curl -sS http://169.254.169.254/latest/meta-data/network/interfaces/macs/) && vpcid=$(curl -sS http://169.254.169.254/latest/meta-data/network/interfaces/macs/${macid}/vpc-id)aws ec2 describe-security-groups --region eu-west-1 --filters Name=vpc-id,Values=${vpcid} Name=egress.ip-permission.to-port,Values='22' --output text
The first command in the code box above, shows the Instance MetaData Service (IMDS) being used to extract information regarding the Virtual Private Cloud (VPC) Id. The VPC Id is then used in conjunction with the aws command line tool to provide detailed information of the security group that the instance belongs to, in particular looking for port 22 entries, port 22 is commonly associated with SSH. This could be seen as a method to perform network discovery to find other hosts to laterally move to.
The last two commands in the bash_history file show that the aws command line to create a S3 bucket and the file staff.txt was copied to it. This could be seen as staging data for exfiltration:
aws s3 mb s3://eu-west-1-prod-data --region eu-west-1
aws s3 cp staff.txt s3://eu-west-1-prod-data --region eu-west-1
Our analysis of i-08e56d6b6c0439daf has found that the scope of our investigation has extended to another system within the network, 10.0.8.25, using the AWS console we identified that within the Security Group that IP address corresponds to an EC2 instance with ID i-020e722274233c7f2 and our Investigative Questions carry over to that host:
Again we start the analysis with the secure log file:
Oct 26 21:31:55 ip-10-0-1-82 sshd[6867]: Accepted publickey for ec2-user from 10.0.2.54 port 45300 ssh2: RSA SHA256:C/JXEdpi4ui1exVUpnkKYCLDX8Xs9WYXdV1E+Mt7aAw
Oct 26 21:31:55 ip-10-0-1-82 sshd[6867]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)
We can see the timings for the ssh connection line up with the findings from our analysis of i-08e56d6b6c0439daf.
Using the same time window and a filter to show LOGIN and USER from the audit log along with the rest of the system and user events gives 53 events for further analysis.Due to the configuration of this system, user executed commands are saved to the audit log, the commands are HEX encoded and the Cado platform automatically decodes them to the attacker executed the following commands:
-bash –login -c psql
psql -d prod_data_01 -c SELECT * FROM staff;
These commands show the attacker was very much interested in information from the staff table in the postgres database. Returning to the timeline we can see that after the last command was executed /tmp/staff.txt was created:
We now have two systems that have references to the file staff.txt, as we are working with a disk image we can view the content of the file staff.txt from both systems.
From i-020e722274233c7f2:
and from i-08e56d6b6c0439daf:
We can see that the file SHA256 hash and content are the same for both files on both systems.
The last section section of timeline analysis for i-020e722274233c7f2 shows the attacker’s ssh session terminating followed by another ssh session starting and terminating within the same minute:
We know from the ec2-user bash_history file there was an scp command executed after the SSH session which ties in with the timeline.
Let’s look at the investigative questions and where we stand:
What actions were performed on the host?
To recap, we’ve performed analysis of i-08e56d6b6c0439daf and i-020e722274233c7f2 and identified that during the attacker’s AWS Session Manager session:
IMDS and aws command line tool were used to perform network discovery to identify other hosts for lateral movement;
i-020e722274233c7f2 for additional suspicious activity.
Are there other traces of suspicious activity?
staff.txt;
staff.txt file from i-020e722274233c7f2 to i-08e56d6b6c0439daf; and
staff.txt to it.
What happened to the database dump file staff.txt and how did the attacker get access to the account DevOps1-bpvbp? All will be revealed in the final part of this blog series.
Ready to perform your own cloud forensics investigation? Check out the 14-day free trial of the Cado platform.