Wipe RAM on shutdown

[adrelanos]

Some stuff that Tails is having in mind.

---

package:

http://git.tails.boum.org/wiperam/tree/

---

Tails currently has a few issues with it.

https://tails.boum.org/support/known_issues/index.en.html#index23h2

On rare systems (non-PAE with big amounts of memory), Tails does not consistently erase all system memory as it should.

The other issue is an obvious one. If shutdown fails for software or hardware reasons, RAM shutdown won't be executed.

https://labs.riseup.net/code/issues/6006

After following the steps for "erase memory on shutdown it turns out some patterns may survive the wipe which indicates that not all memory was wiped. One theory for this is that the new kernel loaded by kexec may allocate some buffer, which then is filled with the pattern and won't get cleared, since sdmem only wipes unallocated memory.

Hi, not sure how to comment so I will just add it here. The test is incorrect, since on a 32-bit architecture the address space of a given process is usually limited at 3 GiB (or less, depends on the kernel configuration). It's not easy to change the Python script to fill all available memory (it's possible, but complex due to kernel's OOM killer behavior), but your best bet would be to spawn many pattern filler processes simultaneously, hoping that at least some executions will overlap. Then, on a 4 GiB machine you will see about 1 GiB of non-wiped memory (and on systems with > 4 GiB RAM, with PAE enabled, you won't wipe that extra memory, too, of course). Sorry to burst your bubble -- wiping memory is not easy, and THC's SecureDelete package is mostly badly implemented snake oil based on wrongly interpreted old theoretical papers. Anyway, to see whether the 3 GiB limit has anything to do with the unwiped patterns you have seen thus far, retry on a system with less than 3 GiB RAM. --MK

In theory, it's supposed to work "most of the time", but it is not reliable as is: sdmem kills itself as soon as it's refused to allocate memory, so there's no guarantee several instances of it will be allocating enough memory at the same time to ensure all memory is erased. In the worst case, this change can make the memory erasure process 32 times longer, with no efficiency improvement.

And more.

---

Tails blueprint:

https://tails.boum.org/blueprint/more_efficient_memory_wipe/

---

Documentation on testing if wiping RAM works:

https://tails.boum.org/contribute/release_process/test/erase_memory_on_shutdown/

---

Test suite recipe:

https://github.com/vjrj/tails/blob/master/features/erase_memory.feature

---

Documentation:

https://tails.boum.org/doc/advanced_topics/cold_boot_attacks/index.en.html

---

Related:

• DisposableVMs: support for in-RAM execution only (for anti-forensics) #904
• Wipe Video RAM on Shutdown #1563

[tasket]

Good feature, but a little reminder: Some memory controllers scramble addresses and data using prng pattern, such as on Intel Core processors using ddr3. So far haven't seen anyone defeat this, so maybe some users will feel less vulnerable. See pp. 26-29 http://www.slideshare.net/codeblue_jp/igor-skochinsky-enpub

[awokd]

Looks like the kernel is compiled with CONFIG_XEN_SCRUB_PAGES=y. That should result in DomUs getting wiped on memory free operations, including a shutdown. Was this issue filed against the host itself? #2024 refers mostly to VMs.

[andrewdavidwong]

Updating title to explicitly include things mentioned in #4488.

[marmarek]

On memory balancing, the memory definitely is zeroed before returning it to Xen. On VM shutdown/crash/etc it is too, but I'm not sure if it's done synchronously, or asynchronously - IOW at which point in time it is guaranteed to be cleared already.

[andrewdavidwong]

On memory balancing, the memory definitely is zeroed before returning it to Xen. On VM shutdown/crash/etc it is too, but I'm not sure if it's done synchronously, or asynchronously - IOW at which point in time it is guaranteed to be cleared already.

Updated title to:

Wipe RAM of VMs synchronously when shut down and killed

(Feel free to change it further.)

[adrelanos]

Progress on this feature has been made. Source code review and suggestions for the sdmem parameters are welcome, see:
https://forums.whonix.org/t/is-ram-wipe-possible-inside-whonix-cold-boot-attack-defense/5596/39

This is now in the development version of security-misc. (#1885)

[adrelanos]

This progressed far, has now a dedicated source code repository and sub project page.

• https://www.kicksecure.com/wiki/Ram-wipe
• https://github.com/Kicksecure/ram-wipe
• https://www.kicksecure.com/wiki/Dev/RAM_Wipe

Nowadays dracut based. But could probably easily ported to mkosi-initrd if that ever becomes the default init system in Qubes dom0 host.

Remaining issue is actually a dracut issue, which is not easy to fix.

• dracut should unmount the root encrypted disk cryptsetup luksClose during shutdown dracutdevs/dracut#1888
  • update: fix(crypt): close crypt devices to release encryption keys get on shutdown dracut-ng/dracut-ng#94
• fix(crypt): close crypt devices to release encryption keys get on shutdown dracutdevs/dracut#2471
  • update: close crypt devices to release encryption keys get on shutdown dracut-ng/dracut-ng#204

[renehoj]

What is the main scenario where this is useful, in what situation is the VM shutdown in a controlled manner, but you can't rely on Xen to clear the memory?

[adrelanos]

I was also wondering what I was thinking when ~9 years ago I set the title to

Wipe RAM of VMs synchronously when shut down and killed

which from nowadays perspective does not make a lot sense to me. Except after you asked, I checked the history and the title I've chosen was actually:

Wipe RAM on Shutdown

This would be a dom0 feature. This is what I was suggesting and what I still think makes sense. Why is it useful? It's for cold boot attack defense. Reference:
https://www.kicksecure.com/wiki/Cold_Boot_Attack_Defense

@adw changed the title Wipe RAM on Shutdown Wipe RAM of VM when it shuts down or is killed, and during memory balancing on Sep 14, 2019

Wipe RAM of VMs synchronously when shut down and killed

I don't think the title should have been changed or the scope of the ticket as complicated as it is by itself should have been extended.

• Wipe RAM of VM when it shuts down or is killed, and during memory balancing #4488 is in my opinion not a duplicate of this ticket.

To answer your question...

What is the main scenario where this is useful, in what situation is the VM shutdown in a controlled manner, but you can't rely on Xen to clear the memory?

I didn't suggest VM memory to be cleared when a VM is closed. Maybe that's #4488. This ticket should only be for Wipe RAM on Shutdown.

[renehoj]

This patch might be relevant, at least it confirms that scrubbing doesn't always get scheduled until the page is allocated.

https://github.com/OpenXT/xenclient-oe/blob/master/recipes-extended/xen/files/memory-scrub-on-domain-shutdown.patch

[brendanhoar]

My understanding is that the Xen default policy has been to scrub RAM only when allocating it to a VM or to some other internal Xen purpose.

Also this would not be a dom0 feature, it would need to be a Xen feature (it's possible there's already a policy flag for this?).

Deallocated VM RAM is released to and managed by Xen, not dom0. Dom0 itself is just a special (often management) VM.

B

[renehoj]

If dom0 just is a VM, does that mean the scrub-domheap option also apply to releasing the pages used by dom0?

[brendanhoar]

If dom0 just is a VM, does that mean the scrub-domheap option also apply to releasing the pages used by dom0?

Yes, when any domain is returning memory to Xen, including dom0.

I hadn't noticed the addition of the scrub-domheap option in 2019, thanks for pointing that out. Presumably this prevents leaks of vm secrets into Xen controlled memory.

My understanding still is that Xen already scrubs memory before giving it to a VM, perhaps this option is meant to ensure VM secrets don't spend any more time in RAM than the lifetime of the VM (system guaranteed full-scrubbed of secrets on VM shutdown).

[renehoj]

My understanding still is that Xen already scrubs memory before giving it to a VM, perhaps this option is meant to ensure VM secrets don't spend any more time in RAM than the lifetime of the VM (system guaranteed full-scrubbed of secrets on VM shutdown).

I read that scrubbing is done during the cpu idle time across all cores, if you have free memory and your cpu never idles, I don't think there is any guarantee the pages are scrubbed until the memory is used again.

As I understand it, scrub-domheap always scrub the memory when it's released, it doesn't wait for the cpu to be idle.

[adrelanos]

Wipe RAM on shutdown is now fully functional using the ram-wipe package, which I mentioned above.

Notes:

• dracut is deprecated by upstream.
• dracut-ng is the successor of dracut.
• Prior mentioned dracut issues can be patched and have been fixed in dracut-ng. (Updated above links to the related ticket / pull request.)
• Most Linux distributions are changing from dracut to dracut-ng.

Therefore ram-wipe has a bright future.

• If Debian implements dracut: Consider switching to the fork dracut-ng in time for Debian 13 (trixie) and above, then ram-wipe will work perfectly in Kicksecure once it has been ported to the next stable release of Debian 13, once available.
• Should be easily re-usable by Fedora based distributions such as Qubes dom0.
  • That is once Qubes dom0 comes with a recent enough Fedora dom0 version that comes with dracut-ng by default.

[tasket]

As I understand it, scrub-domheap always scrub the memory when it's released, it doesn't wait for the cpu to be idle.

I activated it several days ago on my main system using scrub-domheap=on in grub.

First impressions: No difference in system responsiveness or stability.

Going forward, using Xen scrub-domheap has advantages over other approaches in that its simple to enable and is constantly active. It could result in better opsec against some attacks, including coldboot. The question I have at this point is whether the heap memory category accounts for all memory allocated to VMs; my reading so far indicates that it does ("heap" = memory pool that can be allocated to domains like a programming language allocates heap for variables).

[adrelanos]

scrub-domheap has a dedicated ticket, which got closed. -> #4488 (comment)

[tasket]

It would help if the issue defined a threat model and security goal; currently it just states that a function should occur. In "2." it considers the threat from #4488 a problem, in passing. IMO, a shutdown function has limited opportunity to help, and its also not clear whether this is dom0 shutdown you're talking about.

[adrelanos]

I supposed this ticket to be primarily about dom0.

Threat:

• https://www.usenix.org/legacy/event/sec08/tech/full_papers/halderman/halderman.pdf
• https://www.youtube.com/watch?v=JDaicPIgn9U <-- What was demonstrated in that video should be mitigated.
• https://www.kicksecure.com/wiki/Cold_Boot_Attack_Defense

Security goals for this ticket:

• wipe RAM by default on shut down
• avoid contents such as full disk encryption (FDE) keys or other sensitive data can be extracted from RAM using cold boot attacks

ram-wipe relies on the operating system during shutdown to shutdown all processes, VMs, umount all disks, etc. ram-wipe then runs its first pass at the end of the shutdown process to fill up all RAM which will hopefully overwrite everything of importance. ram-wipe will then kexec into the same kernel with a special kernel parameter to make sure even the old kernel gets wiped from RAM and does another pass of RAM wiping.

I don't know about Qubes / Xen specifics.

Future work:

• panic key (as described in Emergency Crash Script to Protect Host FDE)
• usb removal of specific device leading to fast emergency shutdown (panic button / panic shutdown / BusKill - The USB kill cord for your laptop)

No tickets exist yet.

[renehoj]

I don't know about Qubes / Xen specifics.

Design Session B: Downstream Working Group from the Xen Project Summit 2024 is worth watching, they spend some time discussing synchronous memory scrubbing when a VM is destroyed.

https://www.youtube.com/watch?v=6n3vgCciE1I

Can't the attacker hard reset the system, and bypass any protection that rely on the system going through a controlled shutdown sequence?

If so, I think it misleading to label it as Cold Boot Attack protection, if an easy to execute low-tech bypass exists.

[adrelanos]

discussing synchronous memory scrubbing when a VM is destroyed.

VM RAM scrubbing is probably nice to have but might not required in context of ram-wipe, which wipes RAM during shutdown. Only a clean system shutdown (which includes shutting down all VMs) should be needed. This is because I haven't found a way to wipe RAM while it is still assigned to / used by something else (that is without getting into super complicated code and/or crashing the kernel).

If RAM was free'd and is available for allocation, then ram-wipe should work. But in case there was any reserved RAM that cannot be allocated excluding the obvious exceptions (kernel, dracut, ...), that would be an issue for ram-wipe's first pass of wiping RAM.

If there was any reserved RAM, then ram-wipe's second pass of wiping RAM which is kexecing to another Linux kernel would solve this.

Can't the attacker hard reset the system, and bypass any protection that rely on the system going through a controlled shutdown sequence?

Yes. So for ram-wipe or any alternative to be useful, it needs to be triggered. Potential triggers are A) shutdown; B) device removal; or C) a panic key.

If A) was implemented, this would be sufficient to implement this ticket Wipe RAM on shutdown to be on par with / implement [META] Tails-like functionality in Qubes (#2024).

If so, I think it misleading to label it as Cold Boot Attack protection, if an easy to execute low-tech bypass exists.

I predict this could be a long discussion but none of it would make any progress towards getting this ticket implemented. I was starting to write a reply but then I thought this is the wrong place to discuss this. Quote from the Qubes website:

The issue tracker is not a discussion forum

It is a tool for users to report and developers to make progress towards fixing bugs and implementing features.

Debating the framing / terminology what Cold Boot Attack protection means would not make any progress towards implementing this ticket. Therefore, I think it is off-topic and should be moved elsewhere.