Memory exhaustion on AIX 7.2 on Rubygems >= 4.0

[mhashizume]

Describe the problem as clearly as you can

The SafeMarshal feature of Rubygems >= 4.0.0 causes memory exhaustion on AIX 7.2 when adding sources.

With Rubygems 4.0.3 on Ruby 4.0.1:

[0] [AIX] root@aix72-2-pix:~ # /opt/puppetlabs/puppet/bin/gem source --add $internal_artifactory_repo
[FATAL] failed to allocate memory

Did you try upgrading RubyGems?

This issue also seems to affect RubyGems upgrades:

[0] [AIX] root@aix72-2-pix:~ # /opt/puppetlabs/puppet/bin/gem update --system
[FATAL] failed to allocate memory

Post steps to reproduce the problem

The simplest reproduction is to take an AIX 7.2 machine and attempt to gem source --add.

@joshcooper also created this reproduction script:

require 'rubygems'
require 'rubygems/safe_marshal'
require 'objspace'

def safe_marshal_element_classes
  result = []
  Gem::SafeMarshal::Elements.constants(false).each do |const|
    c = Gem::SafeMarshal::Elements.const_get(const)
    result << c if c.is_a?(Class)
  rescue NameError
    # ignore
  end
  result
end

def histogram_counts(limit: 50, min_count: 1)
  klasses = safe_marshal_element_classes
  counts  = klasses.map do |k|
    n = 0
    ObjectSpace.each_object(k) { n += 1 }
    [k, n]
  end

  counts.select! { |(_, n)| n >= min_count }
  counts.sort_by! { |(_, n)| -n }
  counts.first(limit)
end

io = Gem.read_binary(ARGV[0])
data = Gem::SafeMarshal.safe_load(io)
#tuples = Gem::NameTuple.from_list(rval)
histogram_counts(limit: 25).each do |klass, n|
  puts "#{klass}: #{n}"
end

Even when attempting to load specs from rubygems.org, we see the same issue:

[0] [AIX] root@aix72-2-pix:~ # curl -s --output rubygems-specs.4.8.gz https://rubygems.org/specs.4.8.gz
[0] [AIX] root@aix72-2-pix:~ # gunzip rubygems-specs.4.8.gz
[0] [AIX] root@aix72-2-pix:~ # /opt/puppetlabs/puppet/bin/ruby spec_histogram.rb rubygems-specs.4.8
[FATAL] failed to allocate memory

Which command did you run?

gem source --add

What were you expecting to happen?

Rubygems adds another source for gems.

What actually happened?

The command failed with [FATAL] failed to allocate memory.

Run gem env and paste the output below

[0] [AIX] root@aix72-2-pix:~ # /opt/puppetlabs/puppet/bin/gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 4.0.3
  - RUBY VERSION: 4.0.1 (2026-01-13 patchlevel 0) [powerpc-aix7.2.0.0]
  - INSTALLATION DIRECTORY: /opt/puppetlabs/puppet/lib/ruby/gems/4.0.0
  - USER INSTALLATION DIRECTORY: /root/.local/share/gem/ruby/4.0.0
  - CREDENTIALS FILE: /root/.local/share/gem/credentials
  - RUBY EXECUTABLE: /opt/puppetlabs/puppet/bin/ruby
  - GIT EXECUTABLE:
  - EXECUTABLE DIRECTORY: /opt/puppetlabs/puppet/bin
  - SPEC CACHE DIRECTORY: /root/.cache/gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /opt/puppetlabs/puppet/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - powerpc-aix-7
  - GEM PATHS:
     - /opt/puppetlabs/puppet/lib/ruby/gems/4.0.0
     - /root/.local/share/gem/ruby/4.0.0
     - /opt/puppetlabs/puppet/lib/ruby/vendor_gems
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     -
     -
     - /usr/bin
     - /etc
     - /usr/sbin

[hsbt]

Thanks for your report. Is it only happend with AIX? I can't reproduce it with macOS.

[hsbt]

I understood it caused by 32bit platform that is powerpc-aix7.2.0.0.

[mhashizume]

I was speaking with my colleague @johcooper about this. He noticed that, even when a spec is a String, SafeMarshal will wrap an IO around it here:

https://github.com/ruby/ruby/blob/9ecac8d4690ca4cf4e575bc2fdd31c99aa82e754/lib/rubygems/safe_marshal.rb#L65

He suggested that memory consumption could be improved if Strings and IO are treated differently with something like:

io = if input.is_a?(String)
       StringIO.new(input, "r")
     else
       input
     end
root = Reader.new(io).read!

[simi]

@mhashizume can you share the computer specs? What is the final goal - add source and install gem? If so, maybe you can try #9314 if the source is using Compact Index API (most of the modern gem servers do, you can check by trying /versions endpoint like rubygems.org/versions).

I was able to reproduce with docker, 32bit and small memory allocated.

#!/usr/bin/env bash
# Reproduce https://github.com/ruby/rubygems/issues/9368
# SafeMarshal OOMs on 32-bit loading rubygems.org specs.4.8
set -euo pipefail

DIR="$(cd "$(dirname "$0")" && pwd)/repro"
SPECS="$DIR/rubygems-specs.4.8"
IMG="ruby:4.0-slim"
RUBY_SAFE='require "rubygems"; require "rubygems/safe_marshal"; io = Gem.read_binary("/data/rubygems-specs.4.8"); Gem::SafeMarshal.safe_load(io); rss = File.read("/proc/self/statm").split[1].to_i * 4 / 1024; puts "SafeMarshal OK: RSS=#{rss}MB"'
RUBY_MARSHAL='io = File.binread("/data/rubygems-specs.4.8"); Marshal.load(io); rss = File.read("/proc/self/statm").split[1].to_i * 4 / 1024; puts "Marshal OK: RSS=#{rss}MB"'

[ -f "$SPECS" ] || { mkdir -p "$DIR" && curl -s https://rubygems.org/specs.4.8.gz | gunzip > "$SPECS"; }

set +e;

echo "SafeMarshal with 128MB limit (should OOM):"
docker run --rm --platform linux/386 --memory=128m -v "$DIR:/data:ro" $IMG ruby -e "$RUBY_SAFE" 2>&1; echo "exit: $?"; set -e

echo ""
echo "Marshal.load with 128MB limit (should succeed):"
docker run --rm --platform linux/386 --memory=128m -v "$DIR:/data:ro" $IMG ruby -e "$RUBY_MARSHAL" 2>&1

echo ""
echo "SafeMarshal with 512MB limit (memory measurement):"
docker run --rm --platform linux/386 --memory=512m -v "$DIR:/data:ro" $IMG ruby -e "$RUBY_SAFE" 2>&1

[retro@retro2  rubygems (master %=)]❤ ./reproduce-9368.sh 
SafeMarshal with 128MB limit (should OOM):
exit: 137

Marshal.load with 128MB limit (should succeed):
Marshal OK: RSS=93MB

SafeMarshal with 512MB limit (memory measurement):
SafeMarshal OK: RSS=379MB

Can you confirm that's the same issue? Seems SafeMarshal is kind of hungry for memory 🍔 .

[mhashizume]

@simi Thanks for taking a look at this. This is on an AIX 7.2 machine with 8 GB of memory and running on POWER9 architecture. The final goal is to use an internal gem repository instead of reaching out to rubygems.org.

I tried your PR #9314 and still get the same issue when trying to gem source --add both our internal repo and gem.coop.