Bug with openssl_private_decrypt when using RSA-OAEP with SHA-256 from JS

[pashkovsky95]

Description

Hello, I have a problem with RSA-OAEP-256 when decrypting on the server (php) that I receive from the client (js, browser).

$config = [
  "digest_alg" => "sha256",
  "private_key_bits" => 4096,
  "private_key_type" => OPENSSL_KEYTYPE_RSA,
];

...

return [
  'keyId' => $key->id,
  'publicKey' => $publicKey
];

Client receives RSA public key from server and generates AES-GCM-256 key.

const publicKey = await window.crypto.subtle.importKey(
    'spki',
    await pemToArrayBuffer(serverPublicKey),
    { name: 'RSA-OAEP', hash: 'SHA-256' },
    true,
    ['encrypt']
);

const aesKey = await window.crypto.subtle.generateKey(
      { name: 'AES-GCM', length: 256 },
      true,
      ['encrypt', 'decrypt']
);

RSA public key encrypts AES key. AES encrypts data. Client sends AES-OAEP encrypted data with SHA-256 and AES encrypted key.

const rawAesKey = await window.crypto.subtle.exportKey('raw', aesKey);

const encryptedAesKey = await window.crypto.subtle.encrypt(
  { name: 'RSA-OAEP' },
  publicKey,
  rawAesKey
);

const data = new TextEncoder().encode(JSON.stringify({
  login: 'test'
}));

const iv = window.crypto.getRandomValues(new Uint8Array(12));
const encryptedData = await window.crypto.subtle.encrypt(
  { name: 'AES-GCM', iv },
  aesKey,
  data
);

await axios.post(`/login`, JSON.stringify({
  keyId: serverPublicKeyId,
  key: arrayBufferToBase64(encryptedAesKey),
  iv: arrayBufferToBase64(iv),
  encryptedData: arrayBufferToBase64(encryptedData)
})
...

Server receives data and decrypts AES key with RSA private key where php throws error. openssl_private_decrypt function intercepts and openssl_error_string returns error: error:02000079:rsa routines::oaep decoding error. If you encrypt and decrypt RSA-OAEP data with SHA-256 only on the server, everything works fine.

openssl_private_decrypt($encryptedAesKey, $aesKey, $privateKey, OPENSSL_PKCS1_OAEP_PADDING);

if($aesKey === false) {
  throw new Exception("RSA decryption failed: " . openssl_error_string());
}

I think it happens because openssl_decrypt with OAEP has SHA-1 hash by default when receiving encrypted data from external client like in js. If I change the hash on OAEP from SHA-256 to SHA-1 in js, the script runs perfectly and without errors, but it is not safe for production.

const publicKey = await window.crypto.subtle.importKey(
    'spki',
    await dispatch('pemToArrayBuffer', serverPublicKey),
    { name: 'RSA-OAEP', hash: 'SHA-1' },
    true,
    ['encrypt']
);

Maybe I read the documentation poorly or does php actually have this bug? If it is a bug in PHP, maybe it should be added the optional argument ?string hash = null in openssl_private_decrypt to force the current hash to be used for decryption?

openssl_private_decrypt(
  string $data,
  #[\SensitiveParameter] string &$decrypted_data,
  #[\SensitiveParameter] OpenSSLAsymmetricKey|OpenSSLCertificate|array|string $private_key,
  int $padding = OPENSSL_PKCS1_PADDING,
  ?string hash = null
): bool

// openssl_private_decrypt($encryptedAesKey, $aesKey, $privateKey, OPENSSL_PKCS1_OAEP_PADDING, 'sha256');

Temporary solution

Of course, using SHA-1 is a bad idea, but I found a temporary solution. The phpseclib library allows you to bypass this limitation in php and force a hash before decryption, which works fine with SHA-256.

$privateKey = $privateKey->withHash('sha256');
$aesKey = $privateKey->decrypt($encryptedAesKey);

I think this library solves this problem, but it is a very important bug in php that should be fixed.

OpenSSL

OpenSSL 3.2.2 4 Jun 2024 (Library: OpenSSL 3.2.2 4 Jun 2024)

PHP Version

PHP 8.4.10 (cli) (built: Jul  2 2025 02:22:42) (NTS gcc x86_64)

Operating System

Linux