v4.0.0 KMS Keyrings SHOULD explicitly error if non-ARN OnDecrypt

[benkehoe]

Problem:

In previous versions, you could use KMS key ids in addition to ARNs. I believe this is still the intended behavior, but it appears to be broken. You can reproduce it with the aws_kms_keyring_example.py script from the examples. Side note, on line 58 for some reason the example sets the region to us-west-2, which will of course break things if your keys aren't in that region. The example should be changed to remove that. The following output assumes that minor problem is fixed:

First, with the key ARN:

>>> import aws_kms_keyring_example
>>> key_arn = "arn:aws:kms:us-east-2:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
>>> aws_kms_keyring_example.encrypt_and_decrypt_with_keyring(key_arn)
./.venv/lib/python3.12/site-packages/aws_cryptography_primitives/internaldafny/extern/Signature.py:50: CryptographyDeprecationWarning: Curve argument must be an instance of an EllipticCurve class. Did you pass a class by mistake? This will be an exception in a future version of cryptography.
  private_key = ec.generate_private_key(

Some sort of minor problem there that should also be fixed, but not a blocking problem (yet). Now, with the key id:

>>> key_id = key_arn.split("/")[1]
>>> aws_kms_keyring_example.encrypt_and_decrypt_with_keyring(key_id)
Error on closing
Traceback (most recent call last):
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/materials_managers/mpl/cmm.py", line 124, in decrypt_materials
    mpl_output: 'MPL_DecryptMaterialsOutput' = self.mpl_cmm.decrypt_materials(mpl_input)
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./.venv/lib/python3.12/site-packages/aws_cryptographic_material_providers/smithygenerated/aws_cryptography_materialproviders/references.py", line 566, in decrypt_materials
    raise aws_cryptography_materialproviders_deserialize_error(
aws_cryptographic_material_providers.smithygenerated.aws_cryptography_materialproviders.errors.AwsCryptographicMaterialProvidersException: Unable to decrypt data key: No Encrypted Data Keys found to match.
 Expected:
KeyProviderId: aws-kms, KeyProviderInfo: arn:aws:kms:us-east-2:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/__init__.py", line 218, in decrypt
    plaintext = decryptor.read()
                ^^^^^^^^^^^^^^^^
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/streaming_client.py", line 342, in read
    self._prep_message()
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/streaming_client.py", line 941, in _prep_message
    self._header, self.header_auth = self._read_header()
                                     ^^^^^^^^^^^^^^^^^^^
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/streaming_client.py", line 1045, in _read_header
    decryption_materials = self.config.materials_manager.decrypt_materials(request=decrypt_materials_request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/materials_managers/mpl/cmm.py", line 129, in decrypt_materials
    raise AWSEncryptionSDKClientError(mpl_exception)
aws_encryption_sdk.exceptions.AWSEncryptionSDKClientError: Unable to decrypt data key: No Encrypted Data Keys found to match.
 Expected:
KeyProviderId: aws-kms, KeyProviderInfo: arn:aws:kms:us-east-2:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/streaming_client.py", line 310, in __exit__
    self.close()
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/streaming_client.py", line 1233, in close
    raise SerializationError("Footer not read")
aws_encryption_sdk.exceptions.SerializationError: Footer not read
Traceback (most recent call last):
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/materials_managers/mpl/cmm.py", line 124, in decrypt_materials
    mpl_output: 'MPL_DecryptMaterialsOutput' = self.mpl_cmm.decrypt_materials(mpl_input)
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./.venv/lib/python3.12/site-packages/aws_cryptographic_material_providers/smithygenerated/aws_cryptography_materialproviders/references.py", line 566, in decrypt_materials
    raise aws_cryptography_materialproviders_deserialize_error(
aws_cryptographic_material_providers.smithygenerated.aws_cryptography_materialproviders.errors.AwsCryptographicMaterialProvidersException: Unable to decrypt data key: No Encrypted Data Keys found to match.
 Expected:
KeyProviderId: aws-kms, KeyProviderInfo: arn:aws:kms:us-east-2:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./aws_kms_keyring_example.py", line 101, in encrypt_and_decrypt_with_keyring
    plaintext_bytes, _ = client.decrypt(
                         ^^^^^^^^^^^^^^^
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/__init__.py", line 218, in decrypt
    plaintext = decryptor.read()
                ^^^^^^^^^^^^^^^^
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/streaming_client.py", line 342, in read
    self._prep_message()
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/streaming_client.py", line 941, in _prep_message
    self._header, self.header_auth = self._read_header()
                                     ^^^^^^^^^^^^^^^^^^^
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/streaming_client.py", line 1045, in _read_header
    decryption_materials = self.config.materials_manager.decrypt_materials(request=decrypt_materials_request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./.venv/lib/python3.12/site-packages/aws_encryption_sdk/materials_managers/mpl/cmm.py", line 129, in decrypt_materials
    raise AWSEncryptionSDKClientError(mpl_exception)
aws_encryption_sdk.exceptions.AWSEncryptionSDKClientError: Unable to decrypt data key: No Encrypted Data Keys found to match.
 Expected:
KeyProviderId: aws-kms, KeyProviderInfo: arn:aws:kms:us-east-2:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

encrypt() succeeds, but decrypt() fails with Unable to decrypt data key: No Encrypted Data Keys found to match.

I discovered this when upgrading with code that used the key id. At the very least encrypt() should raise an error, it's much worse to end up with a ciphertext you can't decrypt.

Side note: it'd be nice to have a utility that can output the parsed message fields. The structure is documented but it'd be nice to have that provided. The empty string after "Expected:" in the error leads me to think something might be missing from the message, but I'm too lazy to parse it myself. And such a utility would be useful for cataloging, as an example, what keys are in use by a system using the Encryption SDK, without needing to decrypt all the values.

[texastony]

Howdy @benkehoe ,

This ciphertext CAN be decrypted;
though we can improve the decryption error message.

At Encryption,
the library has used KMS to convert the Key ID into a full KMS ARN;
the KMS APIs responses include the full KMS ARN,
even if invoked with only the Key ID or alias.

The full Key ID is persisted as part of the Encrypted Data Key;
this is critical, as Key IDs do not include the region or account ID,
which are very helpful to ensure the encrypted data key can be decrypted correctly.

(The AWS SDK and KMS, if called without a region or account ID, use the principal of the caller to fill in those fields; but the serialized Encrypted Data Key MUST be deterministic, as compared to determined by the caller's context.)

To decrypt this cipher-text,
pass the full KMS ARN.

I have attached a code example that follows your instructions,
including a failing decrypt request,
followed by a successful decrypt request.

This demonstrates that the cipher-text is decryptable.

However, you make an excellent suggestion;
we should improve the error messaging in this edge case.

At Decrypt, if the kms_key_id parameter is not a fully resolved ARN,
an explicit error should be thrown,
as it will never match an encrypted data key.

(Note: I had to name the file a .log instead of .py because GitHub will not let me upload a .py or .text file)
ESDK_Python_GHI_743.log

[benkehoe]

I'm still confused, though. Using key IDs for both encrypt and decrypt used to work with the Encryption SDK. Is there a good reason it shouldn't work now? The encryption process forms up a full ARN from the key ID (for the reasons you detail). Why can't the decryption process also form an ARN from the input key ID in the same way, which would then match (or not, if the principal has a different account or region) the key in the incoming message? It feels very strange for key IDs to be allowed on one end but not the other.