Implement provider interface to support GOST TLS 1.3

[VladGud]

Extension of gost-engine (provider) functionality to enable GOST TLS 1.3 support

Ref: #388
Ref: #167

The proposed changes introduce a minimal set of functionality required to enable GOST TLS 1.3 using the provider interface, along with the corresponding updates to the OpenSSL project (included as patch files).

The following provider modules were implemented or modified:

• provider-base (already existed, modified)
• provider-keymgmt (implemented from scratch)
• provider-signature (implemented from scratch)
• provider-keyexchange (implemented from scratch)
• provider-encoder (implemented from scratch)
• provider-decoder (implemented from scratch)
• provider-cipher (already existed, modified)

Implementation was focused on the functionality necessary for running the TLS 1.3 protocol with GOST. Support for TLS versions below 1.3 was not considered or implemented.

Detailed changelog:

• Refactored the original engine source code and extracted an internal_* interface to allow internal engine functions to be reused in the provider implementation (see gost_lcl.h);
• Implemented provider-keymgmt to support key generation and loading for the following algorithms: gost2001, gost2001dh, gost2012_256, gost2012_512.
  Note: Key import and export are not implemented, as there was no requirement to exchange keys with other providers for TLS 1.3;
• Implemented provider-encoder and provider-decoder to support encoding and decoding of keys in the following formats: text (encoding only), DER, PEM;
• Implemented provider-signature with digest mode support;
• Implemented provider-keyexchange for the ECDHE algorithm; other algorithms are not required for TLS 1.3. This interface may later be extended to support pkey_gost_ec_derive;
• Added TLSTREE-based key derivation support for Magma and Kuznyechik ciphers in MGM mode (see RFC9367, 4.1.2. TLSTREE);
• Extended provider-cipher with support for setting the following parameters:
  • OSSL_CIPHER_PARAM_TLSTREE (sets the sequence number),
  • OSSL_CIPHER_PARAM_TLSTREE_MODE (sets TLSTREE mode: strong or light — see RFC9367, 4.1.2. TLSTREE),
  • and retrieving OSSL_CIPHER_PARAM_AEAD (whether the cipher is AEAD);
• Defined the required GOST TLS 1.3 capabilities in the provider: "TLS-GROUP" and "TLS-SIGALG";
• Added changes to the OpenSSL codebase to enable TLS 1.3 with Russian cryptographic algorithms. These changes are included in the PR as patches;
• Existing tests were modified to validate both engine and provider functionality; additional tests for TLS 1.3 implementation were added.

Notes on GOST TLS 1.3 implementation

Sigalgs

To ensure compatibility with openssl CLI commands for both engine and provider modes, the original key generation and signature algorithm names were preserved:
SN_id_GostR3410_2012_256, SN_id_GostR3410_2012_512, and others.

When registering TLS 1.3 SIGALGs via the "TLS-SIGALG" capability, 7 sigalgs are registered:

• With key algorithm SN_id_GostR3410_2012_256:

  • gostr34102012_256a
  • gostr34102012_256b
  • gostr34102012_256c
  • gostr34102012_256d

• With key algorithm SN_id_GostR3410_2012_512:

  • gostr34102012_512a
  • gostr34102012_512b
  • gostr34102012_512c

It appears that the current OpenSSL implementation does not support multiple SIGALGs sharing the same signature algorithm. OpenSSL lacks an interface to differentiate sigalgs based on key parameters, making it impossible to map a sigalg to a specific key paramset.

Therefore, in the current implementation:

1. A TLS client or server may be initialized with a key that does not match the registered SIGALG (e.g., differing paramsets).
2. The TLS server selects the first matching algorithm from the ClientHello "signature_algorithms" extension (gostr34102012_256*), regardless of the server key's paramset.

Resolving this limitation is planned for a second stage of development. We assume that extending the provider-keymgmt interface (in coordination with OpenSSL maintainers) will be required.

Testing

The GOST TLS 1.3 implementation was tested against the Infotecs test servers:
https://infotecs.ru/products/tls-test.php/
See test: test/05-tls13.t.

OpenSSL version

All changes are based on OpenSSL version 3.4.2.