Proposal: out-of-tree encryption backends

[ndrpnt]

This proposal explores ways to enable out-of-tree encryption backends in SOPS. The goal is to make it easy to add new encryption backends, including private ones.

Other components that could benefit from out-of-tree extensibility, such as stores and publishing backends, are out of scope.

Related PRs:

• Add support for OpenStack Barbican #683
• Add YC KMS provider support #1097
• Add Cloud.ru KMS support #1718
• ocikms: Oracle Cloud KMS support #1226
• Add support for OVH KMS in SOPS #1906
• feat(hckms): Add HuaweiCloud KMS support #2001
• [Feature] Support Tencent Cloud KMS #1984

As stated multiple times, maintainers are open to a plugin system for encryption backends. This avoids adding dependencies and responsibilities to core SOPS. In the future, moving even core providers out-of-tree would make evolving (and forking) them a lighter process.

I've explored two approaches, and I'm looking for maintainer feedback before investing more time.

Option 1: build on the existing key service infrastructure

This approach was discussed in #481, but I don't think it went anywhere.

I Hacked together sops-generic-keyservice, a proof-of-concept key service server compatible with Scaleway Key Manager and OVH Key Management Service. For a lack of better alternative, it currently piggybacks on AWS' --kms flag to pass arbitrary configuration to the KMS.

I believe this approach can get us far without requiring major/breaking changes in SOPS. Some backward-compatible improvements are needed to make "Generic KMS" a first class backend:

1. Add support for generic_key in the key service API:

   message Key {
     oneof key_type {
       ...
       GenericKey generic_key = 7; // or google.protobuf.Struct ?
     }
   }
   
   message GenericKey {
     // Allow key service servers to filter only the requests they can handle.
     string kms_id = 1;
   
     map<string, google.protobuf.Value> configuration = 2;
   }

2. Add the corresponding flags, e.g --generic-kms '{"kms_id": "mykms", "configuration": {"foo": "bar"}}'. Passing a JSON object is inelegant but straightforward. One can also look at how Terraform handles complex dynamic values through flags and environment variables.
3. Add the corresponding configuration options

   creation_rules:
     - generic_kms:
         - kms_id: mykms
           configuration:
             foo: bar

Option 2: build a plugin system

A fully fledged plugin system:

• Offers a sleeker user experience, without the need to start services and export environment variables before running SOPS.
• Offers a cleaner API than the keyservice one, purpose-built for providers to implement.
• Lays the groundwork to make other components pluggable, like stores and publishing backends.
• Requires more upfront design, and more coding effort.

This option is not exclusive with option 1: we may rely on a generic key service to interate on the KMS plugin API without affecting core SOPS. Once the API is stable enough, and providers exist for multiple backends (including core ones), we can merge it into SOPS.

I did not invest time in pursuing that option. Maybe PluginRPC would be a good fit, but it leaves installing the executable up to the user.

Next Steps

I'd appreciate feedback from maintainers on pursuing option 1. I can work on a draft PR if this direction sounds reasonable.

[felixfontein]

From a user point of view, Option 2 is easier to use, and also to set-up. Especially if you need to use multiple different encryption backends at once. But I haven't spent that much time thinking about it so far, so this is just a first reaction :)

CC @getsops/maintainers any opinions?

[felixfontein]

Maybe PluginRPC would be a good fit, but it leaves installing the executable up to the user.

PluginRPC sounds (on a first glance) similar to what I was thinking about when I thought about plugins before. I'm not sure whether that specific project is a good idea though, it doesn't seem to be very actively developed, doesn't seem to have a big community, and might require a CLA (pluginrpc/pluginrpc#1 (comment) seems to indicate it, I didn't found something elsewhere).

It might also be a good idea to look closer at what age is doing - it also supports plugins, and its maintainer has a lot of experience with Go and security, so what they use can't be a too bad choice.