Custom ConfigProvider

[Shiv-hcr]

Has anyone made a ConfigProvider implementation which calls out to a 3rd party (like Azure Key Vault or AWS Secrets Manager)?
I've made a working Azure Key Vault service but am struggling to design a ConfigProvider from it. Any recommendations or examples would be really appreciated!

[wesleysmyth]

Here's how to build a ConfigProvider that fetches secrets from Azure Key Vault or AWS Secrets Manager:

Azure Key Vault ConfigProvider

import { ConfigProvider, Config, Effect, Layer, Redacted } from "effect";
import { SecretClient } from "@azure/keyvault-secrets";
import { DefaultAzureCredential } from "@azure/identity";

const makeAzureKeyVaultProvider = (vaultUrl: string): ConfigProvider.ConfigProvider => {
  const client = new SecretClient(vaultUrl, new DefaultAzureCredential());

  return ConfigProvider.fromFlat(
    ConfigProvider.makeFlat({
      load: (path) =>
        // Convert config path to Key Vault secret name
        // e.g., ["database", "password"] -> "database--password"
        Effect.tryPromise({
          try: async () => {
            const secretName = path.join("--");
            const secret = await client.getSecret(secretName);
            return [secret.value ?? ""];
          },
          catch: (error) =>
            // Return empty array for missing keys (falls through to next provider)
            [],
        }).pipe(Effect.orElseSucceed(() => [])),
      enumerateChildren: (path) =>
        Effect.succeed(new Set<string>()),
      patch: ConfigProvider.patch.empty,
    })
  );
};

// Usage
const vaultProvider = makeAzureKeyVaultProvider(
  "https://my-vault.vault.azure.net"
);

// Compose with env vars as fallback
const provider = ConfigProvider.orElse(vaultProvider, () =>
  ConfigProvider.fromEnv()
);

const ConfigLive = Layer.setConfigProvider(provider);

AWS Secrets Manager ConfigProvider

import { ConfigProvider, Effect } from "effect";
import {
  SecretsManagerClient,
  GetSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";

const makeAwsSecretsProvider = (
  region: string
): ConfigProvider.ConfigProvider => {
  const client = new SecretsManagerClient({ region });

  // Cache to avoid repeated API calls for the same secret
  const cache = new Map<string, string>();

  return ConfigProvider.fromFlat(
    ConfigProvider.makeFlat({
      load: (path) =>
        Effect.gen(function* () {
          const secretName = path.join("/");

          if (cache.has(secretName)) {
            return [cache.get(secretName)!];
          }

          try {
            const result = yield* Effect.tryPromise(() =>
              client.send(
                new GetSecretValueCommand({ SecretId: secretName })
              )
            );

            if (result.SecretString) {
              // If the secret is JSON, try to extract nested keys
              try {
                const parsed = JSON.parse(result.SecretString);
                const lastKey = path[path.length - 1];
                if (typeof parsed === "object" && lastKey in parsed) {
                  const value = String(parsed[lastKey]);
                  cache.set(secretName, value);
                  return [value];
                }
              } catch {
                // Not JSON, use as-is
              }
              cache.set(secretName, result.SecretString);
              return [result.SecretString];
            }
            return [];
          } catch {
            return [];
          }
        }),
      enumerateChildren: (path) =>
        Effect.succeed(new Set<string>()),
      patch: ConfigProvider.patch.empty,
    })
  );
};

Putting it together

import { Config, Effect, Redacted } from "effect";

// Define your config schema
const DatabaseConfig = Config.all({
  host: Config.string("host").pipe(Config.nested("database")),
  port: Config.integer("port").pipe(Config.nested("database")),
  password: Config.redacted("password").pipe(Config.nested("database")),
});

// Compose providers: Key Vault -> Env -> defaults
const provider = ConfigProvider.orElse(
  makeAzureKeyVaultProvider("https://my-vault.vault.azure.net"),
  () =>
    ConfigProvider.orElse(ConfigProvider.fromEnv(), () =>
      ConfigProvider.fromMap(
        new Map([
          ["database.host", "localhost"],
          ["database.port", "5432"],
        ])
      )
    )
);

const program = Effect.gen(function* () {
  const dbConfig = yield* DatabaseConfig;
  console.log("Host:", dbConfig.host);
  console.log("Port:", dbConfig.port);
  // Redacted values are protected from accidental logging
  console.log("Password:", dbConfig.password); // Redacted(<redacted>)
  console.log("Password value:", Redacted.value(dbConfig.password));
});

const ConfigLive = Layer.setConfigProvider(provider);

Effect.runPromise(program.pipe(Effect.provide(ConfigLive)));

Key design decisions

1. ConfigProvider.fromFlat with makeFlat is the lowest-level API — gives you full control over how config paths map to secret names
2. ConfigProvider.orElse chains providers so you can fall through: secrets manager -> env vars -> defaults
3. Use Config.redacted for sensitive values — Effect prevents accidental logging
4. Cache aggressively — secrets manager API calls are slow and rate-limited
5. Map path separators to something your secrets manager supports (Key Vault uses --, AWS uses /)

The load function receives the path as a string[] and must return string[] — an empty array means "not found" and triggers fallthrough to the next provider in the chain.