Use Frostmoln Secrets with External Secrets Operator (ESO)

External Secrets Operator (ESO) syncs secrets from an external store into native Kubernetes Secret objects. Its generic webhook provider can read from any HTTP API — including Frostmoln Secrets — so you can keep your secrets managed in Frostmoln and have ESO mirror them into your clusters, with no ESO-side code and no custom provider.

This works on Frostmoln managed Kubernetes, where the platform installs ESO for you as a cluster add-on, and on any other Kubernetes cluster with internet egress that you run yourself — bring-your-own or on-prem — because ESO reaches the public API edge (api.frostmoln.cloud) over HTTPS.

What this does

You store a secret in Frostmoln once. On each refresh interval, ESO calls the Frostmoln lookup endpoint, reads the value, and writes it into a Kubernetes Secret your workloads consume as usual (env var or mounted file). When you rotate the value in Frostmoln, ESO picks up the new value on the next refresh.

Prerequisites

• ESO installed in the cluster.
  • Frostmoln managed Kubernetes: select the External Secrets Operator add-on (external-secrets) when you create the cluster, or add it to a running cluster. Do not install the chart yourself. See On Frostmoln managed Kubernetes.
  • Any other cluster: install the stock chart —

    helm repo add external-secrets https://charts.external-secrets.io
    helm install external-secrets external-secrets/external-secrets \
      -n external-secrets --create-namespace --set installCRDs=true

• A Frostmoln API key. Create one under Account → API Keys in the portal (or with fm account api-key create --name eso --scopes secrets:read). On Frostmoln managed Kubernetes, use Workload Identity instead.
• Your tenant ID — from fm auth whoami or the portal.

Scope the API key to secrets

When creating the key, grant it only the secrets:read scope. Frostmoln enforces API-key scopes server-side, so a key without secrets:read is refused (403) by the Secrets service at request time — scoping the key genuinely restricts what it can do, not just as a convention. Treat the key as sensitive; ESO stores it in a Kubernetes Secret.

On Frostmoln managed Kubernetes

The platform installs ESO for you, in the external-secrets namespace. Once it runs, authenticate it with Workload Identity — no API key — or with an API key as in Steps 1–3 below.

Add it when you create the cluster, or later

• When you create a cluster — select External Secrets Operator in the portal's create form, pass --addons external-secrets to fm kubernetes cluster create, or include external-secrets in addons on frostmoln_kubernetes_cluster. The cluster reports running first; ESO is applied a few minutes later, once the worker nodes have joined, and the add-on shows as Being applied until then.
• To a running cluster —
  • Portal — the cluster's Add-ons tab.
  • fm CLI —

    fm kubernetes cluster add-addons <cluster-id> --addons external-secrets

  • Terraform — add external-secrets to addons and apply again; the update adds it to the running cluster without replacing the cluster.

Once you set addons in Terraform, keep the key there: an add-on added from the portal or fm that your addons does not list is removed on the next apply.

Adding it to a running cluster is not immediate: the addition starts its own reconciliation, and ESO reaches the cluster typically within minutes; if that reconciliation cannot be started it falls back to the platform's own reconciliation period, up to an hour. On a cluster whose control plane predates the add-on, the addition is accepted but ESO never arrives; create a new cluster to use it.

Your nodes need outbound HTTPS to ghcr.io

The ESO controller image, ghcr.io/external-secrets/external-secrets, is pulled over your cluster's own outbound HTTPS when the pod starts. It is not preloaded onto your nodes. If you filter egress by hostname, allow bothghcr.io and pkg-containers.githubusercontent.com, because ghcr.io redirects image downloads there. ESO itself also needs outbound HTTPS to api.frostmoln.cloud to read your secrets. The platform does not check either; see Create a cluster.

Authenticate with Workload Identity (recommended)

With Workload Identity, ESO reads a short-lived token that the platform keeps in a Secret for you, so there is no API key to create, store or rotate. You need:

• Both add-ons on the cluster: External Secrets Operator and Workload Identity.
• A binding with secrets:read for a ServiceAccount in the namespace where the SecretStore lives:

  fm kubernetes workload-identity binding create \
    --cluster <cluster-id> --namespace default --service-account eso \
    --scope secrets:read

• That ServiceAccount, annotated to keep the token in a Secret named frostmoln-token:

  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: eso
    namespace: default
    annotations:
      frostmoln.cloud/workload-identity-secret: frostmoln-token

The Secret already carries the external-secrets.io/type: webhook label, so skip Step 1. In Step 2, use this SecretStore instead — the same URL, with the token as a bearer header:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: frostmoln
  namespace: default
spec:
  provider:
    webhook:
      url: 'https://api.frostmoln.cloud/api/v1/tenants/<TENANT_ID>/secrets/lookup?name={{ .remoteRef.key }}'
      headers:
        Authorization: 'Bearer {{ .auth.token }}'
      result:
        jsonPath: '$.secretValue'
      secrets:
        - name: auth
          secretRef:
            name: frostmoln-token
            key: token

Step 3 is unchanged. How the Secret is kept, and when it is removed: Credential in a Secret.

What differs from a self-installed ESO

• You cannot see ESO's pods or logs. external-secrets is a protected platform namespace: your kubeconfig can neither read nor write anything in it, so kubectl -n external-secrets get pods and kubectl logs there are refused. Read the status and events of your own objects instead, with kubectl -n <namespace> describe externalsecret <name> and kubectl -n <namespace> describe secretstore <name>.

• Namespaced resources only. SecretStore and ExternalSecret are available. ClusterSecretStore, ClusterExternalSecret and PushSecret are not enabled.

• ESO can read every Secret in the cluster. Like upstream ESO, the add-on's controller has cluster-wide read access to Secrets, including in your namespaces. The platform restricts what it may write, not what it may read.

• Service-account tokens are refused by design. ESO may not write Secrets in platform namespaces. In every namespace it may not create or update a Secret of type kubernetes.io/service-account-token, or one carrying the kubernetes.io/service-account.name annotation, and it may not request tokens for platform service accounts. An ExternalSecret whose target would be such a Secret — through target.template, or with creationPolicy: Merge into an existing one — fails to sync, and the error in kubectl describe externalsecret contains:

  the external-secrets add-on may not create or update a service-account token Secret

• Remove your own ESO first, in this order:

  1. With your own ESO still running, delete or orphan every ExternalSecret, as in Removing the add-on.
  2. Uninstall your ESO.
  3. Delete its CRDs.

  Keep that order. Deleting a CRD deletes every ExternalSecret of that kind, and under creationPolicy: Owner their Secrets with them; with no controller left to remove their finalizers, the CRD stays stuck in Terminating and the add-on cannot install its own. CRDs you leave in place are adopted as they stand, including any field you added.

Removing the add-on

Removing the add-on also leaves the external-secrets namespace in place. It stays reserved for the platform, so your kubeconfig still cannot use it, and adding the add-on again reuses it.

Deal with every ExternalSecret BEFORE you remove the add-on

ESO adds a cleanup finalizer to every ExternalSecret. Removing the add-on deletes the ESO controller and its webhooks but leaves the CRDs, and with the controller gone nothing removes that finalizer. Any ExternalSecret still in the cluster then cannot be deleted, and neither can a namespace that contains one.

1. List the ExternalSecrets in each namespace. A list across all namespaces is refused, so run it once per namespace:

   kubectl -n <namespace> get externalsecrets

2. For each ExternalSecret whose Secret you want to keep, do all of these, in order. Do not wait for ESO to drop its ownership on a sync: it does so only on a successful sync, and a failed fetch, an empty result under Retain, or some refresh settings skip it. <target> is the Secret named in spec.target.name, or the ExternalSecret's own name if that is unset.

   1. Set creationPolicy: Orphan first; under Owner, ESO would add the owner reference back on its next sync:

      kubectl -n <namespace> patch externalsecret <name> \
        --type=merge -p '{"spec":{"target":{"creationPolicy":"Orphan"}}}'

   2. Check whether the Secret still has an owner reference:

      kubectl -n <namespace> get secret <target> -o jsonpath='{.metadata.ownerReferences}'

   3. Only if that printed anything, remove it, then run step 2.2 again until it prints nothing. A 422 (Unprocessable Entity) from this patch is not a failure: the Secret has no owner reference to remove, so it is already safe.

      kubectl -n <namespace> patch secret <target> \
        --type=json -p '[{"op":"remove","path":"/metadata/ownerReferences"}]'

   4. Delete the ExternalSecret; the Secret stays:

      kubectl -n <namespace> delete externalsecret <name>

3. For each ExternalSecret whose Secret you do not need, delete the ExternalSecret while ESO is still running. Under the default creationPolicy: Owner its synced Secret is deleted with it, and anything still using that Secret breaks: pods that reference it fail when they next start, and if it is an image pull secret, new nodes can no longer pull the image.

4. Only then remove the add-on — from the Add-ons tab, with fm kubernetes cluster remove-addons <cluster-id> --addons external-secrets, or by dropping the key from addons in Terraform.

If you already removed the add-on

Stripping the finalizer can delete the synced Secret

Under the default creationPolicy: Owner, the synced Secret is owned by its ExternalSecret. Once the finalizer is gone and the ExternalSecret is deleted, Kubernetes garbage collection deletes the Secret too — even with deletionPolicy: Retain.

For each remaining ExternalSecret:

1. If the namespace itself is being deleted, you cannot keep the Secret: every Secret in it is deleted too, whatever its owner references. Recreate it elsewhere from Frostmoln Secrets. Otherwise, to keep its Secret, first check for an owner reference and remove it if there is one (steps 2.2 and 2.3 above).
2. Strip the finalizer:

   kubectl -n <namespace> patch externalsecret <name> \
     --type=merge -p '{"metadata":{"finalizers":null}}'

3. If the ExternalSecret is not already being deleted, delete it.

Only ExternalSecret carries this finalizer on Frostmoln clusters, because PushSecret and ClusterExternalSecret are not enabled.

Step 1 — Store the API key as a labeled Secret

API key: clusters other than Frostmoln managed Kubernetes

Steps 1 and 2 use an API key. On Frostmoln managed Kubernetes, use Workload Identity instead.

ESO's webhook provider reads the API key from a Kubernetes Secret. The Secret must carry the label external-secrets.io/type: webhook — the webhook provider refuses to read an unlabeled Secret.

Save the key to a local file that holds only the key, with no trailing newline. Create the Secret from that file, label it, then delete the file:

kubectl -n default create secret generic frostmoln-api-key --from-file=apikey=./apikey
# REQUIRED — the webhook provider ignores an unlabeled Secret
kubectl -n default label secret frostmoln-api-key external-secrets.io/type=webhook
rm ./apikey

Use kubectl create, not kubectl apply with a manifest: apply keeps the whole manifest, key included, in plain text in the Secret's kubectl.kubernetes.io/last-applied-configuration annotation. Do not commit the key to git. Only the SecretStore and ExternalSecret manifests below are safe to keep in source control.

Keep the key in a namespace of its own

A SecretStore reads the API-key Secret from its own namespace, so the key, the store and the ExternalSecrets that use it share one namespace. The key can read every secret in your tenant, and so can anyone who can read Secrets in that namespace. Use a dedicated namespace for them and keep its RBAC tight. The examples on this page use default.

Step 2 — Create the SecretStore

The SecretStore points ESO at the Frostmoln lookup endpoint. Use the CRD group external-secrets.io/v1 (not the older v1beta1). Replace <TENANT_ID> with your tenant ID.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: frostmoln
  namespace: default
spec:
  provider:
    webhook:
      url: 'https://api.frostmoln.cloud/api/v1/tenants/<TENANT_ID>/secrets/lookup?name={{ .remoteRef.key }}'
      headers:
        # per-secret MAP: <secrets-entry-name>.<data-key>.
        # {{ .apikey }} renders the whole map and the API returns 401 — use the .apikey key.
        X-API-Key: '{{ .apikey.apikey }}'
      result:
        jsonPath: '$.secretValue'
      secrets:
        - name: apikey
          secretRef:
            name: frostmoln-api-key
            key: apikey

The {{ .remoteRef.key }} in the URL is filled in per ExternalSecret with the Frostmoln secret name (Step 3), so one SecretStore serves all your secrets.

Step 3 — Create an ExternalSecret

Each ExternalSecret maps one Frostmoln secret to one Kubernetes Secret. remoteRef.key is the Frostmoln secret name; target.name is the Kubernetes Secret ESO creates and keeps in sync.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-secret
  namespace: default
spec:
  refreshInterval: 1m
  secretStoreRef: { name: frostmoln, kind: SecretStore }
  target: { name: my-secret-synced }
  data:
    - secretKey: value # key inside the synced Kubernetes Secret
      remoteRef: { key: my-secret-name } # the Frostmoln secret name

ESO creates the my-secret-synced Secret with a value key holding the fetched secret value. Mount it or reference it from your Pod like any other Secret.

Refresh interval

refreshInterval controls how often ESO re-fetches. A short interval propagates rotations faster but makes more API calls; a longer one is lighter. Choose per secret based on how quickly a rotation must reach your workloads — a minute or a few minutes is typical.

Pin a specific version

Frostmoln secrets are versioned. To pin a version, add &version=N to the SecretStore URL (create a second SecretStore for the pinned secret):

url: 'https://api.frostmoln.cloud/api/v1/tenants/<TENANT_ID>/secrets/lookup?name={{ .remoteRef.key }}&version=2'

Without &version, ESO always reads the latest version.

Deleting a secret is fail-safe

If you fm secrets delete a secret that ESO is syncing, the lookup starts returning an error (the secret enters pending-deletion). ESO's refresh fails (SecretSyncedError) and keeps the last synced Kubernetes Secret unchanged — it does not blank it out. Your workloads keep running on the last-known-good value.

Recover to resume syncing

While the secret is deleted, ESO cannot refresh it and the value in your cluster goes stale. Run fm secrets recover <name> within the recovery window to restore it; ESO then resumes syncing on the next interval. If the recovery window passes and the secret is permanently removed, the lookup keeps failing — recreate the secret to resume.

Troubleshooting

Check the ExternalSecret status with kubectl describe externalsecret <name>.

• secret does not contain needed label — the API-key Secret is missing the external-secrets.io/type: webhook label (Step 1).
• 401 / unauthorized — the header template is wrong. It must be {{ .apikey.apikey }} (<secrets-entry-name>.<data-key>); {{ .apikey }} renders the whole map and the API rejects it. Also check the key value and that the key hasn't expired.
• Workload Identity: the frostmoln-token Secret does not appear — check that the Workload Identity add-on is on the cluster, that the ServiceAccount carries the annotation and has a binding, and that no Secret of that name existed before: the add-on never overwrites a Secret it did not create.
• could not get secret data from provider — the secret name or tenant ID is wrong, or the secret is deleted / pending-deletion. Verify the name resolves: fm secrets get <name>.
• Nothing syncs at all — check that the SecretStore uses external-secrets.io/v1, not v1beta1. On a cluster you run yourself, confirm ESO is running (kubectl -n external-secrets get pods). On Frostmoln managed Kubernetes you cannot list those pods: confirm the add-on is on the cluster (fm kubernetes cluster addons <cluster-id> or the Add-ons tab), that it was added more than a few minutes ago (if the platform said it could not start its reconciliation, allow longer: usually no more than an hour, more during platform maintenance), and that your nodes can reach ghcr.io and pkg-containers.githubusercontent.com.

Related

• Secrets · Managed Kubernetes