Managed Kubernetes

Frostmoln Managed Kubernetes runs clusters on Frostmoln compute. You manage your workloads; the platform provisions and manages the control plane, the worker nodes, the API load balancer, and your kubeconfig.

Create a cluster

In the portal, go to Kubernetes → Clusters → Create and choose:

• Name.
• Kubernetes version — currently 1.34 or 1.35 (the recommended version is marked); the dropdown is the source of truth.
• Control-plane tier:
  • Standard (development) — one dedicated control-plane node.
  • Production (production) — three dedicated control-plane nodes (HA quorum), production SLA.
• VPC + subnet the cluster runs in (the region is inherited from the VPC).
• An initial node pool — a name, a node flavor (k8s.gp1.medium / k8s.gp1.large / k8s.gp1.xlarge / k8s.gp1.2xlarge), and a node count.

There is no ingress option at create time. How your own traffic reaches your workloads is decided inside the cluster, afterwards — see Reaching your workloads.

Cluster add-ons are optional components the platform installs for you; the catalog is in the portal, in fm kubernetes addon list, and in the frostmoln_kubernetes_addons data source. Choose them when you create the cluster, or add one afterwards to a running cluster — from the cluster's Add-ons tab in the portal, with fm kubernetes cluster add-addons, or by sending the full desired add-on set to the API or Terraform. A later addition starts its own reconciliation and takes effect typically within minutes; if that reconciliation cannot be started it falls back to the platform's own reconciliation period, up to an hour. You can also remove one — from the same places, with fm kubernetes cluster remove-addons, or on the API by dropping the key from addons and naming it in remove (Terraform does both for you when you drop the key). A removal deletes the objects the add-on installed, so it asks you to name what you are removing rather than inferring it from a shorter set; the portal asks you to confirm. What is never deleted for you: the add-on's namespace, any CustomResourceDefinition, StorageClass or CSIDriver it created, and anything you put in that namespace yourself — including credentials. Clusters created before each capability shipped cannot receive a new add-on, or cannot remove one, and say so when you try. On a cluster whose control plane predates the External Secrets Operator add-on, adding it is accepted but never installed — create a new cluster instead. See its guide, which also covers what to do before you remove it. For the Workload Identity add-on, see Workload Identity.

Add-ons need outbound access from your nodes

An add-on's upstream container images are pulled from a public registry when its pods start — they are not preloaded onto your nodes, unlike everything the cluster itself needs. So your worker nodes need outbound HTTPS to that registry: ExternalDNS pulls its controller image from registry.k8s.io, the External Secrets Operator pulls its image from ghcr.io, and Workload Identity pulls its image from registry.sweden.frostmoln.cloud (and exchanges tokens with api.frostmoln.cloud).

If you filter egress by hostname, allow the known hosts: ghcr.io together with pkg-containers.githubusercontent.com, where it redirects image downloads, registry.sweden.frostmoln.cloud and registry.k8s.io. registry.k8s.io alone is not enough, though — it redirects image downloads to cloud-hosted backing stores on other domains, chosen by your egress address. To find such a host, check your egress log for the host the failed pull was redirected to, and allow that too. As a last resort, allow outbound HTTPS from the node pool; that is wider than necessary, so prefer the specific hosts wherever you can list them.

The platform does not check this, and cannot: reachability belongs to your own VPC at the moment of the pull, you can change your routes and security groups at any time, and every new node pulls the image again. Selecting an add-on your nodes cannot reach therefore succeeds — and then its pods stay in ImagePullBackOff:

kubectl -n external-dns get pods
# NAME                            READY   STATUS             RESTARTS   AGE
# external-dns-7d4b8c9f5d-x2klm   0/2     ImagePullBackOff   0          4m

Open egress from your worker nodes and the pull is retried automatically within a few minutes — you do not need to delete the pod. ExternalDNS's other prerequisite still applies: it needs a credential, either a Workload Identity binding (recommended, from add-on version v0.22.0-4) or a frostmoln-dns API-key Secret. Until it has one, its webhook container restarts with no credential (CrashLoopBackOff). On versions before v0.22.0-4 it waits in CreateContainerConfigError. See ExternalDNS.

While the add-on is not running, the DNS records it manages are neither published nor cleaned up — a record for a deleted Service can outlive it and go on pointing at an address that may later be reassigned.

Your cluster itself is never affected. Everything needed to bring a cluster up and keep it Ready is preloaded on the node image and works with no egress at all.

CLI & Terraform

Managed Kubernetes is managed from the portal, with the fm CLI, or with Terraform. The CLI has a full fm kubernetes command group — clusters, node pools, and the flavor / tier / version / addon catalogs; see the fm kubernetes reference. In Terraform, use frostmoln_kubernetes_cluster (with its initial node pool and cluster addons), frostmoln_kubernetes_node_pool for additional pools, and the frostmoln_kubernetes_versions / _tiers / _flavors / _addons catalog data sources — see the Terraform reference. Once you have a kubeconfig you use kubectl, Helm, and your normal Kubernetes tooling against the cluster as usual.

Add-on versions

Each add-on that has versions is pinned to one of them. The pinned version is the one you asked for; the applied version is the one the platform last confirmed it installed. They are two separate facts, always shown together: after you move a pin, the applied version lags behind until the change's own reconciliation applies it, typically within minutes; if that reconciliation cannot be started it falls back to the platform's next one, up to an hour. A change that fails to apply does not move the pin back — the platform keeps retrying, and you roll back by moving the pin back yourself.

• Portal — the cluster's Add-ons tab has an Add-on versions card with each add-on's pinned version, applied version and state. Change version moves a pin.

• CLI — list the versions you can pin, show pinned and applied side by side, and move a pin:

  fm kubernetes addon versions <addon>
  fm kubernetes cluster addons <cluster-id>
  fm kubernetes cluster pin-addons <cluster-id> --version <addon>=<version>

  add-addons takes the same repeatable --version flag, and so does remove-addons for an add-on you keep; cluster create takes it as --addon-version.

• Terraform — the addon_versions map on frostmoln_kubernetes_cluster pins the add-ons it names (each key must also be in addons) and is changed in place. The frostmoln_kubernetes_addon_versions data source lists the versions you can pin; frostmoln_kubernetes_cluster_addons reads each add-on's pinned and applied version.

Versions are listed in the platform's order, with the recommended one marked. That order is not a ranking: never sort version strings or take the last one as the newest — copy a version exactly as it is listed. A deprecated version can still be pinned but is on its way out. A version at end of life is not listed and cannot be newly pinned; a cluster already pinned to one keeps running it, and its pin shows that status so you know to move it. An add-on you select without naming a version — at creation or later — gets its recommended version. To pin another version when you create a cluster: in the portal, choose it in the add-on's Version select, which appears when the add-on has a version other than the recommended one and starts on Recommended; with the CLI, pass --addon-version <addon>=<version> to fm kubernetes cluster create once per add-on; in Terraform, set addon_versions together with an explicit addons, and it is sent with the create request. Every add-on you pin must be one the cluster gets. A pin for any other add-on, or for a version that is not listed, is refused and no cluster is created.

Get your kubeconfig

When the cluster is running, use Download Kubeconfig on the cluster detail page. Point kubectl at it and you're in:

export KUBECONFIG=~/Downloads/my-cluster-kubeconfig.yaml
kubectl get nodes

The API endpoint is private to your VPC

The Kubernetes API server (port 6443) listens on an address inside your own VPC, and it accepts connections from that VPC only. There is no public address for it, and no setting that gives it one.

So kubectl needs a route into the VPC. There are two:

• Let the CLI tunnel for you. fm kubernetes cluster kubeconfig <cluster-id> --exec --file ./kubeconfig writes a kubeconfig that opens its own tunnel on demand, so one terminal is enough and no client key is written to disk — the credential is fetched per call. It needs an API key carrying both kubernetes:read and sessions:write; a key scoped only to kubernetes:* covers the read and not the session. A session lasts at most 60 minutes and is closed after 10 minutes without traffic, so a long-running or quiet kubectl stream is cut at that point; the plugin opens a fresh tunnel for the next command. A platform restart or a network interruption can end a session sooner, in the same way. Use --proxy instead if you would rather run the tunnel yourself with fm kubernetes cluster proxy in a second terminal — that writes the client key into the kubeconfig file.
• Run kubectl inside the VPC, typically from an instance you own there.

Not every cluster's control plane can serve these tunnels. If fm answers that this one cannot, retrying will not help — use the in-VPC route and contact support.

With neither in place, running the commands above straight from a laptop, or from a CI runner outside the VPC, will time out: the kubeconfig is correct, the network path is not. The same applies to Helm and to Terraform's kubernetes / helm providers.

The cluster's credentials are generated and held securely by the platform — node bootstrapping never transmits secrets back.

The cluster endpoint is not for your traffic. It is for kubectl, Helm and your CI. Your own traffic goes to the address of a load balancer you ask for from inside the cluster — see Reaching your workloads. Pointing an application's DNS record at the cluster endpoint is the mistake this distinction exists to prevent.

Reaching your workloads

A cluster comes with no load balancer for your own traffic. You get one the standard Kubernetes way: create a Service of type: LoadBalancer, and the platform provisions a load balancer for that Service, keeps your Pods wired in as its members, and writes the address back into the Service.

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
    - name: https
      port: 443
      targetPort: 8443

kubectl get service web --watch
# NAME   TYPE           EXTERNAL-IP   PORT(S)
# web    LoadBalancer   10.20.4.17    443:31274/TCP

The address that appears as EXTERNAL-IP is the one to point your DNS at.

An ingress controller works the same way. Install the one you prefer (Traefik, ingress-nginx, …) and expose it with a type: LoadBalancer Service rather than a NodePort one. There are no reserved node ports to pin any more: Kubernetes picks the node ports itself and you never need to know them. TLS terminates in your cluster, with your own certificates — the platform never holds your keys. What else you can install, and the values some charts need, is in What you can do in your cluster.

A few properties worth knowing:

• Each such Service gets its own load balancer. It shows up in your normal load-balancer list and is billed as a load balancer, like any other.
• Manage it from Kubernetes, not from the load-balancer API. The Service is what the load balancer is reconciled against: change its ports or selector and the load balancer follows.
• Delete the Service and its load balancer goes with it. Remove your Services before you delete the cluster, so nothing is left behind.
• The Kubernetes API endpoint is separate and is not one of these — it is provisioned by the platform, not by a Service of yours.

Public or internal

By default a Service's load balancer gets an internal address from the cluster's own subnet — reachable inside the VPC, not from the internet. To expose it publicly, annotate the Service:

• A Public IP from the pool — frostmoln.se/lb-scheme: public. The platform allocates a Public IP, attaches it to the load balancer, publishes it as the Service's EXTERNAL-IP, and releases it when you delete the Service.
• Bring your own Public IP — also set frostmoln.se/lb-public-ip-id: <public-ip-id>. The platform uses a Public IP you already own (it must not already be in use) and keeps it when you delete the Service. The value is the Public IP's id (a UUID), not its address — find it in the Public IPs list or with the CLI.

apiVersion: v1
kind: Service
metadata:
  name: web
  annotations:
    frostmoln.se/lb-scheme: public
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
    - name: https
      port: 443
      targetPort: 8443
  # Who may reach it — see the warning below.
  loadBalancerSourceRanges:
    - 203.0.113.0/24

To reuse a Public IP you already hold, add its id alongside the scheme:

metadata:
  annotations:
    frostmoln.se/lb-scheme: public
    frostmoln.se/lb-public-ip-id: 3f2504e0-4f89-41d3-9a0c-0305e82c3301

A public Service with no source ranges is open to the whole internet

loadBalancerSourceRanges is the access control for a public load balancer: the datapath preserves the client's source address, so it is real filtering, not a hint. Omit it and the Service is reachable from 0.0.0.0/0 — the same default as other clouds. Set it to the CIDRs that should be able to reach the Service.

What the platform records on your Service

As it works on a Service's load balancer, the platform records Kubernetes events on the Service, with the reasons a cloud provider's controller uses, so kubectl describe service <name> shows where it stands:

• EnsuringLoadBalancer when it starts creating the load balancer or applying a change to it;
• EnsuredLoadBalancer when that is done, or UpdatedLoadBalancer when only the set of nodes behind it changed;
• DeletingLoadBalancer, then DeletedLoadBalancer, when the Service is deleted or stops being type: LoadBalancer (for a deleted Service, kubectl get events in its namespace still lists them);
• a Warning, SyncLoadBalancerFailed, on every attempt that fails. The Service stays at EXTERNAL-IP <pending>, and the message starts with Error syncing load balancer: failed to ensure load balancer:, followed by the specific reason — PublicIPNotFound or PublicIPInUse for a bring-your-own id that is not yours or already in use, PublicIPForbidden for an exhausted Public IP pool or quota, RealiseFailed for anything else — and what to do about it.

A Service that has not changed gets no events when the platform re-checks it. A cluster whose control plane predates these events records only the Warning, with the specific reason as the event's reason; the others appear once its control plane has been upgraded. Kubernetes keeps events for one hour.

The automatic worker ingress load balancer has been removed

Clusters used to be given an ingress load balancer at create time, forwarding TCP 80 and 443 to fixed node ports on every worker node, chosen with an ingressScheme option (ingress_scheme in Terraform) and an optional bring-your-own address. All of that is gone: there is no ingress setting when you create a cluster, and no fixed node ports to publish. Use a Service of type: LoadBalancer, as above.

The API endpoint's exposure is not a setting

There is no scheme option on a cluster, and no publicIpId either — both are rejected. The answer is fixed rather than merely platform-decided: the API endpoint is private to your VPC, always. See Get your kubeconfig.

Node pools

A cluster has one or more node pools — groups of worker nodes of a given flavor. From the cluster detail page you can:

• Add a node pool (name, flavor, node count), optionally with labels and taints for scheduling (labels and taints are portal-only for now — the Terraform resource does not support them yet).
• Scale a pool by setting a new node count.
• Delete a pool.

Scale manually for now

Cluster autoscaling is not yet available — set the node count explicitly and scale pools manually as your workload changes.

Persistent storage

Clusters ship with a Frostmoln CSI storage class, frostmoln-block (the default StorageClass), backed by Frostmoln block storage. Create a PersistentVolumeClaim and a volume is provisioned and attached automatically; volumes support online expansion.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: ['ReadWriteOnce']
  storageClassName: frostmoln-block
  resources:
    requests:
      storage: 20Gi

Each claim's volume also appears in your volume list, labelled Kubernetes. See Kubernetes volumes for how they are managed and what happens to them when you delete the cluster.

Networking

The cluster CNI is Cilium. Cluster nodes reach the internet only if the cluster's VPC has a Gateway — without one the nodes have no outbound access, no DNS resolution and no managed-service connectivity, so add a Gateway to the VPC yourself if they need any of that. Creating a cluster never adds one for you. The nodes are isolated from Frostmoln's internal networks either way — your workloads run in their own tenant network. Reach the API server through the cluster endpoint above, and expose your own services with a Service of type: LoadBalancer (see Reaching your workloads).

What you can do in your cluster

You are the cluster administrator. Your kubeconfig can create any resource in your own namespaces, and any cluster-scoped resource the platform does not reserve — including CustomResourceDefinitions, ClusterRoles and ClusterRoleBindings (with any verbs, including *), and admission webhooks. So you install operators from their upstream Helm charts: Traefik (with its CRD provider), Argo CD, cert-manager (including its webhook), Kyverno, CloudNativePG, kube-prometheus-stack and the like.

ServiceAccounts you create are subject to the same rules as your kubeconfig.

What stays reserved to Frostmoln

• Platform namespaces — kube-system, kube-public, kube-node-lease, frostmoln-system, frostmoln-workload-identity, external-secrets, and the namespaces of add-ons Frostmoln installs for you (such as external-dns). You can see what runs there, but not change it. The exception is the frostmoln-dns Secret you create in the ExternalDNS namespace: it stays yours. In the six named first (kube-system to external-secrets), your kubeconfig also cannot read Secrets, ConfigMaps or pod logs. Add-on namespaces hold no platform credential, so their Secrets and ConfigMaps stay readable to you, one namespace at a time: an all-namespaces read (kubectl get secrets -A) is refused, because it spans the platform namespaces.
• Cluster infrastructure — Nodes, StorageClasses, CSI drivers and PersistentVolumes (request storage with a PersistentVolumeClaim, see Persistent storage), APIServices, API Priority and Fairness (FlowSchemas) and ServiceCIDRs.
• Platform API groups — CustomResourceDefinitions in frostmoln.cloud, frostmoln.se, cilium.io, external-secrets.io, k8s.io and kubernetes.io (and their subdomains), and objects in the Cilium (such as CiliumNetworkPolicy) and platform groups. Use a standard NetworkPolicy for network rules; your own ExternalSecrets and SecretStores are fine.
• Services — a Service may not set spec.externalIPs, and you cannot write a Service's status yourself: load-balancer addresses come from the platform, so use type: LoadBalancer. The API server's own kubernetes Service in the default namespace and its endpoints are reserved too. A load-balancer controller you run yourself (such as MetalLB or kube-vip) therefore cannot assign addresses; use type: LoadBalancer, which Frostmoln serves (see Reaching your workloads).
• Admission policies — ValidatingAdmissionPolicies and MutatingAdmissionPolicies. Use an admission webhook instead.
• Identity — impersonating anything other than your own ServiceAccounts, and certificate signing via the kubernetes.io signers.
• Pod Security — Pod Security restricted is enforced cluster-wide and a namespace's Pod Security labels cannot be changed, so privileged, hostNetwork, hostPID and hostPath pods are refused.
• System priority classes — pods in your namespaces cannot use system-node-critical or system-cluster-critical, or any priority above 1000000000, and those two classes cannot be changed; they keep the platform's own pods schedulable. Create your own PriorityClass instead. A chart that defaults to a system class, such as metrics-server, needs its priorityClassName value set to your class or to an empty string.
• Control-plane endpoints — the control plane's metrics and debug endpoints. Through the Kubernetes API, the kubelet API is limited to node metrics and stats (nodes/metrics, nodes/stats); nodes/proxy and nodes/log are not available. Tools that reach the kubelet through the API server proxy — for example Grafana Alloy or k8s-monitoring scraping /api/v1/nodes/<node>/proxy/metrics/cadvisor, or Fluent Bit with Use_Kubelet On — are refused; configure them to scrape the kubelet directly on the node (as kube-prometheus-stack and metrics-server do), which uses the allowed node metrics and stats permissions.

Admission webhooks

A webhook must point to a Service in your cluster, not a URL. If every rule targets your own CRD API groups, the webhook is accepted as it is. A webhook that intercepts built-in resources — Pods, Deployments, Services and so on — must also:

• set timeoutSeconds to 10 or less;
• set sideEffects: None (or NoneOnDryRun);
• exclude the platform namespaces and add-on objects with these selectors:

namespaceSelector:
  matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values:
        [
          kube-system,
          kube-public,
          kube-node-lease,
          frostmoln-system,
          frostmoln-workload-identity,
          external-secrets,
        ]
    - key: frostmoln.se/addon
      operator: DoesNotExist
objectSelector:
  matchExpressions:
    - key: frostmoln.se/addon
      operator: DoesNotExist

• list the resources in each rule — no wildcard resource (* or */status);
• if it intercepts Services, Endpoints or EndpointSlices, also list default in the kubernetes.io/metadata.name NotIn values — the API server maintains its own endpoints there;
• not intercept nodes, RBAC, CRDs, admission configuration, leases, certificates, storage or flowcontrol resources, PersistentVolumes, IPAddresses, ServiceCIDRs or PriorityClasses.

Chart values

Argo CD, Traefik (rbac.enabled=true, with its CRD provider) and CloudNativePG install with their default values. A few other charts need these:

• cert-manager — global.leaderElection.namespace=cert-manager; its default puts leader-election objects in kube-system.

• kube-prometheus-stack — the control plane is managed and cannot be scraped, and its control-plane components would create Services in kube-system. node-exporter needs host access, which Pod Security refuses. Point the kubelet Service at your release namespace:

  kubeControllerManager:
    enabled: false
  kubeScheduler:
    enabled: false
  kubeEtcd:
    enabled: false
  kubeProxy:
    enabled: false
  coreDns:
    enabled: false
  kubeApiServer:
    enabled: false
  nodeExporter:
    enabled: false
  prometheusOperator:
    kubeletService:
      namespace: monitoring # your release namespace

• Kyverno — once you add policies, set its config.webhooks to the namespaceSelector above plus the objectSelector (Kyverno's defaults only exclude kube-system). Its TTL cleanup webhook and its ValidatingAdmissionPolicy generation are not available; policies are enforced through its webhooks.

• ingress-nginx — its admission webhook intercepts Ingress, a built-in resource. Set controller.admissionWebhooks.namespaceSelector to the namespaceSelector above and controller.admissionWebhooks.objectSelector to the objectSelector above (its default timeout and sideEffects: None already comply), or turn the webhook off with controller.admissionWebhooks.enabled=false.

Limits

• One zone per cluster — nodes are placed in the cluster's region/subnet; no multi-AZ spread today.
• Cilium is the only CNI — it isn't customer-selectable.
• No in-place version upgrades yet — choose your version at create time.
• Autoscaling is not yet available (scale node pools manually).
• Quotas cap clusters and nodes per tenant. For a large cluster, check your quotas and request an increase before you create it.
• A workload load balancer is internal by default — a Service of type: LoadBalancer gets an internal VPC address unless you annotate it frostmoln.se/lb-scheme: public (see Reaching your workloads).
• Deleting a cluster removes its nodes, its API load balancer and its stored credentials.

Related

• Compute / Instances · Network: VPCs · Network: Gateway
• Network: Load Balancers · Billing