Skip to main content

K3s

K3s is a lightweight distribution of Kubernetes that is specifically designed for resource-constrained environments and Internet of Things (IoT) applications. Unlike standard Kubernetes with multiple binaries for its various components, K3s is packaged as a single binary with a lightweight storage backend and minimal external dependencies. For more information about K3s, refer to K3s documentation.

Support Lifecycle​

We support other Kubernetes distributions such as K3s, Microk8s, and RKE2 until their official EOL. The EOL is set by the respective owner. Once we stop supporting the minor version, we initiate the deprecation process. Refer to the Kubernetes Support Lifecycle guide to learn more.

warning

Once you upgrade your cluster to a new Kubernetes version, you will not be able to downgrade. We recommend that, before upgrading, you review the information provided in the Kubernetes Upgrades section.

Versions Supported​

Prerequisites​

  • An edge device with AMD64 processor architecture or a Palette Virtual Cluster.
  • A minimum of 2 CPU cores and 1 GB memory.

Parameters​

Since you can deploy both virtual clusters and Edge clusters using K3s, you have different configuration options depending on the cluster type.

ParameterDescription
cluster.config.cluster-cidrSpecifies the CIDR range that can be used by pods in the cluster.
cluster.config.service-cidrSpecifies the CIDR range that can be used by services in the cluster.
kube-apiserver-argThis parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port.
kube-controller-manager-argThis parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling.
kubelet-argThis parameter contains extra arguments for Kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port.
pack.palette.config.oidc.identityProviderDynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the K3s pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to Configure OIDC Identity Provider.

You can add cloud-init stages, which allow you to customize your instances declaratively. The cloud-init stages are exposed by Kairos, an open source project. For more information, check out the Cloud Init Stages reference.

Usage​

K3s is available for Edge host deployments as well as virtual clusters that you can create from cluster groups. Refer to the Create an Infrastructure Profile guide and the Create and Manage Cluster Groups guide for more information.

info

In order to use K3s as part of an Edge deployment, you need to go through the EdgeForge process and specify K3s as your intended Kubernetes distribution when you build your OS image. For more information, refer to the EdgeForge Workflow guide.

Configure OIDC Identity Provider for Edge​

You can modify the configuration file to configure your Edge cluster to use an OpenID Connect (OIDC) Identity Provider (IDP) for authentication. You can use a custom third-party IDP, such as Okta, or use Palette as your IDP.

When you add the K3s pack to a cluster profile, Palette displays the OIDC IDP options listed below:

  • None: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as noauth.

  • Custom: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in Configure Custom OIDC. This setting displays in the YAML file as none.

  • Palette: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper permissions to view and access the project's resources is able to use kubectl CLI to access cluster. This setting displays in the YAML file as palette. When you select Palette, all you have to do to enable OIDC for your cluster is create role bindings to configure authorization. You do not need to provide extra parameters such as oidc-issuer-url as you need to when you configure a custom OIDC provider.

  • Inherit from Tenant: This setting allows you to apply RBAC to multiple clusters and requires you to configure OpenID Connect (OIDC) in Tenant Settings. In Tenant Admin scope, navigate to Tenant Settings > SSO, choose OIDC, and provide your third-party IDP details. This setting displays in the YAML file as tenant. For more information, check out the SSO Setup guide.

All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes role to users and groups, refer to Create Role Bindings.

warning

If your IDP uses Security Assertion Markup Language (SAML) authentication, then the Inherit from Tenant option will not work, and you will need to use the Custom option instead. This is because Kubernetes supports only OIDC authentication and not SAML authentication.

To configure a custom OIDC IDP, choose Custom when adding the K3s pack to your profile, and then follow these steps:

  1. Add the following OIDC parameters to the kube-apiserver-arg section of your configuration file for your Kubernetes layer when creating a cluster profile.

    cluster:
      config:
        kube-apiserver-arg:
          - oidc-issuer-url="provider URL"
          - oidc-client-id="client-id"
          - oidc-groups-claim="groups"
          - oidc-username-claim="email"

  2. Add the following clientConfig section that contains OIDC parameters to your Kubernetes YAML file and replace the placeholders with your third-party OIDC IDP details. The clientConfig section must be placed at the root level of the YAML file.

    clientConfig:
      oidc-issuer-url: "OIDC-ISSUER-URL"
      oidc-client-id: "OIDC-CLIENT-ID"
      oidc-client-secret: "OIDC-CLIENT-SECRET"
      oidc-extra-scope: profile,email,openid

After you have configured the IDP for authentication, you can proceed to create role bindings to configure authorization in your cluster. Refer to Create Role Bindings for more guidance.

Configure OIDC Identity Provider for Palette Virtual Clusters​

If you are using K3s in a virtual cluster inside of a cluster group, you can also configure OIDC for your cluster. Refer to Configure OIDC for a Virtual Cluster for more guidance.

Add a Certificate for Reverse Proxy​

You can use a reverse proxy with a K3s Kubernetes cluster. The reverse proxy allows you to connect to the cluster API of a Palette-managed Kubernetes cluster in private networks or clusters configured with private API endpoints. For more information, refer to the Spectro Proxy pack guide.

Terraform​

You can retrieve details about the K3s pack by using the following Terraform code.

data "spectrocloud_registry" "public_registry" {
  name = "Public Repo"
}

data "spectrocloud_pack_simple" "k8s" {
  name    = "edge-k3s"
  version = "1.27.5"
  type = "helm"
  registry_uid = data.spectrocloud_registry.public_registry.id
}