Reference

kube.tf Configuration¶

Type: Reference (Information-oriented)

Complete reference for the kube.tf file that defines the KUP6S Kubernetes cluster infrastructure.

Overview¶

The kube.tf file is the main OpenTofu configuration file for the KUP6S cluster. It uses the kube-hetzner community module to provision a K3S cluster on Hetzner Cloud.

Location: kube-hetzner/kube.tf

Purpose:

Define cluster topology (control planes, agent nodes)
Configure infrastructure components (storage, networking)
Specify extra Kubernetes manifests to deploy
Manage cluster-wide settings

Module Source¶

module "kube-hetzner" {
  source = "kube-hetzner/kube-hetzner/hcloud"
  # Version managed by kube-hetzner project
}

Uses the official kube-hetzner Terraform/OpenTofu module from the Terraform Registry.

Cluster Identity¶

Cluster Name¶

cluster_name = "kup6s"

Purpose: Identifier for the cluster, used in:

Node naming (kup6s-control-fsn1-xyz)
Network names
Resource tagging

Base Domain¶

base_domain = "cluster.kup6s.com"

Purpose: Base domain for reverse DNS entries in Hetzner Cloud. Node FQDNs take the form nodename.cluster.kup6s.com.

Control Plane Node Pools¶

The KUP6S cluster runs 3 control plane nodes across 3 data centers for high availability (embedded etcd, 3-member quorum). Per list-index immutability, retired pools are kept as count = 0 — never reordered or removed.

control_plane_nodepools = [
  {
    name        = "control-fsn1"
    server_type = "cax21"      # ARM64
    location    = "fsn1"
    count       = 1
    swap_size   = "2G"
    longhorn_volume_size = 0
  },
  {
    name        = "control-nbg1"
    server_type = "cax31"      # inactive — replaced by control-nbg1-2 (cx33)
    location    = "nbg1"
    count       = 0
  },
  {
    name        = "control-hel1"
    server_type = "cax21"      # never provisioned
    location    = "hel1"
    count       = 0
  },
  {
    name        = "control-hel1-2"
    server_type = "cx33"       # x86
    location    = "hel1"
    count       = 1
    swap_size   = "2G"
    longhorn_volume_size = 0
  },
  {
    name        = "control-nbg1-2"
    server_type = "cx33"       # x86 — replaces the retired cax31 control-nbg1
    location    = "nbg1"
    count       = 1
    swap_size   = "2G"
    longhorn_volume_size = 0
  },
]

Active Control Planes¶

Node	Type	Arch	vCPU	RAM	Location
control-fsn1	cax21	ARM64	4	8 GB	fsn1
control-hel1-2	cx33	x86	4	8 GB	hel1
control-nbg1-2	cx33	x86	4	8 GB	nbg1

Rationale:

Multi-DC (fsn1/hel1/nbg1): survives a data-center failure; embedded etcd keeps a 3-member quorum.
Mixed arch: fsn1 stays on cheap ARM cax21; nbg1 + hel1 use x86 cx33 — better Hetzner availability (ARM cax is often sold out in nbg1) at a similar price.
Control planes run no Longhorn (longhorn_volume_size = 0).

Agent Node Pools¶

The worker tier is hybrid. The heavy lifting runs on 2× dedicated EX63 bare-metal servers (Hetzner Robot, Helsinki), joined as external K3S agents — these are not part of agent_nodepools (Robot servers are provisioned and joined separately; see Add a dedicated Robot worker). The only remaining cloud agent is a single ARM node reserved for ARM CI builds.

agent_nodepools = [
  {
    name        = "agent-cax31-fsn1"               # "yim" - dedicated ARM CI builder
    server_type = "cax31"                          # ARM64, 8 vCPU / 16 GB
    location    = "fsn1"
    taints      = ["dedicated=arm-ci:NoSchedule"]  # only gitlab-runner-arm tolerates it
    count       = 1
    swap_size   = "2G"
    longhorn_volume_size = 0
  },

  # --- all other cloud agents inactive (count = 0; index preserved) ---
  { name = "agent-cpx22-fsn1", server_type = "cpx22", location = "fsn1", count = 0,
    taints = ["kubernetes.io/arch=amd64:NoSchedule"] },
  { name = "agent-cax31-nbg1", server_type = "cax31", location = "nbg1", count = 0 },
  { name = "agent-cx43-fsn1",  server_type = "cx43",  location = "fsn1", count = 0 },
  { name = "agent-cx43-nbg1",  server_type = "cx43",  location = "nbg1", count = 0 },
  { name = "agent-cx43-hel1",  server_type = "cx43",  location = "hel1", count = 0 },
]

Worker Capacity¶

Worker	Provisioning	Arch	vCPU	RAM	Storage	Location
kup6s-ex-hel-1	Robot (EX63)	x86	20 cores	64 GB	2× 1 TB NVMe	hel (ex-hel-a)
kup6s-ex-hel-2	Robot (EX63)	x86	20 cores	64 GB	2× 1 TB NVMe	hel (ex-hel-b)
agent-cax31-fsn1 (“yim”)	Cloud (cax31)	ARM64	8	16 GB	— (no Longhorn)	fsn1

Strategy:

EX63 dedicated workers carry all general workloads and Longhorn storage (~2 TB usable, 2-way replication with hard anti-affinity across the two hosts). Joined over vSwitch 82213 (private subnet 10.6.0.0/16); LB targets + node lifecycle are managed by the Robot-enabled hcloud CCM.
yim is tainted dedicated=arm-ci:NoSchedule and runs only gitlab-runner-arm — it is not a general scheduling target.
The count = 0 entries are inactive pools kept only to preserve list indices — kube-hetzner tracks pools by position, so they must never be reordered or removed.

Key Configuration Variables¶

Security¶

WireGuard Encryption¶

enable_wireguard = true

Effect: Encrypts all pod-to-pod communication across nodes using WireGuard.

Rationale: Security requirement for sensitive data in transit.

Storage¶

Longhorn Distributed Storage¶

enable_longhorn = true

Effect: Deploys Longhorn distributed block storage for persistent volumes.

Components:

Longhorn manager on each node
CSI driver for volume provisioning
Backup functionality (CIFS)

Hetzner CSI Driver¶

disable_hetzner_csi = true

Status: Disabled.

Effect: The Hetzner Cloud CSI driver and the hcloud-volumes StorageClass are not deployed, and there are no hcloud Cloud Volumes. All persistent storage is Longhorn on the dedicated EX63 NVMe disks.

Note: The SMB CSI driver (Hetzner Storage Box, CIFS — the Longhorn backup target) and the hcloud CCM remain. Don’t confuse the CCM with the CSI driver.

Scheduling¶

Control Plane Scheduling¶

# allow_scheduling_on_control_plane = true

Status: Commented (disabled by default)

Effect: When enabled, allows non-system pods to schedule on control plane nodes.

Current: Control planes run only system components.

Credential Management¶

CRITICAL: All sensitive values are provided via environment variables, not hardcoded in kube.tf.

Required Environment Variables¶

Variable	Purpose
`TF_VAR_hcloud_token`	Hetzner Cloud API access
`TF_VAR_hetzner_s3_access_key`	Hetzner Object Storage access
`TF_VAR_hetzner_s3_secret_key`	Hetzner Object Storage secret
`TF_VAR_longhorn_cifs_url`	Longhorn backup target URL
`TF_VAR_longhorn_cifs_username`	Backup storage username
`TF_VAR_longhorn_cifs_password`	Backup storage password
`TF_VAR_traefik_basicauth_user`	Traefik dashboard user
`TF_VAR_traefik_basicauth_password`	Traefik dashboard password

Loading Credentials¶

# Create .env file from template
cp .env.example .env
# Edit .env with your credentials

# Load credentials (bash)
set -a
source .env
set +a

# Or use fish shell with dotenv plugin
fish -c "dotenv .env; and tofu plan"

Variable Declaration¶

In kube.tf:

variable "hcloud_token" {
  description = "Hetzner Cloud API token"
  type        = string
  sensitive   = true
}

provider "hcloud" {
  token = var.hcloud_token  # From TF_VAR_hcloud_token
}

Anti-Pattern (NEVER):

# ❌ WRONG - hardcoded secret
provider "hcloud" {
  token = "abc123secrettoken"
}

Extra Manifests¶

extra_manifests = ["kube-hetzner/extra-manifests/kustomization.yaml"]

Purpose: Deploys additional Kubernetes manifests via Kustomize during cluster creation.

Location: kube-hetzner/extra-manifests/

Contents: Infrastructure-tier components (ArgoCD, Crossplane, ESO, CNPG, monitoring)

See: Extra Manifests

Node Pool Parameters Reference¶

Common Parameters¶

Parameter	Type	Description
`name`	string	Unique identifier for the node pool
`server_type`	string	Hetzner server type (cax21, cpx31, etc.)
`location`	string	Hetzner data center (fsn1, nbg1, hel1)
`labels`	list(string)	Kubernetes labels applied to nodes
`taints`	list(string)	Kubernetes taints (e.g., `arch=amd64:NoSchedule`)
`count`	number	Number of nodes in this pool
`swap_size`	string	Swap space size (e.g., “2G”)
`longhorn_volume_size`	number	Longhorn storage reservation (0 = disabled)

Available Server Types¶

ARM64 (CAX series):

cax11: 2 vCPU, 4GB RAM, 40GB SSD - €3.79/month
cax21: 4 vCPU, 8GB RAM, 80GB SSD - €7.99/month (control-fsn1)
cax31: 8 vCPU, 16GB RAM, 160GB SSD - €15.99/month (agent “yim”)

AMD64 (CX series, Intel):

cx33: 4 vCPU, 8GB RAM - €6.49/month (control-hel1-2, control-nbg1-2)
cx43: 8 vCPU, 16GB RAM (retired agent type)

Prices are net/month and change over time — confirm current rates in the Hetzner console.

Dedicated (Robot): the EX63 bare-metal workers are not cloud server_types — they are ordered/managed via Hetzner Robot and joined separately (see the dedicated-worker how-to), not through agent_nodepools.

Locations:

fsn1: Falkenstein, Germany
nbg1: Nuremberg, Germany
hel1: Helsinki, Finland (dedicated EX63 workers live here)

Modifying the Configuration¶

Adding a Node Pool¶

Edit kube.tf:

agent_nodepools = [
  # ...existing pools...
  {
    name        = "agent-arm-4"
    server_type = "cax41"
    location    = "fsn1"
    labels      = ["workload=heavy"]
    taints      = []
    count       = 2
    swap_size   = "4G"
    longhorn_volume_size = 0
  }
]

Preview changes:
```
cd kube-hetzner
source .env
tofu plan
```
Apply:
```
tofu apply
```

Changing Node Count¶

Scaling a pool up or down:

{
  name  = "agent-arm-2"
  count = 5  # Changed from 3 to 5
  # ...other parameters...
}

Note: Scaling down removes nodes. Ensure workloads are drained first.

Modifying Taints¶

Remove AMD64 taint to allow multi-arch scheduling:

{
  name   = "agent-amd-3"
  taints = []  # Removed: ["kubernetes.io/arch=amd64:NoSchedule"]
  # ...
}

Safety Guidelines¶

Before Modifying kube.tf¶

Backup: Ensure recent cluster backup exists
Review: Understand impact of changes
Plan: Always run tofu plan first
Test: Test in dev/staging if possible
Monitor: Watch cluster during apply

Dangerous Operations¶

Changing Control Plane Configuration:

Can cause control plane downtime
May require cluster recreation
Test in staging first

Removing Agent Nodes:

Ensure no critical workloads on nodes

Drain nodes before scaling down:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Changing Storage Settings:

May cause Longhorn data loss
Backup all PVs first