Reference

Cluster Capabilities

Type: Reference (Information-oriented)

See Also: Tutorial: First Deployment | Deploy Multi-Architecture Applications

Target Audience: Developers writing CDK8S charts and ArgoCD application deployments

This document describes the capabilities, services, and features available in the kup6s.com Kubernetes cluster for application developers. If you’re deploying applications via ArgoCD or writing CDK8S charts, this is your reference guide.

Cluster Overview

  • Platform: K3S on Hetzner (hybrid: cloud control plane + dedicated bare-metal workers)

  • Architecture: General workloads run on AMD64 (x86) EX workers; one tainted ARM64 node for CI only

  • High Availability: 3 control plane nodes across 3 data centers

  • Deployment Method: GitOps via ArgoCD

  • Kubernetes Version: v1.31.x (automatically managed)

Compute Resources

Node Pools

Control Plane Nodes (3 nodes - not for workloads):

  • 1x CAX21 (ARM64, 4 vCPU, 8GB RAM) in fsn1

  • 2x CX33 (AMD64, 4 vCPU, 8GB RAM) in hel1 and nbg1

  • Cloud nodes, embedded etcd (3 members)

  • Taints: Workloads not scheduled here by default

Worker Nodes:

AMD64 Workers (default scheduling target - 2 dedicated bare-metal nodes):

  • 2x EX63 (kup6s-ex-hel-1, kup6s-ex-hel-2): Intel Core Ultra 7 265 (20 cores), 64GB RAM, 2x 1TB NVMe each

  • Hetzner Robot bare-metal, Helsinki; joined as external K3S agents over a private vSwitch

  • Untainted - all general workloads (and CNPG databases) schedule here unless specified otherwise

ARM64 CI Node (1 node, tainted):

  • 1x CAX31 (kup6s-agent-cax31-fsn1-yim: 8 vCPU, 16GB RAM), cloud, fsn1

  • Tainted dedicated=arm-ci:NoSchedule - runs only the dedicated ARM64 CI build runner

  • Not a general scheduling target; requires both the dedicated=arm-ci toleration and nodeSelector: kubernetes.io/arch=arm64

Architecture Support

General workloads run on AMD64 (x86) by default:

  • ✅ linux/amd64 (default - the EX workers)

  • ✅ linux/arm64 (only via the tainted CI node; use multi-arch images for portability)

Scheduling Behavior:

  • EX (AMD64) workers: Workloads schedule here by default (untainted)

  • ARM64 CI node: Tainted with dedicated=arm-ci:NoSchedule - requires explicit toleration + nodeSelector

Best Practice:

  • Use AMD64 or multi-arch images: They run on the EX workers with no special config

  • Avoid ARM64-only images: Rebuild them as multi-arch instead of pinning to the CI node

  • Multi-platform builds: Build images for both architectures:

    docker buildx build --platform linux/amd64,linux/arm64 -t myapp:latest --push .

Storage Options

1. Longhorn (Default Persistent Storage)

Use for: Stateful applications, databases, persistent volumes

  • StorageClass: longhorn (default)

  • Access Modes: ReadWriteOnce (RWO), ReadWriteMany (RWX), ReadOnlyMany (ROX)

  • File System: XFS

  • Replication: 2 replicas across nodes with hard anti-affinity (configurable; CNPG DB volumes use 1 replica since CNPG self-replicates HA)

  • Backup: Automatic backup to Hetzner Storage Box (CIFS, via SMB CSI)

  • Snapshots: Supported

  • Capacity: ~2 TB usable across the two EX workers (2x 1TB NVMe each)

Example PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi

Special Use Cases:

  • For Kafka workloads: Use longhorn-kafka StorageClass (dedicated for high-throughput workloads)

2. SMB/CIFS Storage (Hetzner Storage Box)

Use for: Shared file storage, backups, multi-pod read/write

  • StorageClass: hetzner-smb

  • Access Modes: ReadWriteMany (RWX)

  • Capacity: Large (Hetzner Storage Box)

  • Performance: Network-based (slower than Longhorn)

Example PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-uploads
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: hetzner-smb
  resources:
    requests:
      storage: 100Gi

3. S3 Object Storage (via Crossplane)

Use for: Object storage, backups, log storage, static assets

  • Provider: Hetzner Object Storage (S3-compatible)

  • Management: Crossplane-managed buckets

  • Access: Via S3 API (AWS SDK compatible)

How to Request a Bucket: Create a Crossplane Bucket resource (see How-To: Create S3 Bucket)

Networking & Ingress

Ingress Controller: Traefik

Default ingress controller for HTTP/HTTPS traffic

  • Version: v3.4.1 (pinned)

  • Features:

    • Automatic HTTPS via Let’s Encrypt (cert-manager)

    • HTTP to HTTPS redirect (enabled by default)

    • Access logs enabled

    • Proxy protocol support

Creating an Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: traefik
  tls:
    - hosts:
        - myapp.sites.kup6s.com
      secretName: myapp-tls
  rules:
    - host: myapp.sites.kup6s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app
                port:
                  number: 80

TLS/SSL Certificates (cert-manager)

Automatic certificate management via Let’s Encrypt

  • Cluster Issuer: letsencrypt-prod

  • DNS Challenge: Not configured (use HTTP-01 challenge)

  • Renewal: Automatic (30 days before expiry)

Usage: Add annotation to Ingress (see example above)

Domain Structure

Available domain patterns:

  • *.sites.kup6s.com - Customer/project websites

  • *.ops.kup6s.net - Infrastructure tools (ArgoCD, Grafana, etc.)

  • *.nodes.kup6s.com - Node-level DNS (internal only)

Network Policy & Observability

  • CNI: Cilium (eBPF-based) with native routing mode

  • Pod-to-Pod Traffic: High-performance eBPF networking

  • Network Policies: Supported (standard Kubernetes NetworkPolicy + Cilium NetworkPolicy for L7)

  • Hubble Observability: ✅ Enabled

    • Service dependency mapping (automatic service maps)

    • Flow visibility (L3/L4/L7 traffic inspection)

    • Network troubleshooting (DNS, HTTP, TCP flows)

    • Hubble UI available for graphical network visualization

    • Metrics exported to Prometheus

Databases

CloudNativePG (PostgreSQL Operator)

Managed PostgreSQL databases via Kubernetes operator

  • Operator: CloudNativePG (CNPG) v1.27.0

  • Backup Plugin: Barman Cloud Plugin v0.7.0 (installed)

  • High Availability: Supported (with replication)

  • Backups: Integrated with S3/Longhorn via Barman Cloud Plugin

  • Monitoring: Prometheus metrics

Creating a PostgreSQL Cluster:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: myapp-db
spec:
  instances: 3
  storage:
    storageClass: longhorn
    size: 20Gi
  postgresql:
    parameters:
      max_connections: "100"

Connection: Use generated secrets for connection strings

Note: For backup configuration using the Barman Cloud Plugin, create an ObjectStore resource and reference it in your cluster’s plugins section. The plugin is deployed via 60-B-barman-plugin.yaml.tpl. See CloudNativePG documentation for details.

Monitoring & Observability

Prometheus + Grafana (kube-prometheus-stack)

Full observability stack pre-installed

Access:

  • Grafana: https://grafana.ops.kup6s.net

  • Prometheus: Internal cluster access only

Metrics Collection:

  • All cluster components monitored by default

  • Your apps: Add Prometheus annotations to expose metrics

ServiceMonitor Example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s

Loki (Log Aggregation)

Centralized logging with S3 backend

  • Storage: Hetzner S3 Object Storage (Crossplane-managed)

  • Access: Via Grafana (Explore → Loki)

  • Retention: Configurable (check with cluster admin)

Log Collection:

  • Container logs automatically collected

  • Query via LogQL in Grafana

Example Query:

{namespace="my-namespace", pod=~"my-app-.*"}

Security Features

Secrets Encryption at Rest

  • ✅ Kubernetes secrets encrypted in etcd (AES-CBC)

  • ✅ Automatic encryption for all Secret resources

  • No action required from developers

Pod Security

  • Pod Security Standards: Baseline enforced

  • Security Contexts: Supported and recommended

Example:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1000
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true

Network Encryption

  • ✅ Pod-to-pod traffic secured (Cilium eBPF with native routing)

  • ✅ Ingress traffic encrypted (TLS via cert-manager)

  • ✅ Secrets encrypted at rest (etcd encryption enabled)

GitOps Deployment (ArgoCD)

ArgoCD Access

Dashboard: https://argocd.ops.kup6s.net

Deployment Workflow

  1. Write CDK8S Chart in argoapps/ directory

  2. Register in registry (apps/registry.ts)

  3. Generate manifests: npm run build

  4. Deploy: git add apps/ dist/ && git commit && git push (argocd-apps-root auto-syncs dist/)

  5. ArgoCD syncs your application automatically

ArgoCD Application Structure

Example CDK8S Chart:

import { Chart } from 'cdk8s';
import { ArgoCdApplication } from '@opencdk8s/cdk8s-argocd-resources';

export class MyAppChart extends Chart {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    new ArgoCdApplication(this, 'myapp', {
      metadata: {
        name: 'myapp',
        namespace: 'argocd',
      },
      spec: {
        project: 'default',
        source: {
          repoUrl: 'https://github.com/your-org/your-repo',
          path: 'k8s/myapp',
          targetRevision: 'main',
        },
        destination: {
          server: 'https://kubernetes.default.svc',
          namespace: 'myapp',
        },
        syncPolicy: {
          automated: {
            prune: true,
            selfHeal: true,
          },
        },
      },
    });
  }
}

Resource Quotas & Limits

No Hard Quotas (Currently)

  • No namespace-level resource quotas configured

  • Best Practice: Always set resource requests/limits in your pods

Recommended:

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

Node Capacity (Total)

Worker Node Resources:

  • EX63 worker #1 (kup6s-ex-hel-1): 20 cores, 64GB RAM, 2x 1TB NVMe (AMD64)

  • EX63 worker #2 (kup6s-ex-hel-2): 20 cores, 64GB RAM, 2x 1TB NVMe (AMD64)

  • ARM64 CI node (...-yim): 8 vCPU, 16GB RAM — tainted dedicated=arm-ci, CI only

  • General-workload total: 40 cores, 128GB RAM (the 2x EX63)

  • Storage: ~2 TB usable Longhorn pool across the EX NVMe disks

Recommended Allocation:

  • All applications and databases → EX (AMD64) workers (default, no special config)

  • ARM64 CI builds only → the tainted ...-yim node (toleration + nodeSelector)

Plan accordingly for your application’s resource needs.

Service Mesh & Advanced Networking

Cilium Advanced Features

The cluster uses Cilium CNI which provides service mesh-like capabilities without a separate service mesh:

  • ✅ L7 Network Policies: HTTP/gRPC/Kafka protocol-aware policies

  • ✅ Service Mesh Lite: Cilium provides observability and L7 policies without sidecar proxies

  • ✅ Hubble Observability: Service dependency maps, flow visualization, network troubleshooting

  • ✅ High Performance: eBPF-based networking bypasses iptables for better performance

Accessing Hubble UI:

# Via Cilium CLI (recommended)
cilium hubble ui

# Or via kubectl port-forward
kubectl port-forward -n kube-system service/hubble-ui 12000:80
# Then open http://localhost:12000

Not Available

  • ❌ Full service mesh (Istio/Linkerd) with sidecar proxies

  • ❌ Advanced traffic splitting/canary deployments (use ArgoCD Rollouts instead)

Use Traefik features for:

  • Load balancing

  • Path-based routing

  • Header-based routing

  • Rate limiting (via middleware)

Backup & Disaster Recovery

Automatic Backups

etcd: Daily S3 backups (cluster state) Longhorn: Recurring backups to Storage Box PostgreSQL: Configure per-database (CNPG backup)

Application Backups

Your responsibility:

  • Application data backup strategy

  • Database backup verification

  • Backup testing

Limitations & Considerations

Architecture Constraints

  • ✅ AMD64 default: General workloads run on the EX (AMD64) workers with no special config

  • ✅ Multi-arch images work: They simply schedule onto the EX workers

  • ⚠️ ARM64 node is tainted for CI: The only ARM64 node carries dedicated=arm-ci:NoSchedule and runs only the CI runner - not a general scheduling target

  • ⚠️ ARM64-only images: Avoid them; rebuild as multi-arch rather than pinning to the CI node

Storage Performance

  • Longhorn: Good for general workloads

  • Longhorn-Kafka: Optimized for high-throughput

  • SMB/CIFS: Slower, best for shared/backup use

Scaling

  • Node scaling: Contact cluster admin

  • HPA (Horizontal Pod Autoscaler): Supported

  • VPA (Vertical Pod Autoscaler): Not configured

External Services

  • External databases: Not directly supported (use port-forward or VPN)

  • Outbound traffic: Unrestricted (no egress filtering)

Quick Reference: Common Tasks

Deploy an Application

Default (AMD64 / multi-arch):

  1. Create namespace (if needed)

  2. Create ArgoCD Application (CDK8S or YAML)

  3. Deploy: git add apps/ dist/ && git commit && git push (argocd-apps-root auto-syncs)

  4. Monitor in ArgoCD dashboard

  5. Workload automatically schedules to the EX (AMD64) workers (no special config needed)

ARM64 CI builder (special case - the only reason to target the ARM node):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: arm-ci-builder
spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64
      tolerations:
        - key: dedicated
          operator: Equal
          value: arm-ci
          effect: NoSchedule
      containers:
        - name: builder
          image: myorg/arm-builder:arm64

Request a PersistentVolume

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi

Expose an Application (Ingress)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: traefik
  tls:
    - hosts: [myapp.sites.kup6s.com]
      secretName: myapp-tls
  rules:
    - host: myapp.sites.kup6s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app
                port:
                  number: 80

Create a PostgreSQL Database

There is no dedicated database node — CNPG runs on the EX (AMD64) workers and stores its volumes on Longhorn. CNPG’s own anti-affinity spreads replicas across the EX hosts for HA, so no nodeSelector or toleration is needed:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: myapp-db
  namespace: databases
spec:
  instances: 3  # HA with replication (CNPG spreads across the EX workers)

  storage:
    size: 10Gi
    storageClass: longhorn

  postgresql:
    parameters:
      max_connections: "100"
      shared_buffers: "256MB"

Single instance (no HA):

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: simple-db
spec:
  instances: 1
  storage:
    size: 10Gi
    storageClass: longhorn

View Logs (Loki)

  1. Open Grafana: https://grafana.ops.kup6s.net

  2. Go to Explore

  3. Select Loki data source

  4. Query: {namespace="your-namespace"}

Monitor Application Metrics

  1. Add Prometheus annotations to Service

  2. Create ServiceMonitor (optional)

  3. View in Grafana dashboards

Getting Help

Cluster Administration Issues

  • Contact: Cluster admin team

  • Topics: Node issues, cluster upgrades, infrastructure

Application Deployment Issues

  • ArgoCD dashboard for sync status

  • Logs via kubectl logs or Grafana/Loki

  • Metrics via Grafana

CDK8S Development