Explanation

Capsule Multi-Tenancy

Type: Explanation (Understanding-oriented)

Related Concepts: Security Model | Work as Capsule Tenant

Understanding the multi-tenancy architecture using Capsule for namespace isolation and resource management in the KUP6S cluster.

Overview

The KUP6S cluster uses Capsule to provide multi-tenancy capabilities. Capsule enables delegating namespace management to tenant owners while maintaining cluster-level control and isolation.

Key capabilities:

  • Namespace delegation: Tenant owners can create and manage namespaces within their tenant boundaries

  • Resource quotas: Cluster-wide limits on CPU, memory, and namespace count per tenant

  • Namespace prefixing: Automatic association of namespaces to tenants via naming convention

  • RBAC isolation: Tenants cannot access resources outside their boundaries

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Cluster Administrators                    │
│   - Create/manage Tenants                                   │
│   - Define resource quotas                                  │
│   - Manage ClusterSecretStores                              │
└─────────────────────────────────────────────────────────────┘
                              │
                    ┌─────────┴─────────┐
                    │  Capsule Operator  │
                    │  (capsule-system)  │
                    └─────────┬─────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
        ▼                     ▼                     ▼
┌───────────────┐     ┌───────────────┐     ┌───────────────┐
│  Tenant: bd   │     │  Tenant: kup  │     │Tenant: progr..│
│  Owner: bd-sa │     │ Owner: kup-sa │     │Owner: prog-sa │
│  Prefix: bd-* │     │ Prefix: kup-* │     │Prefix: prog-* │
└───────┬───────┘     └───────────────┘     └───────────────┘
        │
        ├── bd-test (namespace)
        ├── bd-praxis (namespace)
        └── bd-secrets (namespace for ESO)

Tenant Configuration

A Tenant is a Kubernetes Custom Resource that defines:

Ownership

spec:
  owners:
  - kind: ServiceAccount
    name: system:serviceaccount:capsule-system:bd-sa
    clusterRoles:
    - admin
    - capsule-namespace-deleter

The owner ServiceAccount can:

  • Create namespaces that automatically belong to the tenant

  • Full admin access within tenant namespaces

  • Delete namespaces within the tenant

Resource Quotas

spec:
  resourceQuotas:
    items:
    - hard:
        limits.cpu: "32"
        limits.memory: 64Gi
        requests.cpu: "16"
        requests.memory: 32Gi
    scope: Tenant
  namespaceOptions:
    quota: 20

Quotas are enforced at the tenant level, not per-namespace. This allows flexible resource distribution across namespaces while maintaining overall limits.

Current Tenants

Tenant

Owner SA

Namespace Prefix

NS Quota

CPU Limit

Memory Limit

bd

bd-sa

bd-*

20

32

64Gi

kup

kup-sa

kup-*

20

32

64Gi

programmatic

prog-sa

prog-*

10

16

32Gi

Namespace Association

Capsule automatically associates namespaces to tenants based on:

  1. Creation context: When a tenant owner creates a namespace, it belongs to their tenant

  2. Label matching: Namespaces with capsule.clastix.io/tenant: <tenant-name> label

In practice, tenant owners simply create namespaces and Capsule handles the association:

# As bd-sa ServiceAccount
kubectl create namespace bd-myapp
# Namespace automatically belongs to 'bd' tenant

Authentication

Tenant owners authenticate using a kubeconfig with their ServiceAccount token:

apiVersion: v1
kind: Config
clusters:
- cluster:
    server: https://api.kup6s.com:6443
    certificate-authority-data: <base64-ca>
  name: kup6s
contexts:
- context:
    cluster: kup6s
    user: bd-sa
  name: bd-tenant
users:
- name: bd-sa
  user:
    token: <serviceaccount-token>

The token is generated from a Secret bound to the ServiceAccount in capsule-system.

Integration with External Secrets Operator

For tenant secret management, we recommend creating a dedicated secrets namespace per tenant:

Tenant: bd
├── bd-secrets (source namespace)
│   ├── Source secrets (bootstrap once)
│   └── ServiceAccount + RBAC for ESO
│
└── bd-* (application namespaces)
    ├── ExternalSecrets → reference ClusterSecretStore
    └── Synced secrets (managed by ESO)

A ClusterSecretStore with namespace regex restriction ensures tenants can only access their own secrets:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: bd-secrets-store
spec:
  conditions:
    - namespaceRegex: "^bd-.*"
  provider:
    kubernetes:
      remoteNamespace: bd-secrets
      auth:
        serviceAccount:
          name: eso-bd-reader
          namespace: bd-secrets

See Work as Capsule Tenant for the complete ESO setup guide.

Security Boundaries

What tenants CAN do

  • Create/delete namespaces within their tenant

  • Deploy any workloads in their namespaces

  • Create Services, ConfigMaps, Secrets in their namespaces

  • Use cluster-wide StorageClasses (Longhorn, SMB)

  • Create Ingresses (with hostname restrictions if configured)

  • Access ClusterSecretStores restricted to their namespaces

What tenants CANNOT do

  • Access other tenants’ namespaces

  • Create cluster-scoped resources (ClusterRoles, ClusterSecretStores)

  • Modify Capsule Tenant configuration

  • Exceed tenant resource quotas

  • Access secrets in application-secrets or other tenants’ secret namespaces

Deployment via ArgoCD

Capsule configuration is managed via ArgoCD from the dp-infra repository:

dp-infra.git
└── capsule/
    └── manifests/
        ├── tenant-bd.yaml
        ├── tenant-kup.yaml
        ├── tenant-programmatic.yaml
        └── serviceaccounts.yaml

Changes to tenant configuration follow the standard GitOps workflow:

  1. Modify tenant manifest in dp-infra

  2. Commit and push

  3. ArgoCD syncs changes to cluster

Design Decisions

Why Capsule over alternatives?

Alternative

Limitation

Namespace RBAC only

No automatic namespace association, manual quota management

Hierarchical Namespaces

More complex, less mature

Virtual Clusters (vcluster)

Heavyweight, separate API servers

Loft/Rancher

Commercial, more features than needed

Capsule provides the right balance: lightweight, open-source, Kubernetes-native multi-tenancy without the overhead of virtual clusters.

ServiceAccount vs User authentication

We use ServiceAccounts instead of user certificates because:

  • Easier token rotation (recreate Secret)

  • No PKI infrastructure required

  • Tokens can be scoped and audited

  • Works with existing Kubernetes RBAC

Tenant-level vs namespace-level quotas

Tenant-level quotas allow flexibility:

  • One namespace can temporarily use more resources

  • No per-namespace quota management overhead

  • Tenant is responsible for internal resource distribution