Reference

kubectl Commands

Type: Reference (Information-oriented)

See Also: Troubleshooting Reference | Resource Requirements

---

Overview

Quick reference for common kubectl operations on GitLab BDA.

Namespace: gitlabbda (all commands assume -n gitlabbda)

---

Status & Health Checks

View All Pods

kubectl get pods -n gitlabbda

Expected output (all Running):

NAME                              READY   STATUS    RESTARTS   AGE
gitlab-gitaly-0                   1/1     Running   0          5d
gitlab-postgres-1                 1/1     Running   0          5d
gitlab-postgres-2                 1/1     Running   0          5d
gitlab-postgres-pooler-xxx        1/1     Running   0          5d
gitlab-webservice-xxx             2/2     Running   0          2d
harbor-core-xxx                   1/1     Running   0          5d
harbor-registry-xxx               1/1     Running   0          5d
redis-0                           1/1     Running   0          5d

Check Pod Status (Specific Component)

# GitLab
kubectl get pods -l app=webservice -n gitlabbda
kubectl get pods -l app=gitaly -n gitlabbda
kubectl get pods -l app=sidekiq -n gitlabbda

# PostgreSQL
kubectl get pods -l cnpg.io/cluster=gitlab-postgres -n gitlabbda

# Harbor
kubectl get pods -l app.kubernetes.io/part-of=harbor -n gitlabbda

# Redis
kubectl get pods -l app.kubernetes.io/name=redis -n gitlabbda

Check Services

# All services
kubectl get svc -n gitlabbda

# Specific service
kubectl get svc gitlab-webservice -n gitlabbda
kubectl get svc gitlab-postgres-pooler -n gitlabbda

Check Ingresses

# HTTP/HTTPS ingresses
kubectl get ingress -n gitlabbda

# Traefik TCP routes (SSH)
kubectl get ingressroutetcp -n gitlabbda

---

Logs & Debugging

View Pod Logs

# Recent logs (last 50 lines)
kubectl logs <pod-name> -n gitlabbda --tail=50

# Follow logs (live stream)
kubectl logs <pod-name> -n gitlabbda -f

# Previous pod logs (after crash)
kubectl logs <pod-name> -n gitlabbda --previous

# Specific container (multi-container pods)
kubectl logs <pod-name> -c <container-name> -n gitlabbda

Examples:

# GitLab Webservice logs
kubectl logs -l app=webservice -n gitlabbda --tail=100

# PostgreSQL logs
kubectl logs gitlab-postgres-1 -n gitlabbda --tail=50

# Harbor Core logs
kubectl logs -l app.kubernetes.io/name=harbor-core -n gitlabbda -f

Search Logs

# Grep for errors
kubectl logs <pod-name> -n gitlabbda | grep -i error

# Filter by pattern
kubectl logs -l app=webservice -n gitlabbda --tail=1000 | grep "POST /api"

# Count occurrences
kubectl logs -l app=sidekiq -n gitlabbda --tail=10000 | grep -c "JobFailed"

Execute Commands in Pod

# Interactive shell
kubectl exec -it <pod-name> -n gitlabbda -- bash

# Single command
kubectl exec <pod-name> -n gitlabbda -- <command>

Examples:

# PostgreSQL shell
kubectl exec -it gitlab-postgres-1 -n gitlabbda -- psql -U postgres -d gitlab

# Redis CLI
kubectl exec -it redis-0 -n gitlabbda -- redis-cli

# GitLab Rails console
kubectl exec -it deploy/gitlab-toolbox -n gitlabbda -- gitlab-rails console

# Check disk usage in Gitaly
kubectl exec gitlab-gitaly-0 -n gitlabbda -- df -h /home/git/repositories

---

Resource Usage

Check CPU & Memory Usage

# All pods
kubectl top pods -n gitlabbda

# Specific pod
kubectl top pod gitlab-webservice-xxx -n gitlabbda

# Sort by CPU
kubectl top pods -n gitlabbda --sort-by=cpu

# Sort by memory
kubectl top pods -n gitlabbda --sort-by=memory

Check Node Resources

# All nodes
kubectl top nodes

# Pods per node
kubectl get pods -n gitlabbda -o wide | awk '{print $7}' | sort | uniq -c

Check Storage Usage

# PVC status
kubectl get pvc -n gitlabbda

# PVC usage details
kubectl exec <pod-name> -n gitlabbda -- df -h

# Longhorn volumes
kubectl get volumes.longhorn.io -n longhorn-system | grep gitlabbda

Examples:

# PostgreSQL disk usage
kubectl exec gitlab-postgres-1 -n gitlabbda -- df -h /var/lib/postgresql/data

# Redis disk usage
kubectl exec redis-0 -n gitlabbda -- df -h /data

# Gitaly repository size
kubectl exec gitlab-gitaly-0 -n gitlabbda -- du -sh /home/git/repositories

---

PostgreSQL Operations

Check Cluster Status

# CNPG Cluster overview
kubectl get cluster gitlab-postgres -n gitlabbda

# Detailed status
kubectl get cluster gitlab-postgres -n gitlabbda -o yaml | grep -A 20 status

Check Replication Lag

# Via CNPG Cluster status
kubectl get cluster gitlab-postgres -n gitlabbda -o jsonpath='{.status.instancesStatus[*].replicationLag}'

# Via psql
kubectl exec -it gitlab-postgres-1 -n gitlabbda -- psql -U postgres -c "SELECT client_addr, state, sync_state, replay_lag FROM pg_stat_replication"

Check Database Connections

# Active connections
kubectl exec -it gitlab-postgres-1 -n gitlabbda -- psql -U postgres -d gitlab -c "SELECT count(*) FROM pg_stat_activity"

# Connections by state
kubectl exec -it gitlab-postgres-1 -n gitlabbda -- psql -U postgres -d gitlab -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state"

# Long-running queries
kubectl exec -it gitlab-postgres-1 -n gitlabbda -- psql -U postgres -d gitlab -c "SELECT pid, now() - query_start AS duration, query FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC"

Check Pooler Status

# Pooler pods
kubectl get pods -l cnpg.io/poolerName=gitlab-postgres-pooler -n gitlabbda

# Pooler logs
kubectl logs -l cnpg.io/poolerName=gitlab-postgres-pooler -n gitlabbda

Backup Status

# Check ObjectStore (Barman config)
kubectl get objectstore gitlab-postgres-backup -n gitlabbda

# Check backups (CNPG)
kubectl exec -it gitlab-postgres-1 -n gitlabbda -- barman-cloud-backup-list \
  s3://postgresbackups-gitlabbda-kup6s/ \
  --endpoint-url=https://fsn1.your-objectstorage.com

---

Redis Operations

Check Redis Status

# Pod status
kubectl get pods -l app.kubernetes.io/name=redis -n gitlabbda

# Redis info
kubectl exec redis-0 -n gitlabbda -- redis-cli info

# Memory usage
kubectl exec redis-0 -n gitlabbda -- redis-cli info memory | grep used_memory_human

Check Redis Keys

# Count keys by database
kubectl exec redis-0 -n gitlabbda -- redis-cli --scan --pattern "*" | wc -l

# Keys in DB 0 (GitLab)
kubectl exec redis-0 -n gitlabbda -- redis-cli -n 0 dbsize

# Keys in DB 2 (Harbor Registry cache)
kubectl exec redis-0 -n gitlabbda -- redis-cli -n 2 dbsize

Clear Redis Cache

# Clear specific database (use with caution!)
kubectl exec redis-0 -n gitlabbda -- redis-cli -n 0 flushdb

# Clear all databases (DANGER!)
kubectl exec redis-0 -n gitlabbda -- redis-cli flushall

---

S3 Operations

Check Bucket Status (Crossplane)

# All GitLab BDA buckets
kubectl get buckets -n crossplane-system | grep gitlabbda

# Specific bucket
kubectl get bucket artifacts-gitlabbda-kup6s -n crossplane-system

# Detailed status
kubectl describe bucket artifacts-gitlabbda-kup6s -n crossplane-system

List Bucket Contents (aws CLI)

# Set credentials
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=yyy

# List bucket
aws s3 ls s3://artifacts-gitlabbda-kup6s --endpoint-url=https://fsn1.your-objectstorage.com

# Bucket size
aws s3 ls s3://artifacts-gitlabbda-kup6s --endpoint-url=https://fsn1.your-objectstorage.com --recursive --summarize --human-readable

Test S3 Connectivity from Pod

kubectl run -it s3-test --image=amazon/aws-cli --rm \
  --env="AWS_ACCESS_KEY_ID=xxx" \
  --env="AWS_SECRET_ACCESS_KEY=yyy" \
  -- s3 ls --endpoint-url=https://fsn1.your-objectstorage.com

---

Secrets Management

Check ExternalSecret Status

# All ExternalSecrets
kubectl get externalsecrets -n gitlabbda

# Specific secret
kubectl get externalsecret gitlab-s3-credentials -n gitlabbda

# Sync status (SYNCED=True means up-to-date)
kubectl get externalsecret gitlab-s3-credentials -n gitlabbda -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

View Secret Data (base64 decoded)

# List secret keys
kubectl get secret gitlab-s3-credentials -n gitlabbda -o jsonpath='{.data}' | jq 'keys'

# Decode specific key
kubectl get secret gitlab-s3-credentials -n gitlabbda -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d

# View all secret data (formatted)
kubectl get secret gitlab-s3-credentials -n gitlabbda -o json | jq '.data | map_values(@base64d)'

Force ExternalSecret Refresh

# Delete and recreate ExternalSecret (triggers resync)
kubectl delete externalsecret gitlab-s3-credentials -n gitlabbda
kubectl apply -f manifests/externalsecret-s3-credentials.yaml

---

Deployment Operations

Restart Deployments

# Specific deployment
kubectl rollout restart deploy/gitlab-webservice -n gitlabbda

# All deployments
kubectl rollout restart deploy --all -n gitlabbda

# StatefulSet restart (delete pod, recreates automatically)
kubectl delete pod gitlab-gitaly-0 -n gitlabbda
kubectl delete pod redis-0 -n gitlabbda

Scale Deployments

# Scale up/down
kubectl scale deploy/gitlab-webservice -n gitlabbda --replicas=3

# Scale to zero (maintenance mode)
kubectl scale deploy --all -n gitlabbda --replicas=0

# Scale back to normal
kubectl scale deploy/gitlab-webservice -n gitlabbda --replicas=2

Check Rollout Status

# Deployment rollout status
kubectl rollout status deploy/gitlab-webservice -n gitlabbda

# Rollout history
kubectl rollout history deploy/gitlab-webservice -n gitlabbda

# Undo rollout (rollback)
kubectl rollout undo deploy/gitlab-webservice -n gitlabbda

---

ArgoCD Operations

Check Application Status

# GitLab BDA application
kubectl get application gitlab-bda -n argocd

# Detailed status
kubectl describe application gitlab-bda -n argocd

Sync Application

# Via kubectl
kubectl patch application gitlab-bda -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{}}}'

# Via argocd CLI (if installed)
argocd app sync gitlab-bda

Check Sync Waves

# Show resources with sync waves
kubectl get all -n gitlabbda --show-labels | grep sync-wave

# Resources in specific wave
kubectl get all -n gitlabbda -l argocd.argoproj.io/sync-wave=3

---

Backup & Restore

GitLab Backup

Create backup:

# Exec into Toolbox
kubectl exec -it deploy/gitlab-toolbox -n gitlabbda -- bash

# Create backup (inside pod)
gitlab-backup create

# List backups in S3
gitlab-rake gitlab:backup:list

Restore backup:

# Stop GitLab (prevent writes during restore)
kubectl scale deploy --all -n gitlabbda --replicas=0

# Exec into Toolbox
kubectl exec -it deploy/gitlab-toolbox -n gitlabbda -- bash

# Restore (inside pod)
gitlab-backup restore BACKUP=<timestamp>_gitlab_backup.tar

# Start GitLab
kubectl scale deploy --all -n gitlabbda --replicas=1

PostgreSQL Backup

Manual backup:

# Trigger backup via CNPG
kubectl annotate cluster gitlab-postgres -n gitlabbda \
  cnpg.io/reconciliationLoop="backup-$(date +%s)" --overwrite

# Check backup status
kubectl get backups -n gitlabbda

Point-in-time recovery (PITR):

# Update Cluster spec
spec:
  bootstrap:
    recovery:
      source: gitlab-postgres
      recoveryTarget:
        targetTime: "2025-10-27 10:00:00"

---

Monitoring & Alerts

Check Prometheus Metrics

# Port-forward to Prometheus (cluster-wide)
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

# Open browser: http://localhost:9090
# Query: up{namespace="gitlabbda"}

Check ServiceMonitors

# GitLab BDA ServiceMonitors
kubectl get servicemonitor -n gitlabbda

# Check metrics endpoint
kubectl exec -it gitlab-postgres-1 -n gitlabbda -- curl http://localhost:9187/metrics

Query Loki Logs (via CLI)

# Port-forward to Loki
kubectl port-forward -n monitoring svc/loki 3100:3100

# Query logs (logcli, if installed)
logcli query '{namespace="gitlabbda", app="gitlab"}' --since=1h

# Or via curl
curl -G 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={namespace="gitlabbda"}' \
  --data-urlencode 'start=2025-10-27T10:00:00Z'

---

Emergency Procedures

Force Delete Stuck Pod

# Delete with grace period 0
kubectl delete pod <pod-name> -n gitlabbda --grace-period=0 --force

Cordon Node (Drain Pods)

# Mark node unschedulable
kubectl cordon <node-name>

# Drain pods from node
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Uncordon (allow scheduling again)
kubectl uncordon <node-name>

Delete Namespace (DANGER)

# This will delete ALL resources in namespace!
kubectl delete namespace gitlabbda

# Safer: Delete via ArgoCD Application
kubectl delete application gitlab-bda -n argocd

---

Useful Aliases

Add to ~/.bashrc or ~/.zshrc:

# Namespace alias
alias kg='kubectl -n gitlabbda get'
alias kd='kubectl -n gitlabbda describe'
alias kl='kubectl -n gitlabbda logs'
alias ke='kubectl -n gitlabbda exec -it'

# Component-specific
alias kgp='kubectl -n gitlabbda get pods'
alias kgs='kubectl -n gitlabbda get svc'
alias kgi='kubectl -n gitlabbda get ingress'

# Logs with follow
alias klf='kubectl -n gitlabbda logs -f'

# Top
alias ktp='kubectl -n gitlabbda top pods'
alias ktn='kubectl top nodes'

Usage:

kgp  # Same as: kubectl -n gitlabbda get pods
klf gitlab-webservice-xxx  # Follow logs

---

Quick Troubleshooting

Pod CrashLoopBackOff

# 1. Check pod status
kubectl describe pod <pod-name> -n gitlabbda

# 2. Check previous logs (from crash)
kubectl logs <pod-name> -n gitlabbda --previous

# 3. Check events
kubectl get events -n gitlabbda --sort-by='.lastTimestamp' | tail -n 20

Service Not Accessible

# 1. Check service endpoints
kubectl get endpoints <service-name> -n gitlabbda

# 2. Test connectivity from another pod
kubectl run -it curl-test --image=curlimages/curl --rm -- curl http://<service-name>.<namespace>.svc:8080

# 3. Check ingress
kubectl describe ingress <ingress-name> -n gitlabbda

High Resource Usage

# 1. Check top consumers
kubectl top pods -n gitlabbda --sort-by=memory

# 2. Check resource requests/limits
kubectl describe pod <pod-name> -n gitlabbda | grep -A 5 "Limits:\|Requests:"

# 3. Check node pressure
kubectl describe nodes | grep -A 10 "Conditions:\|Allocated resources:"

---

Summary

Most common commands:

• Status: kubectl get pods -n gitlabbda

• Logs: kubectl logs <pod> -n gitlabbda --tail=50

• Exec: kubectl exec -it <pod> -n gitlabbda -- bash

• Resources: kubectl top pods -n gitlabbda

• Restart: kubectl rollout restart deploy/<name> -n gitlabbda

For troubleshooting workflows, see Troubleshooting Reference.