Skip to main content

Observability: platform operator guide

Audience: Cluster operators and platform engineers
Scope: Day-2 operations for the shared observability stack

1. Overview​

The platform ships a shared, multi-tenant observability stack covering metrics, logs, and alerting for all Kube-DC Organizations on a cluster. The key properties from an operator perspective:

  • Shared services with tenant-aware scoping. Grafana, Mimir, Loki, and Alertmanager are shared. Identity claims, datasource headers, and backend tenant IDs scope each request; operators must keep those controls aligned and test them.
  • Organizations are tenants; Projects are data scopes. Metrics and logs are stored under namespace-derived backend tenant IDs. An Organization view federates its Organization namespace and accessible Project backing namespaces.
  • GitOps managed. Component versions, dashboards, alert rules, and capacity limits live in kube-dc-fleet. Apply durable changes through Git and Flux.
  • Controller managed. Organization reconciliation provisions Grafana state and datasource scope. Readiness still depends on Keycloak, the authorization path, storage, and the observability backends.

1.1 Component overview​

ComponentRole
MimirMulti-tenant time-series store. Metrics use namespace-derived backend tenant IDs and Organization views query the authorized IDs through tenant federation.
LokiMulti-tenant log store. Pod logs from eligible nodes are routed to the backend tenant ID derived from their source namespace.
Alertmanager (Mimir's)Multi-tenant alerting. Each Organization manages its alert routes through Grafana.
GrafanaSingle deployment with one Grafana Organization per Kube-DC Organization. Platform admins log in through SSO; Organization users enter through the Kube-DC console.
kube-prometheus-stackPrometheus hot tier, cluster-level alerting rules, and Grafana. Retention is configured per cluster.
AlloyThe log collector runs as a DaemonSet on eligible nodes; the metrics scraper runs as a Deployment.

1.2 Fleet repo layout​

All observability configuration lives under kube-dc-fleet:

platform/monitoring/
  mimir/                  # Mimir HelmRelease + values
  loki/                   # Loki HelmRelease + values
  prom-operator/          # kube-prom-stack HelmRelease (Grafana + Prometheus + Alertmanager)
  alloy/                  # Alloy DaemonSet (logs)
  alloy-metrics/          # Alloy Deployment (metrics)
  cortex-tenant/          # Metrics write-path proxy
  grafana-pg/             # CNPG PostgreSQL backend for Grafana
  observability-routes/   # Envoy Gateway HTTPRoutes + auth policies
  routes/                 # Grafana ingress HTTPRoute
  dashboards/             # Platform dashboards (JSON + kustomize)

clusters/<cluster>/platform/
  kustomization.yaml      # Per-cluster overlay (Grafana settings, secrets)

The per-tenant metrics write path is enabled per cluster. cortex-tenant

  • alloy-metrics (the components that route each Project's metrics into its own Mimir tenant) are not in the shared platform/monitoring root. Their cloud-sized values would over-provision small clusters. A capable cluster opts in with a monitoring-writepath.yaml Flux Kustomization that pulls the shared platform/monitoring-writepath bundle. kube-dc bootstrap init scaffolds this by default for new installs wherever Mimir is present (any non-disabled object-storage mode), so a fresh cluster's tenant Grafana Orgs show metrics, and not only logs, out of the box. Without it, tenant metrics dashboards render empty while logs work.

2. Product and backend identity model​

2.1 Product and storage identities​

IdentityEffective view
Kube-DC OrganizationFederated metrics and logs from the Organization namespace and its authorized Projects
ProjectMetrics and logs for the Project backing namespace, named {organization}-{project}
system backend tenant IDExplicitly allowlisted platform namespaces and cluster-scoped data; platform-administrator use only

The observability backends use namespace-derived backend tenant IDs as storage and request-routing keys. They are implementation details, not additional product tenants: the product tenant is the Organization, and a Project is its governed workload boundary.

Routing is automatic after the collectors and controller reconcile:

  • metrics use the Kubernetes namespace label to select a backend tenant ID;
  • logs are tagged with their source namespace during collection;
  • an explicit allowlist collapses platform namespaces such as monitoring, kube-system, and kube-dc into the system backend tenant ID;
  • metrics without a namespace label and cluster-wide Kubernetes events also route to system;
  • Organization datasource configuration federates the relevant backend tenant IDs.

2.2 Adding or removing a Project​

No separate observability registration is required. When a Project changes, kube-dc-manager updates the Organization's Grafana datasource scope. Allow for controller reconciliation, collector discovery, and the next scrape or log batch before expecting data.

3. Dashboards​

3.1 How dashboards work​

Platform dashboards are stored as JSON files in kube-dc-fleet/platform/monitoring/dashboards/ and deployed as Kubernetes ConfigMaps through Kustomize. The kube-dc-manager controller distributes them to the appropriate Grafana Organizations on every reconciliation.

Dashboards are declarative: the Git JSON is the source of truth. Manual edits made in the Grafana UI are overwritten on the next reconcile. Organizations that need custom dashboards should create new ones in their Grafana Organization. Those dashboards are persisted in the Grafana database and are not touched by the controller.

3.2 Shipped dashboards​

DashboardGrafana folderShown to
Namespace Resource UsagePlatformAll Organizations (home dashboard)
Logs ExplorerPlatformAll Organizations
Active AlertsPlatformAll Organizations
Shared GPU WorkloadsPlatform / AcceleratorsAll Organizations
Storage / CephStoragePlatform administrators only
GPU OperationsPlatform / AcceleratorsPlatform administrators only

3.3 Adding a platform dashboard​

  1. Create the dashboard JSON.
    Export from Grafana (Share → Export → Save to file) or author from scratch.
    Requirements:

    • Must have a stable, unique uid field (for example, kube-dc-my-new-dashboard).
    • Use the well-known datasource UIDs. Do not embed a datasource name:
      • mimir: federated metrics (queries across all Organization backend tenant IDs)
      • mimir-alerts: alert rule management for one backend tenant ID at a time
      • loki: federated logs
      • alertmanager: Alertmanager API
    • Use $namespace template variable populated through label_values(kube_pod_info, namespace) (not kube_namespace_created, which includes terminated namespaces).
    • Set "editable": false for platform-owned dashboards.

  2. Add the JSON to the fleet repo.

    kube-dc-fleet/platform/monitoring/dashboards/<my-dashboard>.json

  3. Register it in the kustomization.
    Edit kube-dc-fleet/platform/monitoring/dashboards/kustomization.yaml and add an entry under configMapGenerator:

    - name: dashboard-my-new-dashboard
      files:
        - my-dashboard.json
      options:
        labels:
          kube-dc.com/grafana-dashboard: "true"
        annotations:
          kube-dc.com/grafana-folder: "Platform"
          kube-dc.com/grafana-scope: "all-tenants"   # see §3.4

  4. Commit and push. Watch the Flux Kustomization and the Organization controller logs until both reconciliation loops have observed the change.

  5. Request a new Organization reconcile if needed:

    kubectl annotate organization <org-name> reconcile-trigger="$(date)" --overwrite -n <org-name>

3.4 Scope annotation​

Control which Grafana Organizations receive a dashboard through the kube-dc.com/grafana-scope annotation:

ValueDistributed to
all-tenants (default)Every Kube-DC Organization's Grafana Organization
main-onlyGrafana Organization 1 only (platform admin view)
tenants:acme,foo,barExplicit list of Kube-DC Organization slugs

3.5 Setting a home dashboard​

Add the annotation kube-dc.com/grafana-home: "true" to a dashboard's ConfigMap entry. Only one dashboard per scope should carry this annotation; if multiple do, the last one wins (alphabetical ConfigMap order).

3.6 Editing an existing dashboard​

Edit the JSON file in kube-dc-fleet/platform/monitoring/dashboards/, commit, and push. Keep the uid field unchanged. A changed UID creates a new dashboard and the old one to remain (orphaned). If you need to retire a dashboard, delete the JSON and ConfigMap entry; then manually delete it from Grafana Organizations or wait for the next full re-provision.

4. Alert rules​

To connect platform alerts to Slack and Opsgenie, follow Set up alert notifications with Fleet. That guide covers per-installation configuration, console activation, delivery tests, and rollback for the management cluster's Alertmanager.

4.1 Where rules live​

Rule typeStorage locationWho manages
Platform rules (cluster health, node, kube-system)Prometheus PrometheusRules, evaluated by kube-prometheus-stack PrometheusManaged through prom-operator/values-configmap.yaml or additional PrometheusRule CRDs
Organization alert rulesMimir Ruler, stored under backend tenant IDs in S3Organization users (Grafana Alert Rules UI) or operators through mimirtool

4.2 Adding or changing platform alert rules​

Platform alerting rules are part of the kube-prometheus-stack chart. Add a PrometheusRule manifest in kube-dc-fleet/platform/monitoring/prom-operator/ (or through a kustomize overlay) and commit. The Prometheus Operator observes the resource and updates the selected rule configuration through its normal reconciliation loop.

The following example adds a custom platform rule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: platform-custom-rules
  namespace: monitoring
  labels:
    release: prom-operator    # must match the HelmRelease label selector
spec:
  groups:
    - name: platform.custom
      interval: 1m
      rules:
        - alert: MyAlert
          expr: |
            sum(kube_pod_container_status_restarts_total{namespace="kube-dc"}) > 50
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High restart count in kube-dc"

4.3 Managing backend alert rules through mimirtool​

For bulk operations across backend tenant IDs (for example, pushing a common baseline rule set):

# List all rule groups for one backend tenant ID
mimirtool rules list \
  --address https://mimir-ruler.kube-dc.cloud \
  --id <backend-tenant-id>

# Upload rule groups for one backend tenant ID
mimirtool rules load ./rules/my-rules.yaml \
  --address https://mimir-ruler.kube-dc.cloud \
  --id <backend-tenant-id>

Authentication requires a valid Keycloak access token from the master realm (platform admin group). See the kube-dc CLI docs for obtaining tokens.

4.4 Default alertmanager routing​

Each Kube-DC Organization has its own Alertmanager configuration managed through Grafana (Contact Points and Notification Policies). Platform-level Alertmanager config (for the system backend tenant ID) is set in prom-operator/values-configmap.yaml under alertmanager.config.

5. Capacity and retention​

Capacity values are deployment configuration, not product constants. Read the base values and the selected cluster overlay together before changing them:

AreaFleet source
Mimir limits and retentionplatform/monitoring/mimir/values-configmap.yaml plus the cluster overlay
Loki limits and retentionplatform/monitoring/loki/values-configmap.yaml plus the cluster overlay
Prometheus hot-tier retentionplatform/monitoring/prom-operator/ plus the cluster overlay
Object-store quotasclusters/<cluster>/cluster-config.env and the rendered OBC resources

Important dimensions include active series, ingestion rate and burst, block retention, log ingestion rate and burst, stream cardinality, and log retention. Increase them only after checking object-store capacity, compaction, query latency, and failure recovery. A backend tenant ID normally represents one namespace, not an entire Organization.

5.1 Grafana database​

Grafana uses the grafana-pg CloudNativePG cluster in the monitoring namespace. Confirm the live PVC size and backup policy rather than relying on a copied estimate:

kubectl -n monitoring get cluster grafana-pg
kubectl -n monitoring get scheduledbackup grafana-pg-daily -o yaml
kubectl -n monitoring get backup --sort-by=.metadata.creationTimestamp

The current base Fleet manifest schedules grafana-pg-daily at 02:30 UTC and retains backups for 30 days. A cluster overlay can change that. To request an immediate run through the scheduled policy:

kubectl -n monitoring annotate scheduledbackup grafana-pg-daily   cnpg.io/immediateBackup="true" --overwrite

5.2 Metrics and log object storage​

Mimir and Loki use S3-compatible buckets provisioned through ObjectBucketClaims. Inspect the live claims and their backing storage before raising retention:

kubectl -n monitoring get obc
kubectl -n monitoring describe obc mimir-blocks
kubectl -n monitoring describe obc loki-chunks

Do not copy quota values from another cluster: storage topology, ingestion, and retention differ between installations.

6. Grafana administration​

6.1 Logging in as a platform admin​

Navigate to https://grafana.<domain>/login and click Sign in with Keycloak. This uses the master realm. In the current Grafana configuration, members of the master-realm admin group receive GrafanaAdmin. The Envoy observability authorization policy must recognize the same platform-admin claim; verify both paths after identity or monitoring changes.

6.2 Inspecting an Organization's Grafana space​

  1. Log in as platform admin (§6.1).
  2. Open the Organization switcher (top-left globe icon or through Admin → Organizations).
  3. Switch to the target Grafana Organization. You now see its dashboards, datasources, and alert rules as its users do.
  4. Switch back to Main Org. when finished to avoid accidental edits in the Organization's space.

6.3 Manually triggering Organization reconciliation​

If an Organization's dashboards, datasources, or membership is stale, force a reconcile:

kubectl annotate organization <org-slug> \
  reconcile-trigger="$(date)" --overwrite -n <org-slug>

Watch the kube-dc-manager logs for the result:

kubectl logs -n kube-dc deployment/kube-dc-manager -c manager --follow | grep <org-slug>

6.4 Token lifetime​

Per-Organization realms default to 15-minute access tokens in internal/organization/helpers.go. The Fleet bootstrap workflow ensures that the master realm is at least 15 minutes through bootstrap/setup-keycloak-oidc.sh. Inspect the live realm before assuming an exact value because an operator can configure a longer lifetime.

Shorter tokens reduce the replay window but increase refresh and login pressure. Treat a lifetime change as identity configuration and test the console, Grafana, CLI, and long-running API clients.

6.5 Kube-DC component logs​

The manager and the backend (API and worker) write one JSON object per line. Alloy ships them to Loki in the system tenant, and Loki detects the level from the level field (detected_level).

Chart valueDefaultEffect
manager.logFormatjsonconsole for human-readable lines
manager.logLevel1info or error; higher numbers add debug verbosity
backend.logFormatjsontext for [LEVEL] ts - msg lines
backend.logLevelinfodebug, info, warn, error
backend.accessLogtrueOne line per HTTP request; probes and /metrics are skipped

Every backend response carries an X-Request-Id header. The backend uses the gateway's id when there is one. All lines logged while the request runs carry that id as request_id, with realm and user once the caller is authenticated. The audit log records the same id.

Useful queries in Grafana Explore (system tenant):

# Manager errors, with the object being reconciled (the JSON `namespace`
# field is renamed namespace_extracted next to the stream label)
{namespace="kube-dc", container="manager"} | json | level="error"
  | line_format "{{.controller}} {{.namespace_extracted}}/{{.name}}: {{.msg}} {{.error}}"

# Everything one API request logged (id from the X-Request-Id response header)
{namespace="kube-dc"} | json | request_id="<id>"

# Slow or failing API requests
{namespace="kube-dc"} | json | component="access" | status >= 500 or duration_ms > 2000

# Backend errors by component over the last hour
sum by (component) (count_over_time({namespace="kube-dc"} | json | level="error" [1h]))

7. Troubleshooting​

7.1 Organization users see no data in dashboards​

  1. Verify the Grafana Organization. Its name appears in the top-left and should match the user's Kube-DC Organization slug. If it does not, the membership was not provisioned; trigger reconciliation (§6.3).

  2. Check the Mimir datasource. As platform admin, switch to the affected Grafana Organization and open Configuration → Datasources. The Mimir datasource should exist and the Custom HTTP Headers section should show an X-Scope-OrgID value containing the Organization ID and its Project backing namespace IDs (for example, shalb|shalb-docs|shalb-jumbolot). If it shows only the Organization ID, trigger reconciliation (§6.3) after confirming the Projects exist.

  3. Check the metrics write path. Look for rejected writes or tenant-header errors for the backing namespace in Mimir distributor logs:

    kubectl logs -n monitoring \
      -l app.kubernetes.io/component=distributor \
      --tail=100 | grep <backing-namespace>

    No matching error does not prove ingestion. Check the Alloy metrics collector next:

    kubectl logs -n monitoring \
      -l app.kubernetes.io/name=alloy-metrics --tail=100

7.2 Alert rules tab shows 404 or is empty​

This happens when the Mimir Alerts datasource (the per-backend-tenant alerting datasource) is missing or misconfigured. As platform admin:

  1. Switch to the affected Grafana Organization.
  2. Open Configuration → Datasources and confirm Mimir Alerts exists alongside Mimir.
  3. If it's missing, trigger reconciliation (§6.3).

Also verify the mimir-ruler deployment is running:

kubectl get pods -n monitoring -l app.kubernetes.io/component=ruler

The datasource must use the chart gateway, not the Ruler Service directly:

http://mimir-nginx.monitoring.svc/prometheus

An empty Alert Rules list can be valid when the tenant has authored no Mimir rules. It is different from the Active Alerts dashboard, which shows platform-Prometheus alerts exported read-only through Alloy. To verify that export, compare a firing namespace-scoped series in platform Prometheus with the same series in the project's Mimir tenant:

kubectl -n monitoring port-forward svc/prom-operator-prometheus 9090:9090
curl -G http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=ALERTS{alertstate="firing",namespace="<org>-<project>"}'

kubectl -n monitoring port-forward svc/mimir-nginx 8080:80
curl -G -H 'X-Scope-OrgID: <org>-<project>' \
  http://127.0.0.1:8080/prometheus/api/v1/query \
  --data-urlencode 'query=ALERTS{alertstate="firing"}'

The second query should contain the same alert within two Alloy scrape intervals. Platform paging remains independent of tenant Mimir silences.

7.3 Dashboards missing from a Grafana Organization​

Dashboards are distributed on every Organization reconciliation. If they are missing after a new dashboard was added to the Fleet repository:

  1. Confirm Flux has reconciled the ConfigMap: 
    flux get kustomization platform -n flux-system
    kubectl get cm -n monitoring -l kube-dc.com/grafana-dashboard=true
  2. Trigger reconciliation (§6.3).
  3. Check the kube-dc-manager logs for ensureDashboards errors. The most common cause is a dashboard JSON missing the uid field.

7.4 Grafana pod keeps restarting​

Check the CNPG grafana-pg cluster status first. Grafana crash-loops if the PostgreSQL backend is unavailable:

kubectl get cluster -n monitoring grafana-pg
kubectl get pods -n monitoring -l cnpg.io/cluster=grafana-pg

If PostgreSQL is healthy, check Grafana pod events and logs:

kubectl describe pod -n monitoring -l app.kubernetes.io/name=grafana
kubectl logs -n monitoring -l app.kubernetes.io/name=grafana --tail=50

7.5 Alloy / metrics pipeline​

Check each layer:

# Alloy metrics scraper
kubectl logs -n monitoring -l app.kubernetes.io/name=alloy-metrics --tail=50

# cortex-tenant proxy (translates __tenant_id__ to per-tenant writes)
kubectl logs -n monitoring -l app.kubernetes.io/name=cortex-tenant --tail=50

# Mimir distributor
kubectl get pods -n monitoring -l app.kubernetes.io/component=distributor

If cortex-tenant is OOMKilled, inspect its current working set and restart behavior, then adjust the memory request and limit in platform/monitoring/cortex-tenant/values-configmap.yaml. Verify the pipeline after Flux applies the change.

7.6 Mimir ring health alerts​

The MimirIngesterRingMembersMismatch and MimirRingMembersMismatch alerts fire when ingester or ruler pod counts diverge from what the ring expects. Usually transient during rolling restarts. If persistent:

kubectl get pods -n monitoring -l app.kubernetes.io/component=ingester
kubectl get pods -n monitoring -l app.kubernetes.io/component=ruler

Look for Pending or CrashLoopBackOff pods.

8. Component upgrades​

Component versions are pinned per cluster in clusters/<cluster>/cluster-config.env. Upgrade procedure for each component:

ComponentVariableNotes
kube-prometheus-stack (Grafana + Prometheus)PROM_OPERATOR_VERSIONTest in staging first. Grafana major versions may require DB schema migration.
MimirMIMIR_VERSIONCheck Mimir upgrade docs for breaking compactor/store-gateway changes.
LokiLOKI_VERSIONCheck chunk format compatibility; breaking changes require re-ingest.
AlloyALLOY_VERSIONValidate log pipeline config syntax after upgrade.

General process:

  1. Bump the version in cluster-config.env.
  2. Commit and push.
  3. Flux reconciles the HelmRelease.
  4. Verify pods come healthy: kubectl get pods -n monitoring.
  5. Check the observability dashboards for any ingestion gap.

Do not bump multiple stateful components (Mimir, Loki, CNPG) in the same commit.

9. Keycloak bootstrap​

The Fleet bootstrap/setup-keycloak-oidc.sh workflow reconciles the Grafana OIDC client and other platform clients in the master realm. Run the script that belongs to the checked-out Fleet version:

cd <fleet-repository>
KUBECONFIG=<management-cluster-kubeconfig>   bash bootstrap/setup-keycloak-oidc.sh <cluster>

Review and push any generated encrypted Fleet changes, then verify both direct Grafana login and datasource queries.

10. Reference​

TopicPath
Mimir HelmRelease + valuesplatform/monitoring/mimir/
Loki HelmRelease + valuesplatform/monitoring/loki/
Grafana / Prometheus (kube-prom-stack)platform/monitoring/prom-operator/
Dashboard JSON sourcesplatform/monitoring/dashboards/
Per-cluster version pinsclusters/<cluster>/cluster-config.env