Security model
Kube-DC combines identity, Kubernetes authorization, admission policy, and Kube-OVN routing controls. No single layer is a complete tenant boundary, and platform administrators remain privileged across Organizations.
This page describes the controls applied to Projects on the management cluster. A Managed Cluster has its own Kubernetes API, RBAC, and workload security configuration.
GPU workloads add privileged node components and deployment-specific isolation assumptions. See the GPU security threat model before enabling a GPU profile.
Control layers
| Layer | Protects | Scope |
|---|---|---|
| OIDC and Keycloak | User identity and group claims; where a login code may be returned | Platform and Organization realms |
| Kubernetes RBAC | Which resources a user can read or change | Organization namespaces and Project backing namespaces |
| ValidatingAdmissionPolicy | Dangerous Pod fields, exec/attach, protected annotations, quota, and selected accelerator rules | Resources selected by each policy or binding |
| Kube-OVN VPCs | Primary Project network separation | One VPC and workload subnet per Project |
| Router-policy isolation | Traffic on shared cloud/public external networks | Project VPCs when ingress or egress isolation is enabled |
| Workload policy | Application-specific traffic rules | Optional operator-provided Kubernetes NetworkPolicy; default Project Roles cannot author it |
Pod admission
Project backing namespaces carry the kube-dc.com/project label. The
restrict-pod-security-in-projects policy denies user-created or user-updated
Pods that request:
hostPathvolumes;- privileged containers or init containers;
hostNetwork;hostPID;hostIPC.
Trusted platform service-account namespaces, node identities, and
system:masters are excluded because controllers such as KubeVirt and Kamaji
must create infrastructure Pods in Project backing namespaces. Those exclusions are
part of the platform trust boundary; they are not a user-configurable bypass.
Users can still run ordinary Pods, Deployments, StatefulSets, Jobs, and DaemonSets, mount supported volume sources, and expose workloads through Services or Gateway routes.
Exec and attach
Pod exec and attach are unsupported in Projects. Current standard Role
templates omit pods/exec and pods/attach, and
restrict-pod-exec-in-projects denies CONNECT requests even if a custom or
stale Role grants the subresource.
Run administrative container tasks as a purpose-built Job that mounts the
required volume. Use the VM console or VNC for virtual machines. Platform
service accounts, nodes, and system:masters are exempt and must be protected
accordingly.
Protected annotations
The protect-kube-dc-resource-annotations policy prevents non-platform users
from changing annotations on Organization, Project, and
OrganizationGroup resources during UPDATE. Users can still perform the
specification changes allowed by their RBAC.
Cluster administrators can set controller-consumed annotations. For example:
kubectl -n <organization> annotate project <project> \
network.kube-dc.com/egress-allowlist="10.8.0.0/24" \
--overwrite
Treat annotations as privileged configuration: admission protects changes, but does not validate the business reason for an allowlist entry.
External-network isolation
Project VPC separation is the primary east-west boundary. Optional ingress and egress router policies add protection when multiple tenants share an external cloud or public subnet.
Egress isolation
When egress_network_isolation is enabled, Kube-DC drops Project traffic whose
destination is another address on a configured external subnet, except for:
- the external subnet gateway, which is needed for SNAT and internet access;
- external addresses owned by the Project;
- the global egress allowlist;
- the Project's
network.kube-dc.com/egress-allowlistentries.
This control does not block all outbound traffic. Internet destinations do not match the external-subnet drop rule and continue through the Project's SNAT path.
Ingress isolation
When ingress_network_isolation is enabled, Kube-DC drops traffic whose
source is a configured external subnet, except for:
- the external subnet gateway, needed for return traffic;
- the global ingress allowlist;
- the Project's
network.kube-dc.com/ingress-allowlistentries.
Ingress allowlists describe trusted source addresses. Unlike egress rules, they are not populated from Project EIPs automatically.
Platform configuration
The master configuration holds cluster-wide switches and allowlists:
{
"egress_network_isolation": true,
"egress_global_allowlist": ["10.8.0.0/24"],
"ingress_network_isolation": true,
"ingress_global_allowlist": ["192.0.2.10"],
"tenant_client_extra_redirect_uris": ["https://portal.example.com/auth/callback"]
}
tenant_client_extra_redirect_uris extends the OAuth redirect allow-list of
every Organization realm's kube-dc client. The manager registers the console
(<console_url>/*), the API reference (<backend_url>/*, rendered by the
chart from the backend hostname) and the CLI's loopback callback
(http://localhost:*, http://127.0.0.1:*), and converges existing realms to
that list (and the client's web origins to the console's origin) on each
Keycloak sync, so an entry added by hand in Keycloak does not survive. Extras
must be absolute http(s) URIs without user-info; a trailing * must follow
a /. A bare *, a host-prefix wildcard or another scheme is ignored and
logged; an extra that needs browser CORS on the token endpoint is not
supported (exchange the code server-side). The master realm's kube-dc-admin
client is held to the CLI loopback only by bootstrap/setup-keycloak-oidc.sh.
Project-specific values are comma-separated IP addresses or CIDRs:
kubectl -n <organization> annotate project <project> \
network.kube-dc.com/ingress-allowlist="192.0.2.10,198.51.100.0/28" \
network.kube-dc.com/egress-allowlist="10.8.0.0/24" \
--overwrite
Changes apply when the Project reconciles. Validate the resulting behavior from both directions; an allowlist does not replace application authentication, TLS, or a workload NetworkPolicy.
Project RBAC
Kube-DC creates four standard Roles in every Project backing namespace:
| Role | Intended access |
|---|---|
admin | Broad lifecycle access to supported Project resources plus namespaced Roles and RoleBindings; quota is read-only |
project-manager | Read and monitor Project resources and use the VM console; update or patch existing managed secrets, certificates, and database credential policies; create, update, or patch KMS keys; no Project, membership, or quota administration |
developer | Manage supported workloads, Services, VMs, and managed services; raw Kubernetes Secrets are get/list only |
user | Read-only Project visibility without raw Secret or VM-console access |
The exact rules come from the platform's default Role templates. Review live rules rather than inferring privileges from the role name:
kubectl -n <backing-namespace> get role \
admin developer project-manager user -o yaml
Namespaced RBAC does not grant access to cluster-scoped resources. Organization Groups create RoleBindings only for the selected Projects. See Multi-tenancy and access control.
Operate the admission policy
Admission policy and bindings are installed through the Fleet GitOps path. Inspect them with:
kubectl get validatingadmissionpolicy
kubectl get validatingadmissionpolicybinding
kubectl describe validatingadmissionpolicy restrict-pod-security-in-projects
Change exemptions or enforcement in the Fleet source, review the impact, and let Flux reconcile it. Deleting a live binding creates an immediate enforcement gap and Flux may recreate it; it is not a normal debugging procedure.
Policy denials appear in API responses and, when audit logging is enabled, in the API-server audit log. Capture the denied request, user, policy name, and message before changing policy.
Managed services placed inside a Project
A managed database or message broker can run in the tenant's own Project
namespace. A project admin holds broad rights there, so the platform draws
the boundary at admission rather than at RBAC: every object the platform
manages carries services.kube-dc.com/managed-by, admission policies keep
tenant identities off marked objects, and the manager's webhooks cover what a
policy cannot see: a tenant object aimed at a managed engine.
What that means in practice for an operator:
-
The engine's own operation APIs are the platform's. A CNPG
BackuporScheduledBackup, or a MariaDB backup/restore/SQL job, naming a managed engine is refused, whether it is created, changed, or deleted. Those run through the managed service's own operations, which carry approval, concurrency control and evidence, and deleting the schedule of a managed database would stop its backups silently. A tenant's own, unmanaged engine stays fully controllable. -
A tenant database may not read a managed one. A CNPG
Clusterthat bootstraps from a managed engine's backup, streams from it withpg_basebackup, replicates it, or subscribes to it is refused, as is aPublicationorSubscriptionnaming it: each is a way to act on the platform's engine without an operation. Restoring a managed database is an operation of that managed service.What this check is worth, precisely. It resolves the forms people write (the engine's Service names, a Service in the namespace by name or cluster IP, and
ExternalNamealiases) and refuses those. It does not stop someone who is trying: a selectorless Service with a hand-written EndpointSlice pointing at the engine's pods, anExternalNamethat leaves the cluster and resolves back, or any proxy the tenant runs, all reach the engine and are admitted. That is inherent once a tenant can choose arbitrary addresses in its own namespace, and it is accepted rather than papered over: what bounds the damage is the engine's own authorization and the platform's disk alerting. Treat the rule as protection against the wrong manifest, not against a determined tenant. The same-namespace trust model already says a project admin can reach its project's databases. -
A tenant PodDisruptionBudget cannot cover managed pods. Admission appends one
DoesNotExistrequirement per ownership marker to every tenant budget's selector, so it applies to that tenant's own pods and never to a platform engine's.minAvailable: 100%over engine pods would otherwise block the node drains maintenance needs. Selecting explicitly on the platform's or an engine operator's keys is refused outright; a null selector is left alone (inpolicy/v1it matches no pods).Two consequences are worth knowing. A budget written before the platform installed this rule is not changed until something updates it.
hack/audit-managed-service-boundary.shlists those and--applynarrows them (a compare-and-swap patch; it exits non-zero if any budget could not be narrowed, and 2 if it could not audit at all). And a GitOps tool that compares live objects against its source sees the appended requirements as drift: the tenant-facing guidance is to write the two requirements into their manifests, withignoreDifferences/driftDetection.ignore(or Argo'sServerSideDiff=true,IncludeMutationWebhook=true, both options, since server-side diff alone does not account for mutating webhooks) as the alternative. See Scaling and performance.Admission narrows a budget instead of refusing it. Refusal would end the mutation drift, but a refused apply leaves the old object live and the updated one unapplied. That is a louder divergence than drift: drift is noise, a broken apply is an incident.
-
Protection needs its enforcement.
projectPolicies.protectManagedServicesrequiresmanager.webhook.enabledandDenyinvalidationActions; the chart refuses the contradictory combinations rather than rendering an installation that looks protected. -
Fail-closed admission requires two manager replicas. With one replica, every rollout, drain or eviction is a window in which tenant pods, claims and Secrets cannot be created. Set
manager.replicaCount: 2on two nodes, or state the trade-off explicitly withmanager.singleReplicaAdmissionAccepted: trueon a development or single-node installation.
Which managed-service families are supported here
Protection is generic: the platform marks what it manages and the policies key on that marker, whatever the engine. The families an installation offers are the ones its catalog publishes. See Publish the catalog. Four things are not generic, and a family has to be checked against each:
-
The management-API route. Where
manager.infraAttachmentis enabled, a pod reaches the Kubernetes API only when the platform can prove what it is. That proof walks the list of engine roots: CNPGClusterandMariaDBby default, plus the roots of registered families. Every other pod gets the plain lock. An ordinary tenant pod lands in the Project's workload security group (from themanager.infraAttachment.securityGrouptemplate, for exampleinfra-lock-<project-namespace>) and cannot reachkubernetes.defaultat all. A managed CNPG engine's pod lands in the-api-clientvariant of that group, which has an extra route.A family whose pods consume the Kubernetes API therefore does not work on such a cluster until its engine root is in that list.
-
Operation CRs and reserved selector keys. Each engine's own operation API is named explicitly in the reference gate (CNPG's
Backup,ScheduledBackup,Publication,Subscription; MariaDB's backup, restore and SQL jobs), as are the selector keys reserved against tenant identities (cnpg.io/*,strimzi.io/*). A family with its own operation CRs or labels needs them added, or those objects are ungated. -
Quota attribution. The same classifier splits a Project's CPU into workload and platform overhead; an unclassified engine's pods count as tenant workload.
-
Retained-engine identity. When an instance is retained, its ManagedService is gone but its pods must still be classifiable. The runner writes an attestation the platform reads from a fixed namespace, and the same engine roots decide what it proves. A family that is retained on a cluster where
manager.infraAttachmentis enabled needs the same entry as the management-API route.
Families are registered per cluster, not compiled in. The manager loads a
service-family registry rendered from serviceFamilies.extra in the chart's
values. CNPG and MariaDB are the built-in defaults. Adding a family is a values
change that rolls the manager, not a Kube-DC release. A registry entry declares
these things:
- which engine kinds a pod's owner chain may lead to, and through which controller kinds;
- which operator selector keys are reserved;
- which operation CRs are gated by their target.
Trust is unchanged. Every hop is still UID-verified, and the root still has to carry the platform's marker with a ManagedService (or a retained-engine attestation) behind it. The registry only says what a chain may look like, and it says so per root: the hops a chain walked must all be in the through list of the root it reaches.
The registry is validated by these rules:
- The manager decodes it strictly at load. Roots and operations must be unique, every through kind must be guarded by the owner-forgery policy by its resource, and label references must be reserved. A manager whose configured registry file is missing or invalid does not start.
- The chart validates every identifier and every semantic rule in a family entry before a policy renders. The manager validates them again when it loads the file.
- A family may not claim the platform's own kinds as roots or hops.
- An operator that creates the children of a through kind itself (Strimzi's pod sets) is admitted under that kind by an explicit flag. It is never admitted under Deployments or ReplicaSets, whose children are kube-system's.
Install the operator before you register the family. At startup the manager checks each declared plural against API discovery and refuses to start on one the server does not serve as declared. The check repeats every minute. A hop whose mapping stops matching is walked by nobody until it matches again.
A family also declares its whole API surface. The entire group/version of its operations is routed to the reference gate. The gate admits the resources the family lists as passthrough and refuses the rest to tenants. An operation that an operator upgrade renames is therefore unknown, not unguarded, and a later API version of it is refused until the registry names it.
The family's operator ServiceAccounts are trusted inside those groups only at
the reference gate. Their namespace is, like cnpg-system, an exempt platform
identity for the marker policies and the child mutator. The registry derives
it, so registering a family is the whole of the configuration.
The platform trusts two kinds of owner edge:
- the platform's own kinds and apps/batch workloads, through controller references only (a tenant's garbage-collection reference to their own Deployment is theirs);
- the engine kinds (CNPG Cluster, MariaDB, and every registered family's roots and custom through kinds), through any reference, because Strimzi owns its pod sets through a non-controller reference to the node pool, and a tenant has no business referencing a platform engine as an owner.
The classifier follows the same edges.
A managed CNPG Cluster's status is written by its own instance and nobody else. The managed-services policy excludes that subresource, and the reference gate judges it with UID-verified provenance: the instance ServiceAccount is marked and owned by the Cluster, and the token is bound to a pod the Cluster owns. A name alone is not proof, because a tenant can pre-own a name. The judge answers on its own webhook path, so an older manager pod refuses the write during a rolling upgrade instead of admitting it. The policy and the webhook are separate objects that the API server caches independently. The policy excludes the subresource only after every cluster serves the judge, because a relaxation observed before the judge exists would admit the write unjudged.
CNPG does not adopt a pre-existing ServiceAccount, so a tenant may not create one under a name the platform has claimed for an engine. If an account was created before the claim, the platform refuses that engine's instance and names the reason. The platform must detect the collision before it builds the engine.
The reference gate refuses a tenant object that names no engine at all unless the family declared such an object inert. It also refuses a resource it has no rule for instead of guessing. The one surface still written per family in CEL is the per-instance ServiceAccount authorization. Only an operator that runs instances under a per-instance account needs it.
The supported boundary is therefore a conjunction, not one test. A family ships from the managed-services release alone only if all of these hold (clauses 1, 3 and 4 through a registry entry; clause 2 through a registry entry unless the operator needs per-instance authorization):
- its engine pods do not need the Kubernetes API on clusters where
manager.infraAttachmentis enabled (they may anywhere else); - it brings no operation CRs of its own that act on the engine, or they are already named in the reference gate;
- it introduces no selector keys that must be reserved against tenants;
- it needs no retained-engine classification on a cluster where
manager.infraAttachmentis enabled.
CNPG is a built-in engine root, so PostgreSQL satisfies all four clauses. For
every other family, check each clause on a cluster where
manager.infraAttachment is enabled before you offer the family there. A
family that fails a clause needs a Kube-DC platform release together with the
managed-services release.
Placement: a managed service runs either in the tenant's Project namespace (shared, protected as this section describes) or on a separate data plane. Private per-instance namespaces inside a Project are not implemented. A sibling namespace does not inherit the Project's networking, quota accounting or identity, and adding a label does not create them.
What it does not mean: the model protects integrity, not confidentiality. A project admin can read and mount the engine's Secrets, including its CA material, and the backup object store is the Project's own bucket. That is the same-namespace trust model working as designed: a tenant who can create workloads in a namespace can read what those workloads read. Use a separate data plane when a tenant must not be able to read its engine's credentials.
An operator that runs inside the Project. Some operators (Strimzi's entity operator) run in the project namespace under an account they create per instance. The policies see that account as a tenant. A registered family therefore declares the account's shape and the few writes it must make: the status of the family's operation objects, and the Secrets it derives from them. The manager admits exactly those writes, on a webhook path of its own, with the same UID-verified provenance as a managed PostgreSQL's status write:
- the account is marked and owned by the marked root;
- the token is bound to a pod the classifier recognizes as that instance's;
- the object is marked with the instance.
The policies hand those writes to the webhook only after every cluster serves that path.
All of these proofs read the object as it is at admission. A marker is
therefore only ever stamped when the platform creates an object, and is never
added to one that already exists. The policy
protect-managed-service-marker-introduction-in-projects refuses that
transition on UPDATE, on every resource and subresource, for every identity
except cluster admins. No service account is exempt, the platform's own
included. An operator whose reconcile meets a name a tenant pre-created fails
on it instead of adopting it. Neither the object's UID nor any token minted
against it ever becomes the platform's.
To migrate databases created before markers existed, run
hack/migrate-managed-db-markers. It recreates every authority-bearing child
under fresh UIDs instead of labeling what exists.
Continuous enforcement is the invariant
Every trust decision in this section that reads an owner reference or an
ownership marker rests on one platform invariant. That covers a pod's chain to
a managed engine through controller references, the marked root itself, and a
registered family's non-controller edge. The invariant: the owner-forgery
guard (reserve-platform-identities-in-projects), the marker policy
(protect-managed-service-markers-in-projects) and their Deny bindings are
continuously enforced, for CREATE and UPDATE, on every write-serving apiserver
and in every project namespace. A tenant object admitted while that was not
true can carry a marker or an owner reference the platform never granted, and
nothing in the classifier can tell it apart later.
The manager does not prove this invariant; it checks it. Every minute each
manager replica runs dry-run probes as a fixed synthetic tenant
(owner-guard-probe@kube-dc.internal) in the platform-owned canary namespace
kube-dc-owner-guard-canary. The probes are these:
- a plain Deployment create, which must be admitted;
- a Deployment with a non-controller owner reference to a managed engine kind, which the guard must refuse;
- a Deployment carrying a marker key, which the marker policy or the object-protection policy must refuse (both deny it, and the apiserver names only the first binding that denied, in no stable order);
- a marker stamped onto the chart's unmarked zero-replica target Deployment in the canary, refused the same way (the UPDATE path).
The non-controller edge is climbed only for objects created after the
owner-guard epoch. The epoch is the server-set creation time of an immutable
ConfigMap owner-guard-epoch in the manager's namespace, plus ten minutes. The
manager records it the first time the probes pass. A lapse the probes observe
retires that epoch, and a later one is recorded when enforcement is observed
again. An inconclusive tick withholds the edge. The probes cannot see a lapse
between two ticks, one overlapping an inconclusive tick, or one while no
manager runs.
Operate accordingly. Policy maintenance follows the order in this procedure. Maintenance means any of these:
- rolling the chart back to a release whose owner guard does not cover non-controller owner references;
- editing either policy or its binding;
- disabling
projectPolicies.reservePlatformIdentities.
-
Set
manager.nonControllerOwnerEdges: off, and wait untilkubectl -n kube-dc rollout status deployment/kube-dc-managerreports the Deployment rolled out. The same upgrade does these things:- It installs the off fence, a ValidatingAdmissionPolicy
(
owner-guard-off-fence). The fence refuses any manager pod created in the manager's namespace without the literal knob in itsmanagercontainer, whoever creates it. Leader election is unfenced, so without the fence an old controller-manager leader could still create an attested pod after every observation the off manager can make. - An off manager retires the epoch record every minute. Its first tick must succeed or it exits, so a completed rollout means the epoch is retired.
- The maintenance boundary, the ConfigMap
owner-guard-boundaryin the manager's namespace, is a lease. Each off replica acknowledges it (ack.<pod-uid>, a timestamp) only on a tick that observed all of these conditions hold:- the Deployment satisfies the controller's own DeploymentComplete
condition (at least as strict on replica counts as
rollout status); - none of its ReplicaSets desires, holds or can still create a pod without the knob (their controllers have observed the generation that zeroed them);
- no manager pod without the knob is listed (the knob is read from the
managercontainer alone); - the fence refuses an attested manager pod (a server-side dry run).
- the Deployment satisfies the controller's own DeploymentComplete
condition (at least as strict on replica counts as
- The acknowledgment comes only after all of that has held on that replica for a full ten-minute settle since the fence was first observed refusing. One apiserver's denial says nothing about another's policy cache.
- Any break, any inconclusive tick, or a restart starts that replica's window over and drops its acknowledgment. If the drop itself cannot be written, the acknowledgment expires.
- A replica that sees every listed manager pod acknowledged within the
last three minutes marks the boundary
complete.
- It installs the off fence, a ValidatingAdmissionPolicy
(
-
Before you weaken either policy, run
hack/owner-guard-boundary-check.shand require READY. Expect READY no earlier than ten minutes after the rollout completed. The check is one stateful operation with two samples ninety seconds apart (longer than a tick plus jitter):- Each sample is bracketed by two reads of the manager Deployment that must agree on UID, generation and revision, with the rollout status pinned to that revision. Every accepted sample therefore held at a single point in time.
- In each sample, the boundary's
stampmust equal that sample's manager Deployment UID and rollout revision.completeand every manager pod's acknowledgment must be within three minutes, and every manager pod must run off. - Between the samples, the Deployment (UID, generation, revision), the
boundary object, its stamp and the manager pod set must be unchanged,
while
complete(a fresh aggregate write) and every pod's acknowledgment must be strictly newer. This proves renewal, not survival, which two separate checks cannot give: an acknowledgment that a replica could no longer drop would otherwise pass twice. - An off manager deletes a boundary of any other stamp (from a previous maintenance, or from a previous incarnation of the Deployment).
During this procedure, never force-delete a manager pod, and never delete a Node object whose death is unproven. Both remove the Pod object before the process has stopped, and the quiescence check would then miss a live attested writer. If either happened, abort until the process or its node is proven stopped (fence the node first).
-
Do the maintenance, restore the policies, and confirm they are applied.
-
Clear the knob. The same upgrade removes the fence. Attested pods that Helm rolls a few seconds before that deletion lands are refused and then retried by their ReplicaSet. An attested manager that finds the boundary retires whatever record stands behind it, removes the boundary, and only then records a fresh epoch. It does this on every tick, including its first, so no epoch from before or during the maintenance is ever adopted, not even one written by a replica that overlapped the rollout.
The ten-minute settle is the bound that must hold for policy-cache convergence across apiservers plus clock skew. Keep the control plane NTP-synced. The reasoning uses two more invariants:
- A namespace is a project (carries
kube-dc.com/project) for the whole tenant-writable life of every object in it. A namespace must not become, or become again, a project while it holds objects tenants created when the selector was absent. - Objects reach the store only through admission. A storage restore that bypasses admission is outside the claim.
The synthetic probe user owner-guard-probe@kube-dc.internal is issued by no
authentication provider, and tenants cannot impersonate it (project roles
carry no impersonate). The canary admits nothing but dry runs from user
identities regardless.
An unplanned lapse of either policy is an incident, not a state the classifier detects. After a lapse, do these steps:
- Run
hack/audit-owner-edges.sh --apply. Registered custom-through objects with non-controller owners are recreated by their operators. - Run
hack/audit-managed-service-boundary.shto find marked objects that predate the rules. - Treat anything marked during the lapse as untrusted.
Boundaries and residual risk
- Platform controllers and
system:masterscan cross Project boundaries. - Compromise of a trusted service account can bypass the admission exclusions granted to that account.
- Underlay VLAN attachments inherit the physical network's isolation model.
- Network isolation does not replace encryption, application authorization, or backup.
- A Project is not a separate Kubernetes cluster. Use a Managed Cluster when a tenant needs its own cluster-scoped administration boundary.
- Managed services placed in a Project namespace are protected for integrity, not confidentiality; their credentials and backup bucket are readable by that Project's administrators.