Project Resources Documentation ¶

This document provides a comprehensive overview of all resources created by the Kube-DC Project controller, their finalizers, ownership patterns, and deletion dependencies.

Resource Creation Order ¶

When a Project is created, resources are synchronized in this order:

Namespace - Project namespace ({org}-{project})
VPC - Kube-OVN Virtual Private Cloud
EIp (Default Gateway) - External IP for project gateway
Subnet - Kube-OVN subnet for project pods
NetworkAttachmentDefinition - CNI network configuration
OvnSnatRule - SNAT rule for outbound traffic
Secrets - SSH keypairs and authorized keys
RBAC - Roles and RoleBindings
VpcDns - DNS configuration for VPC

Detailed Resource Breakdown ¶

1. Namespace ¶

Resource: v1.Namespace
Name: {organization}-{project} (e.g., shalb-envoy)
Finalizer: None (managed by Kubernetes)
Created by: NewProjectNamespace() in internal/project/res_namespace.go
Dependencies: None (first resource created)

2. VPC (Virtual Private Cloud) ¶

Resource: kubeovn.io/v1.Vpc
Name: {organization}-{project} (e.g., shalb-envoy)
Finalizer: kubeovn.io/kube-ovn-controller
Created by: NewProjectVpc() in internal/project/res_vpc.go
Dependencies: Namespace must exist
Configuration:
Static routes to external subnets
Extra external subnets based on egressNetworkType

3. EIp (External IP - Default Gateway) ¶

Resource: kube-dc.com/v1.EIp
Name: default-gw
Namespace: {organization}-{project}
Finalizer: eip.kube-dc.com/finalizer
Created by: NewProjectEip() in internal/project/res_eip_default.go
Dependencies: Namespace must exist
Ownership States:
Released: No active owners (initial state)
Shared: Has SNAT rule and/or LoadBalancer services as owners
Exclusive: Used by FIp resources

4. OvnEip (Underlying OVN External IP) ¶

Resource: kubeovn.io/v1.OvnEip
Name: {organization}-{project}-{external-subnet} (e.g., shalb-envoy-ext-public)
Finalizer: kubeovn.io/kube-ovn-controller
Created by: EIp controller via NewOvEipRes() in internal/eip/ovn_eip_res.go
Dependencies: EIp must exist, external subnet must be available
Labels:
network.kube-dc.com/eip: {namespace}.{eip-name}
Annotations:
kube-dc.com/ovn-eip-created-by-eip: {namespace}.{eip-name}

5. Subnet ¶

Resource: kubeovn.io/v1.Subnet
Name: {organization}-{project}-default (e.g., shalb-envoy-default)
Finalizer: kubeovn.io/kube-ovn-controller
Created by: NewProjectSubnet() in internal/project/res_subnet.go
Dependencies: VPC must exist
Configuration:
CIDR block from project spec
Associated with project VPC
Gateway IP (first IP in CIDR)

6. NetworkAttachmentDefinition ¶

Resource: k8s.cni.cncf.io/v1.NetworkAttachmentDefinition
Name: default
Namespace: {organization}-{project}
Finalizer: project.kube-dc.com/finalizer
Created by: NewProjectNad() in internal/project/res_nad.go
Dependencies: Subnet must exist
Configuration: Kube-OVN CNI configuration pointing to project subnet

7. OvnSnatRule ¶

Resource: kubeovn.io/v1.OvnSnatRule
Name: {organization}-{project} (e.g., shalb-envoy)
Finalizer: kubeovn.io/kube-ovn-controller
Created by: NewProjectSnat() in internal/project/res_snat.go
Dependencies: OvnEip must exist and have IP assigned
Configuration:
Links project subnet to external IP
Enables outbound internet access for pods

8. Secrets ¶

SSH Key Pair Secret ¶

Resource: v1.Secret
Name: ssh-keypair-default
Namespace: {organization}-{project}
Finalizer: project.kube-dc.com/finalizer
Created by: NewProjectKeyPairSeret() in internal/project/res_secret.go
Content: Generated SSH public/private key pair

Authorized Keys Secret ¶

Resource: v1.Secret
Name: authorized-keys-default
Namespace: {organization}-{project}
Finalizer: project.kube-dc.com/finalizer
Created by: NewProjectAuthKeySecret() in internal/project/res_secret.go
Content: SSH public keys for VM access

9. RBAC Resources ¶

Role ¶

Resource: rbac.authorization.k8s.io/v1.Role
Name: admin
Namespace: {organization}-{project}
Finalizer: project.kube-dc.com/finalizer
Created by: NewProjectRole() in internal/project/res_role.go
Permissions: Full access to project resources (pods, services, VMs, etc.)

RoleBinding ¶

Resource: rbac.authorization.k8s.io/v1.RoleBinding
Name: org-admin
Namespace: {organization}-{project}
Finalizer: project.kube-dc.com/finalizer
Created by: NewProjectRoleBinding() in internal/project/res_role_binding.go
Subject: {organization}:org-admin group

10. VpcDns Service ¶

Resource: v1.Service
Name: slr-vpc-dns-{organization}-{project}
Namespace: kube-system
Finalizer: None (managed by service controller)
Created by: NewProjectVpcDns() in internal/project/res_vpc_dns.go
Purpose: DNS resolution for VPC

Finalizers Summary ¶

Resource Type	Finalizer	Controller
Project	`project.kube-dc.com/finalizer`	kube-dc-manager
EIp	`eip.kube-dc.com/finalizer`	kube-dc-manager
OvnEip	`kubeovn.io/kube-ovn-controller`	kube-ovn-controller
Vpc	`kubeovn.io/kube-ovn-controller`	kube-ovn-controller
Subnet	`kubeovn.io/kube-ovn-controller`	kube-ovn-controller
OvnSnatRule	`kubeovn.io/kube-ovn-controller`	kube-ovn-controller
NetworkAttachmentDefinition	`project.kube-dc.com/finalizer`	kube-dc-manager
Secrets	`project.kube-dc.com/finalizer`	kube-dc-manager
Role	`project.kube-dc.com/finalizer`	kube-dc-manager
RoleBinding	`project.kube-dc.com/finalizer`	kube-dc-manager

Deletion Order and Dependencies ¶

When a Project is deleted, resources must be removed in reverse dependency order:

Phase 1: Application Resources ¶

Pods, Services, VMs - User workloads (deleted by users/operators)
FIp resources - Floating IPs (if any exist)

Phase 2: SNAT and Networking ¶

OvnSnatRule - Must be deleted before OvnEip
OvnEip - Must be deleted before EIp and Subnet

Phase 3: Project Infrastructure ¶

EIp - External IP resource
NetworkAttachmentDefinition - CNI configuration
Secrets - SSH keys and authorized keys
RBAC - Roles and RoleBindings
Subnet - Must be deleted before VPC
VPC - Virtual Private Cloud
VpcDns - DNS service
Namespace - Project namespace (last)

Common Deletion Issues ¶

Stuck Finalizers ¶

OvnEip: May get stuck if SNAT rule deletion fails
Subnet: May get stuck if pods are still running
EIp: May get stuck if project is deleted before EIp controller processes it

Manual Cleanup Commands ¶

# Remove stuck finalizers (use with caution)
kubectl patch ovn-snat-rule {name} -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl patch ovn-eip {name} -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl patch subnet {name} -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl patch eip {name} -n {namespace} -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl patch project {name} -n {org-namespace} -p '{"metadata":{"finalizers":null}}' --type=merge

EIp Ownership Patterns ¶

Ownership States ¶

Released: ownershipType: Released, owners: [] - No active users
Shared: ownershipType: Shared, owners: [...] - Multiple users (SNAT + Services)
Exclusive: ownershipType: Exclusive, owners: [single] - Single FIp owner

Owner Types ¶

Snat: SNAT rule using the EIP for outbound traffic
ServiceLb: LoadBalancer service using the EIP
FIp: Floating IP using the EIP exclusively

Ownership Transitions ¶

Released → Shared (first owner added)
Shared → Released (last owner removed)
Released → Exclusive (FIp claims EIP)
Exclusive → Released (FIp releases EIP)

Organization Limits ¶

Organizations have a configurable limit on the number of ready projects they can contain (default: 3). This limit is enforced by the Project controller during reconciliation.

Configuration: Set via MasterConfig.OrganizationProjectsLimit
Enforcement: Projects exceeding the limit will not be reconciled until space becomes available
Status: Projects blocked by limits show as not ready but remain in the cluster

Enhanced Limit Enforcement (v0.1.31-dev1+) ¶

The Project controller now provides comprehensive feedback when organization limits are hit:

Detailed Logging: - Organization project status with ready/pending counts and project names - Clear error messages with organization, namespace, and limit context - Debug-level logs showing available slots and project lists

Status Conditions: Projects blocked by limits receive a LimitCheck condition:

status:
  ready: false
  conditions:
  - type: LimitCheck
    status: "False"
    reason: LimitExceeded
    message: "organization limit (3 projects) reached - ready projects: 3 (limit: 3)"
    lastTransitionTime: "2025-01-19T16:45:00Z"

Automatic Retry: - Projects are automatically requeued every 30 seconds - Reconciliation proceeds when limit space becomes available - No manual intervention required

Projects can use different external network types: - cloud: Uses ext-cloud subnet (default) - public: Uses ext-public subnet (real public IPs)

The egressNetworkType in Project spec determines which external subnet is used for the default gateway EIP.