Skip to content

Project Resources Documentation

This document provides a comprehensive overview of all resources created by the Kube-DC Project controller, their finalizers, ownership patterns, and deletion dependencies.

Resource Creation Order

When a Project is created, resources are synchronized in this order:

  1. Namespace - Project namespace ({org}-{project})
  2. VPC - Kube-OVN Virtual Private Cloud
  3. EIp (Default Gateway) - External IP for project gateway
  4. Subnet - Kube-OVN subnet for project pods
  5. NetworkAttachmentDefinition - CNI network configuration
  6. OvnSnatRule - SNAT rule for outbound traffic
  7. Secrets - SSH keypairs and authorized keys
  8. RBAC - Roles and RoleBindings
  9. VpcDns - DNS configuration for VPC

Detailed Resource Breakdown

1. Namespace

  • Resource: v1.Namespace
  • Name: {organization}-{project} (e.g., shalb-envoy)
  • Finalizer: None (managed by Kubernetes)
  • Created by: NewProjectNamespace() in internal/project/res_namespace.go
  • Dependencies: None (first resource created)

2. VPC (Virtual Private Cloud)

  • Resource: kubeovn.io/v1.Vpc
  • Name: {organization}-{project} (e.g., shalb-envoy)
  • Finalizer: kubeovn.io/kube-ovn-controller
  • Created by: NewProjectVpc() in internal/project/res_vpc.go
  • Dependencies: Namespace must exist
  • Configuration:
  • Static routes to external subnets
  • Extra external subnets based on egressNetworkType

3. EIp (External IP - Default Gateway)

  • Resource: kube-dc.com/v1.EIp
  • Name: default-gw
  • Namespace: {organization}-{project}
  • Finalizer: eip.kube-dc.com/finalizer
  • Created by: NewProjectEip() in internal/project/res_eip_default.go
  • Dependencies: Namespace must exist
  • Ownership States:
  • Released: No active owners (initial state)
  • Shared: Has SNAT rule and/or LoadBalancer services as owners
  • Exclusive: Used by FIp resources

4. OvnEip (Underlying OVN External IP)

  • Resource: kubeovn.io/v1.OvnEip
  • Name: {organization}-{project}-{external-subnet} (e.g., shalb-envoy-ext-public)
  • Finalizer: kubeovn.io/kube-ovn-controller
  • Created by: EIp controller via NewOvEipRes() in internal/eip/ovn_eip_res.go
  • Dependencies: EIp must exist, external subnet must be available
  • Labels:
  • network.kube-dc.com/eip: {namespace}.{eip-name}
  • Annotations:
  • kube-dc.com/ovn-eip-created-by-eip: {namespace}.{eip-name}

5. Subnet

  • Resource: kubeovn.io/v1.Subnet
  • Name: {organization}-{project}-default (e.g., shalb-envoy-default)
  • Finalizer: kubeovn.io/kube-ovn-controller
  • Created by: NewProjectSubnet() in internal/project/res_subnet.go
  • Dependencies: VPC must exist
  • Configuration:
  • CIDR block from project spec
  • Associated with project VPC
  • Gateway IP (first IP in CIDR)

6. NetworkAttachmentDefinition

  • Resource: k8s.cni.cncf.io/v1.NetworkAttachmentDefinition
  • Name: default
  • Namespace: {organization}-{project}
  • Finalizer: project.kube-dc.com/finalizer
  • Created by: NewProjectNad() in internal/project/res_nad.go
  • Dependencies: Subnet must exist
  • Configuration: Kube-OVN CNI configuration pointing to project subnet

7. OvnSnatRule

  • Resource: kubeovn.io/v1.OvnSnatRule
  • Name: {organization}-{project} (e.g., shalb-envoy)
  • Finalizer: kubeovn.io/kube-ovn-controller
  • Created by: NewProjectSnat() in internal/project/res_snat.go
  • Dependencies: OvnEip must exist and have IP assigned
  • Configuration:
  • Links project subnet to external IP
  • Enables outbound internet access for pods

8. Secrets

SSH Key Pair Secret

  • Resource: v1.Secret
  • Name: ssh-keypair-default
  • Namespace: {organization}-{project}
  • Finalizer: project.kube-dc.com/finalizer
  • Created by: NewProjectKeyPairSeret() in internal/project/res_secret.go
  • Content: Generated SSH public/private key pair

Authorized Keys Secret

  • Resource: v1.Secret
  • Name: authorized-keys-default
  • Namespace: {organization}-{project}
  • Finalizer: project.kube-dc.com/finalizer
  • Created by: NewProjectAuthKeySecret() in internal/project/res_secret.go
  • Content: SSH public keys for VM access

9. RBAC Resources

Role

  • Resource: rbac.authorization.k8s.io/v1.Role
  • Name: admin
  • Namespace: {organization}-{project}
  • Finalizer: project.kube-dc.com/finalizer
  • Created by: NewProjectRole() in internal/project/res_role.go
  • Permissions: Full access to project resources (pods, services, VMs, etc.)

RoleBinding

  • Resource: rbac.authorization.k8s.io/v1.RoleBinding
  • Name: org-admin
  • Namespace: {organization}-{project}
  • Finalizer: project.kube-dc.com/finalizer
  • Created by: NewProjectRoleBinding() in internal/project/res_role_binding.go
  • Subject: {organization}:org-admin group

10. VpcDns Service

  • Resource: v1.Service
  • Name: slr-vpc-dns-{organization}-{project}
  • Namespace: kube-system
  • Finalizer: None (managed by service controller)
  • Created by: NewProjectVpcDns() in internal/project/res_vpc_dns.go
  • Purpose: DNS resolution for VPC

Finalizers Summary

Resource Type Finalizer Controller
Project project.kube-dc.com/finalizer kube-dc-manager
EIp eip.kube-dc.com/finalizer kube-dc-manager
OvnEip kubeovn.io/kube-ovn-controller kube-ovn-controller
Vpc kubeovn.io/kube-ovn-controller kube-ovn-controller
Subnet kubeovn.io/kube-ovn-controller kube-ovn-controller
OvnSnatRule kubeovn.io/kube-ovn-controller kube-ovn-controller
NetworkAttachmentDefinition project.kube-dc.com/finalizer kube-dc-manager
Secrets project.kube-dc.com/finalizer kube-dc-manager
Role project.kube-dc.com/finalizer kube-dc-manager
RoleBinding project.kube-dc.com/finalizer kube-dc-manager

Deletion Order and Dependencies

When a Project is deleted, resources must be removed in reverse dependency order:

Phase 1: Application Resources

  1. Pods, Services, VMs - User workloads (deleted by users/operators)
  2. FIp resources - Floating IPs (if any exist)

Phase 2: SNAT and Networking

  1. OvnSnatRule - Must be deleted before OvnEip
  2. OvnEip - Must be deleted before EIp and Subnet

Phase 3: Project Infrastructure

  1. EIp - External IP resource
  2. NetworkAttachmentDefinition - CNI configuration
  3. Secrets - SSH keys and authorized keys
  4. RBAC - Roles and RoleBindings
  5. Subnet - Must be deleted before VPC
  6. VPC - Virtual Private Cloud
  7. VpcDns - DNS service
  8. Namespace - Project namespace (last)

Common Deletion Issues

Stuck Finalizers

  • OvnEip: May get stuck if SNAT rule deletion fails
  • Subnet: May get stuck if pods are still running
  • EIp: May get stuck if project is deleted before EIp controller processes it

Manual Cleanup Commands

# Remove stuck finalizers (use with caution)
kubectl patch ovn-snat-rule {name} -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl patch ovn-eip {name} -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl patch subnet {name} -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl patch eip {name} -n {namespace} -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl patch project {name} -n {org-namespace} -p '{"metadata":{"finalizers":null}}' --type=merge

EIp Ownership Patterns

Ownership States

  • Released: ownershipType: Released, owners: [] - No active users
  • Shared: ownershipType: Shared, owners: [...] - Multiple users (SNAT + Services)
  • Exclusive: ownershipType: Exclusive, owners: [single] - Single FIp owner

Owner Types

  • Snat: SNAT rule using the EIP for outbound traffic
  • ServiceLb: LoadBalancer service using the EIP
  • FIp: Floating IP using the EIP exclusively

Ownership Transitions

Released → Shared (first owner added)
Shared → Released (last owner removed)
Released → Exclusive (FIp claims EIP)
Exclusive → Released (FIp releases EIP)

Organization Limits

Organizations have a configurable limit on the number of ready projects they can contain (default: 3). This limit is enforced by the Project controller during reconciliation.

  • Configuration: Set via MasterConfig.OrganizationProjectsLimit
  • Enforcement: Projects exceeding the limit will not be reconciled until space becomes available
  • Status: Projects blocked by limits show as not ready but remain in the cluster

Enhanced Limit Enforcement (v0.1.31-dev1+)

The Project controller now provides comprehensive feedback when organization limits are hit:

Detailed Logging: - Organization project status with ready/pending counts and project names - Clear error messages with organization, namespace, and limit context - Debug-level logs showing available slots and project lists

Status Conditions: Projects blocked by limits receive a LimitCheck condition: 

status:
  ready: false
  conditions:
  - type: LimitCheck
    status: "False"
    reason: LimitExceeded
    message: "organization limit (3 projects) reached - ready projects: 3 (limit: 3)"
    lastTransitionTime: "2025-01-19T16:45:00Z"

Automatic Retry: - Projects are automatically requeued every 30 seconds - Reconciliation proceeds when limit space becomes available - No manual intervention required

Projects can use different external network types: - cloud: Uses ext-cloud subnet (default) - public: Uses ext-public subnet (real public IPs)

The egressNetworkType in Project spec determines which external subnet is used for the default gateway EIP.