Design Concepts

Understand the design philosophy, patterns, and trade-offs behind Conductor Framework.

Philosophy

Design Principles

  • Manifest-First Approach: Manifests are the source of truth, embedded at compile time
  • Kubernetes-Native: Uses standard Kubernetes APIs and patterns
  • Developer-Friendly: Minimal boilerplate, sensible defaults, clear APIs
  • Extensible: Custom template functions, custom templates, plugin points

Core Values

  • Simplicity: Easy to understand and use
  • Reliability: Robust error handling and event tracking
  • Flexibility: Support for various use cases and customization
  • Performance: Efficient resource usage and fast operations

Trade-offs Made

  • Manual Reconciliation: No automatic periodic reconciliation (explicit control via API)
  • Embedded Manifests: Requires rebuild for manifest changes (but ensures version control and single binary)
  • No Built-in Auth: API security left to users (flexibility vs. convenience)

Manifest-First Approach

Why Manifests Are Embedded

Manifests are embedded at compile time using Go's embed directive. This design choice provides several benefits:

Benefits

  • Version Control Integration: Manifests are versioned with code
  • Single Binary Deployment: No external file dependencies
  • No External File Dependencies: Everything needed is in the binary
  • Template Flexibility: Templates are processed at runtime with parameters
  • Compile-Time Validation: Invalid manifests are caught at build time

Trade-offs

  • Rebuild required for manifest changes
  • Larger binary size (but typically minimal impact)
  • No runtime manifest updates (but API overrides are supported)

Template System Design

Go Template Engine Choice

Uses Go's standard library text/template package for several reasons:

  • Built into Go standard library (no external dependencies)
  • Familiar syntax for Go developers
  • Secure by default (no arbitrary code execution)
  • Extensible with custom functions

Sprig Function Integration

Integrates Sprig template functions (same library used by Helm) for:

  • String manipulation (upper, lower, trim, replace, etc.)
  • Math operations (add, sub, mul, div, max, min)
  • List and dictionary operations
  • Encoding/decoding (base64, etc.)
  • 60+ functions total

Security Considerations

  • Excluded Functions: env and expandenv are excluded to prevent environment variable access
  • No Code Execution: Templates cannot execute arbitrary code
  • Parameter Injection Safety: Parameters are injected safely into template context

Custom Function Support

Users can provide custom template functions via Config.TemplateFuncs, allowing:

  • Domain-specific helper functions
  • Override of Sprig functions if needed
  • Integration with external systems

Parameter Injection Model

Parameters flow from CRD spec → template context:

  1. CRD spec is retrieved (or defaults used)
  2. Global and service-specific parameters are merged
  3. Template context is created with full spec
  4. Templates access parameters via .Spec.Global and .Spec.Services

Reconciliation Model

Manual Reconciliation

Reconciliation is triggered manually via API endpoints rather than automatically. This provides:

  • Explicit control over when reconciliation happens
  • Predictable behavior
  • Integration with external systems (CI/CD, schedulers)
  • Reduced resource usage (no background polling)

Event-Driven Architecture

All reconciliation operations generate events:

  • Success events for successful operations
  • Error events for failures
  • Info events for status updates
  • Warning events for non-critical issues

Idempotency Guarantees

All operations are idempotent:

  • Kubernetes server-side apply ensures idempotency
  • Safe to retry operations
  • No side effects from repeated calls

Resource Ownership Model

Framework tracks managed resources:

  • Maintains a set of managed resource keys
  • Orphaned resources (not in current manifests) are cleaned up
  • Prevents resource leaks

API Design

RESTful Principles

  • Standard HTTP methods (GET, POST, PUT, DELETE)
  • Resource-based URLs
  • JSON request/response bodies
  • Standard HTTP status codes

Resource Naming

Resources are identified by keys in format: namespace/Kind/name

Example: default/Deployment/redis

Error Response Format

Structured JSON error responses:

{
  "error": "error_code",
  "message": "Human-readable message",
  "details": {
    "key": "value"
  }
}

Versioning Strategy

Currently no API versioning. Future consideration for:

  • URL-based versioning (/api/v1/...)
  • Header-based versioning
  • Backward compatibility guarantees

Endpoint Organization

  • /api/* for REST API endpoints
  • /* for Web UI routes
  • /healthz and /readyz for health checks

Storage Design

BadgerDB Choice Rationale

BadgerDB was chosen as the storage backend because:

  • Embedded Database: No external database required
  • High Performance: Fast key-value operations
  • Go-Native: Written in Go, no CGO dependencies
  • No External Dependencies: Single binary deployment
  • ACID Transactions: Reliable data persistence

Data Model

  • Events: Time-series data with timestamps
  • Manifest Overrides: Key-value storage for overrides

Indexing Strategy

Events are indexed by:

  • Resource key
  • Event type
  • Timestamp (for time-range queries)

Retention Policies

Configurable event retention:

  • Default: 7 days
  • Configurable via LogRetentionDays
  • Scheduled cleanup via LogCleanupInterval

CRD Integration

Why CRDs for Parameters

DeploymentParameters CRD provides:

  • Kubernetes-Native: Uses standard Kubernetes APIs
  • Declarative Configuration: GitOps-friendly
  • Version Control Friendly: YAML files in version control
  • Operator Pattern Alignment: Follows Kubernetes operator patterns

Dynamic Client Usage

Uses Kubernetes dynamic client for:

  • No need for generated clients
  • Flexible schema handling
  • Runtime type discovery

Fallback Mechanisms

When Kubernetes is unavailable:

  • Falls back to default parameters
  • Framework continues to function
  • Manifests are rendered with defaults

Web UI Design

Embedded UI Approach

Web UI is embedded as HTML templates in Go:

  • No separate frontend build process
  • Single binary includes UI
  • Server-side rendering

Template Customization

Users can provide custom templates via CustomTemplateFS:

  • Override default templates
  • Customize UI appearance
  • Add custom pages

Real-Time Updates

UI polls API endpoints for updates:

  • JavaScript polling for status
  • Event log refresh
  • Health status updates

Extensibility

Custom Template Functions

Add custom functions via Config.TemplateFuncs:

cfg.TemplateFuncs = template.FuncMap{
    "myCustomFunc": func(s string) string {
        return "custom-" + s
    },
}

Custom Templates

Provide custom HTML templates via Config.CustomTemplateFS

Integration Points

  • API endpoints for external integration
  • Event hooks (future consideration)
  • Middleware (future consideration)

Security Model

Template Security

  • No environment variable access (env/expandenv excluded)
  • No arbitrary code execution
  • Parameter injection is safe

API Security

Currently no built-in authentication. Production recommendations:

  • Add authentication middleware (JWT, API keys)
  • Implement rate limiting
  • Restrict CORS to specific origins
  • Use TLS/HTTPS

Kubernetes RBAC Integration

Framework uses kubeconfig for authentication:

  • Respects Kubernetes RBAC
  • Uses service account permissions when running in cluster
  • Follows cluster security policies

Secret Management

Secrets are managed via Kubernetes secrets, not by the framework:

  • Framework doesn't store secrets
  • Uses Kubernetes native secret management
  • Follows Kubernetes security best practices

Scalability Considerations

Horizontal Scaling

Framework design supports horizontal scaling:

  • Stateless design (except BadgerDB storage)
  • Shared storage for multiple instances
  • No coordination required between instances

Resource Limits

  • Max 10 concurrent reconciliations
  • Configurable via code (future: configuration)
  • Prevents resource exhaustion

Performance Bottlenecks

  • Kubernetes API rate limits
  • BadgerDB write performance
  • Template rendering for large manifests

Optimization Opportunities

  • Batch Kubernetes API calls
  • Cache rendered templates
  • Optimize BadgerDB usage
  • Parallel manifest processing

Future Design Directions

Planned Improvements

  • API versioning support
  • Built-in authentication options
  • Webhook support for events
  • Metrics and observability

Known Limitations

  • Manual reconciliation only
  • No built-in authentication
  • Single BadgerDB instance (no clustering)

Extension Points

  • Custom template functions
  • Custom templates
  • API endpoints
  • Event hooks (future)

Next Steps