[WIP] Requirements Definition for ControlPlane Component

[Copilot]

Thanks for assigning this issue to me. I'm starting to work on it and will keep this PR's description up to date as I form a plan and make progress.

Original issue description:

Overview

The ControlPlane serves as the central orchestrator for migration jobs, managing the lifecycle from job creation through execution tracking and completion. It ensures efficient resource utilisation, supports interruption handling, and adheres to data sovereignty and tiered service requirements.

Functional Requirements

Job Management

• Job Creation & Validation:

  • Validate incoming job configurations for correctness and completeness.
  • Reject invalid jobs with clear error reporting.

• Queue Management:

  • Support tiered queues (Standard, Premium, Platinum).
  • Prioritise job execution based on tier.

• Job Assignment:

  • Dynamically assign jobs to agents based on availability, capability, and regional constraints.

• Job Tracking:

  • Maintain real-time state and progress indicators for jobs.
  • Allow querying of current job status via REST API.

Interruption Handling

• Preemption Support:

  • Gracefully interrupt lower-priority jobs to accommodate higher-tiered jobs.
  • Ensure interrupted jobs are safely paused, with state persisted accurately.

• Resume Capability:

  • Provide clear mechanisms to resume paused or interrupted jobs, ensuring minimal loss of progress.

State Management

• Persistent Storage:

  • Reliably store job state, including progress counters, checkpoints, reflected work item ID mappings, and git commit checkpoints.
  • Ensure data integrity and durability during agent or infrastructure disruptions.

Regional Sovereignty

• Regional Deployment:

  • Support deployment and operation across multiple Azure regions.
  • Ensure compliance with regional data sovereignty requirements.

Non-Functional Requirements

Scalability

• Horizontal Scalability:

  • Scale dynamically to accommodate increasing workload.
  • Use autoscaling mechanisms provided by Azure Container Apps.

• Dedicated Resources:

  • Provide dedicated agent pools for Platinum tier customers.

Reliability

• Fault Tolerance:

  • Automatically recover from agent or network failures.
  • Minimise downtime and maintain data consistency.

• Observability:

  • Integrate with OpenTelemetry for monitoring and logging.
  • Provide comprehensive logging and metrics for diagnostics and performance monitoring.

Security

• Access Control:

  • Enforce strict access control mechanisms for APIs.
  • Implement secure authentication and authorisation protocols.

• Data Security:

  • Ensure data in transit and at rest is encrypted.
  • Comply with applicable data privacy regulations.

Performance

• Latency:

  • Maintain low-latency responses for job assignments and status queries.

• Throughput:

  • Efficiently handle multiple concurrent job operations without significant performance degradation.

Interface Requirements

• REST API:

  • Provide a RESTful API to interact with the ControlPlane for all job operations, including creation, querying, and managing job lifecycle.

Hosting Requirements

• Deployable as an ASP.NET Core Web API.
• Support Azure App Service, Azure Container Apps, and self-hosted deployment models.

Compliance Requirements

• Compliant with GDPR, CCPA, and other relevant data protection standards.

Documentation & Support

• Comprehensive API and user documentation provided.
• Clearly documented error handling, logging, and troubleshooting procedures.

Fixes #3.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[CLAassistant]

All committers have signed the CLA.

[MrHinsh]

@copilot Ive resoved the issue with the build

[MrHinsh]

Can you merge main into your branch?

[MrHinsh]

SHould now work