Design a service-provider view into workload health and consumer troubleshooting for compute

[scotwells]

Summary

Datum operates compute on behalf of consumers, but today the platform team has no dedicated way to see what's actually running across the fleet, which workloads are unhealthy, or what a specific consumer is experiencing when they open a support request. Provider-side visibility is limited to ad-hoc kubectl/cluster access, which doesn't scale and isn't something we can hand to on-call or support staff.

This issue tracks designing a service-provider view into the compute service: an operator-facing capability (distinct from the consumer-facing console) that surfaces deployed workloads, flags unhealthy ones that need attention, and gives support/on-call the context needed to troubleshoot a consumer's issue without shelling into clusters.

Problem

• No fleet-wide view of what's deployed across projects/consumers from the provider side.
• No proactive signal for "these workloads are unhealthy and need attention" — issues are found reactively, usually after a consumer reports them.
• When a consumer opens a support ticket about a broken workload, there's no shared surface for support/on-call to see what the consumer sees plus the provider-side signals (placement, scheduling, quota, infra health) needed to diagnose it.

Goals (draft — to be refined during design)

• Give service-provider staff visibility into workloads deployed across the fleet, scoped appropriately (e.g. by project, location, health state).
• Surface unhealthy or degraded workloads proactively, with enough signal to triage severity and root cause.
• Support consumer troubleshooting: given a consumer's workload, show provider-side state (placement, status/conditions, events, quota, dependencies) alongside what the consumer sees, so support can diagnose without cluster access.

Non-goals (draft)

• This is not the consumer-facing dashboard/console — it's for provider-side users (support, on-call, platform ops).
• Not intended to replace existing infra observability (metrics/logs/tracing) — it should complement and link into those, not duplicate them.
• Delivery mechanism (UI surface, CLI, API, existing tooling extension, etc.) is intentionally undecided — that's a design question, not a premise of this issue.

Acceptance criteria

• Design document for the service-provider view into compute — problem statement, goals/non-goals, target users (support, on-call, platform ops), key workflows (fleet overview, unhealthy-workload triage, per-consumer troubleshooting), and the delivery mechanism best suited to those workflows. Reviewed and merged.
• Data/API requirements identified — what compute needs to expose (status, conditions, events, placement, quota) to support these workflows, and any gaps against what's currently surfaced.
• Scoping/access model defined — what a provider operator can see vs. consumer-owned data boundaries.
• Follow-up implementation issues opened once the design is settled.

References

• Platform activity/audit enhancement already scopes a service-provider support persona ("view a customer's activity to troubleshoot issues they report"). Compute's provider view should build on this rather than inventing a parallel audit surface.
• "Service-contributed UI for the cloud portal and service-provider portal": the architecture proposal for how services surface UI to provider-side users. Likely the delivery mechanism for this issue's design.
• Dynamic portal plugin system prototype (service-owned UI, no portal deploy): companion prototype for the enhancement above, shows what's technically available today.
• Telemetry system design (full enhancement suite PR) defines portable MetricDefinitions service providers can expose; likely source of the health/status signal this view would surface.