Skip to content

Integrate Milo IPAM service to CNI #197

Description

@privateip

High-Level Summary

galactic-cni currently allocates pod IP addresses using its own local IPAM logic, independent of Milo, the IPAM service that already owns and tracks address space for Galactic VPC. This proposes delegating galactic-cni's IP allocation to Milo's IPAM service directly, so every pod address comes from, and is accounted against, the same authoritative pool a tenant sees for their VPC. Instead of two systems separately claiming to know what's allocated, there is one source of truth.

Motivation

A tenant's Galactic VPC CIDR is managed and reported through Milo. Pod IPs assigned by galactic-cni today are not. This produces concrete, observable problems:

  • A tenant can look at their VPC address usage in Milo and it will not reflect the pod IPs actually in use on their clusters, because galactic-cni never told Milo about them.
  • Two clusters attached to the same tenant VPC can independently allocate the same address if their local IPAM state diverges, since neither is checking against a shared authority.
  • When a pod is deleted, the address release is a local CNI-side operation. If that release doesn't propagate, Milo's view of the pool stays wrong indefinitely, and the tenant eventually runs out of usable space in a VPC that looks half-full in the address space they were sold.
  • Any tooling built against Milo's IPAM data (utilization dashboards, capacity alerts, address audits) is blind to actual pod-level consumption, because that consumption never enters Milo's ledger.

This isn't a nice-to-have integration, it's closing a correctness gap between what the CNI hands out and what the system of record thinks exists.

Goals

  • galactic-cni's IPAM plugin invocation (ADD) requests a pod address from Milo's IPAM service instead of a local backend (host-local, whereabouts, or similar).
  • Pod address releases (DEL) call Milo to release the lease synchronously as part of CNI teardown, not as a best-effort or async cleanup step.
  • CNI CHECK validates the pod's current address against Milo's recorded lease and surfaces a mismatch rather than silently trusting local state.
  • Two clusters attached to the same tenant VPC cannot allocate the same address, because both are leasing from the same Milo-managed pool.
  • Pod-level address consumption is visible in Milo's IPAM inventory, so tenant-facing utilization views reflect real allocations, not just VPC-level CIDR bookkeeping.
  • Milo IPAM API failures during ADD/DEL have a defined, bounded failure mode (fail the CNI call, don't fall back to silently allocating from an untracked local pool).

Non-Goals

  • No changes to how pod subnet routes are distributed into the SRv6 overlay (BGP/EVPN service distribution is a separate concern from address leasing).
  • No support for IPAM backends other than Milo, this isn't a pluggable-backend abstraction, it's a direct delegation.
  • No migration tooling for clusters currently running with local IPAM state; this covers new allocations going forward, not backfilling existing leases.
  • No changes to the CNI's dataplane behavior (bridge, veth, routing setup), this is scoped to the IPAM delegate only.:q

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    No fields configured for Enhancement.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions