How Can We Integrate Ironic Network into Kubernetes Ecosystem

[mboukhalfa]

Manage Ironic Natively in Metal3 Kubernetes Management Cluster

Currently, Ironic network is not managed in a Kubernetes-native way because of the requirements associated with each Ironic service. Additionally,the network is not a first-class concept in Kubernetes! This integration between the incompatible ironic needs and Kubernetes network model is where metal3 potentially can shine and demonstrate its strengths.
This discussion sheds light on the challenging areas and highlights the corner cases that may arise with any approach towards improving integration. It also provides an open space for brainstorming, discussing all proposed ideas, and demonstrating small PoCs.

Goals

• Address existing issues such as Issue #21 and Issue #3 with long-term roadmap in mind.
• Develop a roadmap for native Kubernetes network management for Ironic.

Overview of the Problem

Kubernetes uses a single network to connect all pods, meaning each pod is managed through one interface. However, the Metal3 network model requires two separate networks: the provisioning network (Ironic network) and the external network (Kubernetes network), also known as the baremetal network.

Due to this, Metal3 manages the provisioning network in a custom way that Kubernetes is not aware of. Currently, Metal3 needs an additional NIC on every node, connected to a bridge (named ironicendpoint). Ironic pods require host networking to manage this endpoint. Metal3 includes Keepalived in the Ironic pods to attach a VIP to this bridge on the node hosting Ironic, ensuring Ironic is always reachable by a static IP, even when moved to another node or cluster. Additionally, for PXE provisioning, Metal3 uses dnsmasq within the Ironic stack to provide DHCP services to baremetal nodes, and this service needs to be in the same baremetal nodes LAN, which is another requirement for Ironic pods to access host networking.

/help needed: I would like to draw the network topology of Metal3 with a focus on the Ironic network, showing the changes and protocols that need to be considered.
WIP Ironic network diagram, collaborate here: Excalidraw Link

Main Problematic Areas

1. Managing Two Separate Networks:

   • Metal3 requires managing two separate networks (provisioning and external) within Kubernetes. Multi-Network is an active research area in Kubernetes networking, mainly conducted by Kubernetes Network Plumbing Working Group.

2. Discoverability of Ironic:

   • Ensuring Ironic is discoverable regardless of its location, especially as it pivots between clusters, and potentially removing the dependency on static IP and keepalived.

3. Providing DHCP Services:

   • Providing DHCP services for a LAN from a native Kubernetes pod.

Long-term Vision

Our long-term vision is to provide a Kubernetes native solution for managing baremetal hosts via a provisioning stack that is also running on Kubernetes. By addressing our issues with a focus on this mission, we aim to create a comprehensive roadmap that integrates Metal3 components seamlessly into the Kubernetes ecosystem.

List of Proposed Ideas

• Meta CNI Solution:

  • Use a meta CNI (e.g., Multus) to manage two networks within Kubernetes, giving Ironic pods two interfaces. This was proposed when we adopted iroinc as a tool to provision images on baremtal check the design doc - use-ironic

• Custom Load Balancer:

  • Implement a custom load balancer based on httpd, inspired by the OpenShift proxy, and run Ironic as pods managed by the native Kubernetes network. This is being discussed in the Issue #21

Short-term Focus

1. PoC reach ironic from k8s LoadBalancer service:

   • A quick PoC to test reaching ironic running on k8s network from the provisioning network. Aim to demonstrate the solution, explore the limits, and identify corner cases.

2. Experiment with Meta CNI (e.g., Multus):

   • Conduct PoCs to validate the feasibility of using Multus or similar solutions to manage multiple networks.

For investigation

3. Native DHCP Service:
   • Explore methods for providing DHCP services from Kubernetes-native pods, ensuring compatibility with the existing Metal3 network model.

[tuminoid]

Hostnetworking also is a matter of improving Ironic security. It prevents using PSS as it violates all standards besides privileged, and also makes Ironic API exposed in host's network which might be accessible to actors not supposed to have access (local unprivileged users, same network nodes, ...)

[dtantsur]

makes Ironic API exposed in host's network which might be accessible to actors not supposed to have access

This is not going to change initially: Ironic API has to be accessed by unauthenticated parties (= IPA ramdisk).

Something to potentially consider as part of this conversation: in the future we may want to limit (at least optionally) access from the outside only to the ramdisk endpoints. THAT would be an improvement in security, but it's possible that not all solutions considered here will allow that.

JSON RPC will fully suffer from the problem you describe once supported.

[mboukhalfa]

I have created a small PoC as part of the Short-term focus section to demonstrate running Ironic without hostNetwork, limited to the VirtualMedia use case, using a nodePort service.

You can find the PR here: PoC Hostnetworkless Ironic with VirtualMedia Using NodePorts #1433.

I replaced keepalived with manual commands to add ironic external address to the ironicendpoint branch and remove it during pivoting. We need a better solution to ensure Ironic remains reachable even if the node where the branch is configured goes down.

Your feedback and ideas would be greatly appreciated as we continue to refine and improve this approach.

[mboukhalfa]

I wanted to update you on our progress. I've created a new PoC to demonstrate running Ironic without hostNetwork, limited to the VirtualMedia use case, using a LoadBalancer service with MetalLB.

You can find the PR here: PoC Hostnetworkless Ironic with VirtualMedia Using MetalLB LoadBalancer

[dtantsur]

There are a few things that are still unclear to me about MetalLB.

1. Can we make sure the address is only served from control plane nodes? I don't think we want the provisioning traffic to go through workers. Will https://metallb.io/configuration/_advanced_l2_configuration/#limiting-the-set-of-nodes-where-the-service-can-be-announced-from work for us?

2. Are we adding a requirement on a separate IP (has to be provided by a user?) even when provisioning network is not used?

3. How can we prevent users from explicitly requesting an IP from "our" pool? Through https://metallb.io/configuration/_advanced_ipaddresspool_configuration/#reduce-scope-of-address-allocation-to-specific-namespace-and-service I assume?

Could you add these aspects to your PoC?