Add k8s client to pull config from k8s #1131
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mvachhar
force-pushed
the
pr/mvachhar/k8s-direct-3
branch
8 times, most recently
from
6d445ff to
3615b26
Compare
Previously we were using ahash to hash packets for ecmp routing. However, ahash makes not attempt to be stable across architectures or even feature flags. To make sure all gateway instances would hash packets to the same nexthop when ECMP was in play, there was a test to ensure that ahash didn't change behavior without anyone noticing. With the subsequent commits, a dependency somewhere uses ahash with std or rng enabled, which changes ahash's behavior so that each instance of ahash starts with a random seed breaking the gurantees we want for ECMP. Because of how rust merges all used features together when building, the use of std or rng in a different crate affects the net crate as well. As a result, the ahash consistency test starts to fail. This commit replaces ahash with rapidhash which has a goal of hashing consistently across all platforms and instances of the hasher. Accordingly, the test and artifact to test for ahash consistency is removed as well. Signed-off-by: Manish Vachharajani <[email protected]>
spec.gateway.protocol_ip is a UnicastIPv4Addr/mask not a plain IPv4 addr. Fix generator and parser to reflect this. Signed-off-by: Manish Vachharajani <[email protected]>
mvachhar
force-pushed
the
pr/mvachhar/k8s-direct-3
branch
2 times, most recently
from
305718b to
41dc4d4
Compare
mvachhar
requested review from
Fredi-raspall,
Frostman,
Copilot,
daniel-noland and
qmonnet
and removed request for
a team
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds Kubernetes client functionality to enable the dataplane to fetch configuration directly from Kubernetes when GRPC command line arguments are not specified. The key changes include:
- Replacing the
ahashhashing library withrapidhashfor packet hashing operations - Adding a new K8s client module that watches
GatewayAgentCRDs for configuration changes - Modifying launch configuration to support both GRPC-based and K8s-based configuration sources
- Updating the protocol IP field in configuration converters to parse CIDR notation instead of plain IP addresses
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| net/src/packet/hash.rs | Replaces ahash with rapidhash hasher and removes ahash fingerprint tests |
| net/artifacts/ahash_fingerprint.txt | Deletes the ahash fingerprint reference file |
| net/Cargo.toml | Removes ahash dependency and adds rapidhash dependency |
| mgmt/src/processor/mod.rs | Adds k8s_client module declaration |
| mgmt/src/processor/launch.rs | Adds LaunchError enum and implements dual-mode launch logic for GRPC and K8s clients |
| mgmt/src/processor/k8s_client.rs | Implements K8s client that watches GatewayAgent CRD and applies configuration |
| mgmt/Cargo.toml | Adds k8s-intf dependency |
| k8s-intf/src/lib.rs | Exports client module and watch function |
| k8s-intf/src/client.rs | Implements GatewayAgent CRD watcher with streaming list strategy |
| k8s-intf/src/bolero/gateway.rs | Updates protocol_ip generation to include CIDR prefix |
| k8s-intf/Cargo.toml | Adds kube client dependencies with required features |
| dataplane/src/main.rs | Retrieves hostname and passes it to management initialization |
| dataplane/Cargo.toml | Adds hostname dependency |
| config/src/converters/k8s/underlay.rs | Parses protocol_ip as Ipv4Net instead of Ipv4Addr |
| config/Cargo.toml | Adds ipnet dependency |
| args/src/lib.rs | Makes GRPC configuration optional and updates help text |
| Cargo.toml | Adds hostname, rapidhash, and rustls to workspace dependencies |
Previously we assumed a grpc server would be used. This commit changes that so that a grpc server is started if a grpc related command line arguments are passed. In their absence, a k8s server is assumed. This is a reasonable default setup as we plan to deprecate the gRPC configuration. Signed-off-by: Manish Vachharajani <[email protected]>
Adds a method that can start a k8s watcher and call a user specifed callback for every object change. The watcher specifically watches for a particular gateway agent crd corresponding to the gateway name passed (which is typically the hostname of the gateway in the Hedgehog Open Network Fabric). This also explicitly lists rustls as a dependency so that the ring provider can be specified as the kube-rs libraries do not explicitly specify the provider causing a panic at runtime. Signed-off-by: Manish Vachharajani <[email protected]>
mvachhar
force-pushed
the
pr/mvachhar/k8s-direct-3
branch
from
41dc4d4 to
fabb038
Compare
Adds a k8s config processor using the k8s-intf watcher client. The watcher needs the hostname as well, so MgmtParams is augmented to accept hostname as a configuration parameter. Signed-off-by: Manish Vachharajani <[email protected]>
mvachhar
force-pushed
the
pr/mvachhar/k8s-direct-3
branch
from
fabb038 to
a218efd
Compare
daniel-noland
approved these changes
Dec 14, 2025
Collaborator
daniel-noland
left a comment
daniel-noland
left a comment
There was a problem hiding this comment.
I'm a big fan of the hash switch.
My only possible bug is the multicast thing.
Otherwise I think we should just make todo items
Member
@mvachhar For once we had an actual breaking change, and we didn't even use the Conventional Commit markup to reflect that, I'm soooooo disappointed 🙀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR does change the behavior of the default cli arguments. If no GRPC command line arguments are specified (e.g.,
--grpc-address), then instead of assuming a default GRPC address of0.0.0.0:50051, it will use the k8s client to fetch configuration directly from kubernetes.VLAB can be made to use dataplane in agentless mode as well with the
agentless: trueoption on the gateway controller configuration (set via the fabricator object). However, this PR does not change CI to do that as we still do not report dataplane status through the k8s direct interface. That will be the subject of the next PR in this series.Because VLAB does not fully work with agentless mode because of the aforementioned lack of status updates, this PR has been tested manually and CI will still use GRPC for the vlab tests.
Fixes #1111