-
Notifications
You must be signed in to change notification settings - Fork 57
Add docs for enabling Istio native nftables feature #1122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
yxun
wants to merge
14
commits into
istio-ecosystem:main
Choose a base branch
from
yxun:nft-doc
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
5d59776
update docs: add terminology
yxun bf8c65e
Add docs for istio nftables
yxun 92b36e1
Add installation steps on OpenShift
yxun ef831d2
Merge from main
yxun e27c7e6
Merge remote-tracking branch 'eco/main' into nft-doc
yxun 329ec6e
Update Istio nftables doc
yxun bacfac1
update doc
yxun 2c68a3a
update docs
yxun 364bab8
add converted ascii istio-nftables.adoc
yxun 1057712
remove markdown doc file
yxun 1b488f7
update docs: fix typo and commit check
yxun 8a2d4a7
update docs
yxun 1340b2b
review updates
yxun f89ca91
update istio-nftables doc
yxun File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| apiVersion: sailoperator.io/v1 | ||
| kind: Istio | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| version: master | ||
| namespace: istio-system | ||
| values: | ||
| global: | ||
| nativeNftables: true | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| apiVersion: sailoperator.io/v1 | ||
| kind: IstioCNI | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| version: master | ||
| namespace: istio-cni | ||
| values: | ||
| global: | ||
| nativeNftables: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,252 @@ | ||
| // Variables embedded for GitHub compatibility | ||
| :istio_latest_version: 1.27.2 | ||
| :istio_latest_version_revision_format: 1-27-2 | ||
| :istio_latest_tag: v1.27-latest | ||
| :istio_release_name: release-1.27 | ||
| :istio_latest_minus_one_version: 1.27.2 | ||
| :istio_latest_minus_one_version_revision_format: 1-27-2 | ||
|
|
||
| link:../README.md[Return to Project Root] | ||
|
|
||
| == Table of Contents | ||
|
|
||
| * link:#istio-nftables-backend[Istio nftables backend] | ||
| ** link:#prerequisites[Prerequisites] | ||
| ** link:#installation[Installation] | ||
| *** link:#install-using-sail-operator[Install using Sail Operator] | ||
| *** link:#install-in-ambient-mode[Install in Ambient Mode] | ||
| ** link:#validation[Validation] | ||
| ** link:#Upgrade[Upgrade] | ||
| *** link:#upgrade-using-sail-operator[Upgrade using Sail Operator] | ||
| *** link:#upgrade-in-ambient-mode[Upgrade in Ambient Mode] | ||
|
|
||
| === Istio nftables backend | ||
|
|
||
| This document outlines the configuration steps for the nftables backend in Istio. As the official successor to iptables, nftables offers a | ||
| modern, high-performance alternative for transparently redirecting traffic to and from the Envoy sidecar proxy. | ||
| Many major Linux distributions are actively moving towards adopting native nftables support. | ||
|
|
||
| === Prerequisites | ||
|
|
||
| * *nftables version*: Requires `+nft+` binary version 1.0.1 or later. | ||
|
|
||
| === Installation | ||
|
|
||
| The support for native nftables in Istio sidecar mode was implemented in the upstream istio https://github.com/istio/istio/blob/master/releasenotes/notes/nftables-sidecar.yaml[release-1.27]. | ||
| It is disabled by default. To enable it, you can set a feature flag as `+values.global.nativeNftables=true+`. | ||
|
|
||
| ==== Install using Sail Operator | ||
|
|
||
| When installing `Istio` and `IstioCNI` resources with the Sail Operator, you can enable nftables by setting `spec.values.global.nativeNftables=true` in the resource. This option configures Istio to use the nftables backend for traffic redirection instead of iptables. | ||
|
|
||
| . Create the `+istio-system+` and `+istio-cni+` namespaces. | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl create namespace istio-system | ||
| kubectl create namespace istio-cni | ||
| ---- | ||
|
|
||
| [start=2] | ||
| . Create the `+IstioCNI+` resource with `+spec.values.global.nativeNftables=true+`: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| cat <<EOF | kubectl apply -f- | ||
| apiVersion: sailoperator.io/v1 | ||
| kind: IstioCNI | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| version: v1.27-latest | ||
| namespace: istio-cni | ||
| values: | ||
| global: | ||
| nativeNftables: true | ||
| EOF | ||
| ---- | ||
|
|
||
| [start=3] | ||
| . Create the `+Istio+` resource with | ||
| `+spec.values.global.nativeNftables=true+`: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| cat <<EOF | kubectl apply -f- | ||
| apiVersion: sailoperator.io/v1 | ||
| kind: Istio | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| version: v1.27-latest | ||
| namespace: istio-system | ||
| values: | ||
| global: | ||
| nativeNftables: true | ||
| EOF | ||
| ---- | ||
|
|
||
| ==== Install in Ambient Mode | ||
|
|
||
| The support for native nftables in Istio ambient mode was implemented in the upstream istio `+release-1.28+`. | ||
| To enable native nftables in ambient mode, you can set the same feature flag with ambient profile. For example, | ||
|
|
||
| . Create the `+istio-system+`, `+istio-cni+` and `+ztunnel+` namespaces. | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl create namespace istio-system | ||
| kubectl create namespace istio-cni | ||
| kubectl create namespace ztunnel | ||
| ---- | ||
|
|
||
| [start=2] | ||
| . Create the `+IstioCNI+` resource with `+profile: ambient+` and `+spec.values.global.nativeNftables=true+`: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| cat <<EOF | kubectl apply -f- | ||
| apiVersion: sailoperator.io/v1 | ||
| kind: IstioCNI | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| version: v1.28-latest | ||
| namespace: istio-cni | ||
| profile: ambient | ||
| values: | ||
| global: | ||
| nativeNftables: true | ||
| EOF | ||
| ---- | ||
|
|
||
| [start=3] | ||
| . Create the `+Istio+` resource with `+profile: ambient+` and `+spec.values.global.nativeNftables=true+`: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| cat <<EOF | kubectl apply -f- | ||
| apiVersion: sailoperator.io/v1 | ||
| kind: Istio | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| version: v1.28-latest | ||
| namespace: istio-system | ||
| profile: ambient | ||
| values: | ||
| pilot: | ||
| trustedZtunnelNamespace: ztunnel | ||
| global: | ||
| nativeNftables: true | ||
| EOF | ||
| ---- | ||
|
|
||
| [start=4] | ||
| . Create the `+ZTunnel+` resource: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| cat <<EOF | kubectl apply -f- | ||
| apiVersion: sailoperator.io/v1alpha1 | ||
| kind: ZTunnel | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| version: v1.28-latest | ||
| namespace: ztunnel | ||
| profile: ambient | ||
| EOF | ||
| ---- | ||
|
|
||
| === Validation | ||
|
|
||
| When using the `+nftables+` backend, you can verify the traffic redirection rules using the `+nft list ruleset+` command in any pod that is part of the mesh. | ||
| You can find all rules are in the `+inet+` table. The following example installs a sample application `+curl+` in a namespace `+test-ns+` that is part of the mesh. | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl create ns test-ns | ||
| ---- | ||
|
|
||
| Enable sidecar injection for the namespace `+test-ns+` when using sidecar mode: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl label namespace test-ns istio-injection=enabled | ||
| ---- | ||
|
|
||
| As an alternative, enable ambient mode for the namespace `+test-ns+`: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl label namespace test-ns istio.io/dataplane-mode=ambient | ||
| ---- | ||
|
|
||
| Deploy a sample application: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl apply -n test-ns -f https://raw.githubusercontent.com/istio/istio/refs/heads/master/samples/curl/curl.yaml | ||
| ---- | ||
|
|
||
| Attach a debug container and you can see the nftable rules in the `+inet+` table: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl -n test-ns debug --image istio/base --profile netadmin --attach -t -i \ | ||
| "$(kubectl -n test-ns get pod -l app=curl -o jsonpath='{.items..metadata.name}')" | ||
|
|
||
| root@curl-6c88b89ddf-kbzn6:$ nft list ruleset | ||
| ---- | ||
|
|
||
| Verify the connectivity between two pods is working. For example, deploy a httpbin application using the following step: | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl apply -n test-ns -f https://raw.githubusercontent.com/istio/istio/refs/heads/master/samples/httpbin/httpbin.yaml | ||
|
|
||
| kubectl exec -n test-ns "$(kubectl get pod -l app=curl -n test-ns -o jsonpath={.items..metadata.name})" -c curl -n test-ns -- curl http://httpbin.test-ns:8000/ip -s -o /dev/null -w "%{http_code}\n" | ||
|
|
||
| 200 | ||
| ---- | ||
|
|
||
| More guidelines: | ||
| https://github.com/istio/istio/tree/master/tools/istio-nftables/pkg#debugging-guidelines[Debugging Guidelines] | ||
|
|
||
| === Upgrade | ||
|
|
||
| The migration from iptables backend to nftables backend can be done by upgrading `+Istio+` and `+IstioCNI+` resources. | ||
| Because the CNI component runs as a cluster singleton, it is recommended to operate and upgrade the CNI component separately from the Istio control plane. | ||
|
|
||
| ==== Upgrade using Sail Operator | ||
|
|
||
| To upgrade an iptable based Istio service mesh with nftables backend, use the following steps: | ||
|
|
||
| . Check existing `+Istio+` and `+IstioCNI+` resources’ state are Healthy. | ||
|
|
||
| [start=2] | ||
| . Enable nftables by setting `spec.values.global.nativeNftables=true` in the `+Istio+` and `+IstioCNI+` resources. You can find the same example manifests in the Installation section above. | ||
|
|
||
| [start=3] | ||
| . Update the data plane namespace `+test-ns+` by restarting all deployments. For example, | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl rollout restart deployment -n test-ns | ||
| ---- | ||
|
|
||
| ==== Upgrade in Ambient mode | ||
|
|
||
| . Enable nftables with `+ambient+` profile by setting `spec.values.global.nativeNftables=true` in the `+Istio+` and `+IstioCNI+` resources. You can find the same example manifests in the Installation section above. | ||
|
|
||
| [start=2] | ||
| . Update the data plane namespace `+test-ns+` by restarting all deployments so that new rules can be applied. For example, | ||
|
|
||
| [source,sh] | ||
| ---- | ||
| kubectl rollout restart deployment -n test-ns | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. hmm... we should find a way to support re-programming the new rules without having to restart the pods. Did you take a look at the code to understand what it does when config changes and the rules do not match? |
||
| ---- | ||
|
|
||
| [start=3] | ||
| . You can follow same steps in the link:#validation[Validation] section to validate traffic redirection is working. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While this looks good. In the longer run, we should use templates and update it based on the args.