nix build system part 1

[daniel-noland]

This is the first part of our shiny new build system.

In the name of making things easy to review I am cutting this PR into small chapters.

I added detailed notes on every commit and many comments in the name of keeping my design decisions documented and clear.

(nix code has the habit of becoming an opaque magic soup, and I really want to avoid that here)

Quick check list

Right out of the gate you need to

1. checkout this PR,
2. on a linux machine.

git checkout pr/daniel-noland/dpdk-sys-take-2

Remove fake-nix if you have installed it

If you have fake-nix setup you need to remove that

if [ -L /nix ]; then
  rm /nix && >&2 echo "removed fake-nix symlink, remember to restore it after the test :)"
elif [ -d /nix ]; then
  >&2 echo "/nix is already a real directory"
  if ! which nix; then
    >&2 echo "nix is a real directory, but we can't find nix on the path.  Talk to me?"
  fi
elif  [ -a /nix ]; then
  >&2 echo "/nix exists but is not a directory or a symlink.  What are you up to?"
else;
  >&2 echo "/nix doesn't exist, you should be good to install nix now"
fi

Install nix (if you don't already have it)

Caution

Follow this next step if and only if you don't already have nix installed

If you don't have nix installed, go ahead and install single user nix now.
Formal instructions are available here but the tl;dr is

sh <(curl --proto '=https' --tlsv1.2 -L https://nixos.org/nix/install) --no-daemon

This command should also print a note about a file you need to source.
Do that or the next steps won't work.

Debug build

This is the table-stakes test: build the debug version of dpdk.

time nix build -f default.nix pkgs.debug.dpdk.static --out-link debug.dpdk --extra-experimental-features nix-command

If that worked, you should have a symlink file named debug.dpdk-static.
If you run this

ls ./debug.dpdk-static/lib

You should see the dpdk archive files.

Release build

This is the next table-stakes test: build the release version of dpdk.

time nix build -f default.nix pkgs.release.dpdk.static --out-link release.dpdk --extra-experimental-features nix-command

Smoke test build (musl)

This is one step removed from table stakes: build against musl64 so we can confirm our sysroot works.

These two commands are expected to work on basically any machine with nix.

time nix build -f default.nix pkgs.debug.pkgsCross.musl64.dpdk.static --out-link debug.musl64.dpdk --extra-experimental-features nix-command

time nix build -f default.nix pkgs.release.pkgsCross.musl64.dpdk.static --out-link release.musl64.dpdk --extra-experimental-features nix-command

Smoke test build (aarch64 debug / release)

This is all expected to work.

This only matters (at this time) in that it provides an excelent smoke test for the next PR (which builds a full sysroot).

time nix build -f default.nix pkgs.debug.pkgsCross.aarch64-multiplatform.dpdk.static --out-link debug.aarch64.dpdk --extra-experimental-features nix-command
time nix build -f default.nix pkgs.debug.pkgsCross.aarch64-multiplatform-musl64.dpdk.static --out-link debug.aarch64-musl.dpdk --extra-experimental-features nix-command
time nix build -f default.nix pkgs.release.pkgsCross.aarch64-multiplatform.dpdk.static --out-link release.aarch64.dpdk --extra-experimental-features nix-command
time nix build -f default.nix pkgs.release.pkgsCross.aarch64-multiplatform-musl64.dpdk.static --out-link release.aarch64-musl.dpdk --extra-experimental-features nix-command

[Copilot]

Pull request overview

This PR introduces the foundation of a Nix-based build system for building DPDK and its dependencies with custom compilation profiles. The system uses npins for dependency pinning and provides two build profiles (debug and release) with carefully configured compiler and linker flags for performance optimization and security hardening.

Key changes:

• Introduces compilation profiles (debug/release) with LTO, security hardening, and performance optimization flags
• Adds npins-based dependency management for DPDK, rdma-core, and nixpkgs
• Creates Nix overlays for building dataplane dependencies (DPDK, rdma-core, libnl, numactl, libbsd, libmd) with custom stdenv and static linking

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
shell.nix	Adds npins to the development environment
profiles.nix	Defines compilation flag profiles for debug and release builds with LTO and security features
npins/sources.json	Pins specific versions of DPDK (25.11), rdma-core, and nixpkgs-unstable
npins/default.nix	Generated npins helper code for loading pinned dependencies
nix/pkgs/dpdk/default.nix	DPDK package definition with extensive library/driver customization and meson configuration
nix/overlays/default.nix	Entry point for overlay definitions
nix/overlays/dataplane.nix	Main overlay that applies custom stdenv and builds dataplane dependencies with static linking
default.nix	Top-level entry point that combines profiles and overlays to create debug and release package sets

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

[Copilot]

The disabledDrivers list contains redundant entries. Line 109 specifies the wildcard pattern "compress/*" which already covers the specific drivers listed on lines 110-111 ("compress/mlx5" and "compress/zlib").

Similarly, line 112 specifies "crypto/*" which already covers all the specific crypto drivers listed on lines 113-125.

And line 208 specifies "raw/*" which covers lines 209-211, and line 212 specifies "regex/*" which covers line 213, and line 214 specifies "vdpa/*" which covers line 215.

These specific entries after wildcards are unnecessary and should be removed for clarity and maintainability, unless there's a specific reason to list them explicitly (which should be documented if so).

Suggested change

	"raw/ioat"
	"raw/ntb"
	"raw/skeleton"
	"regex/*"
	"regex/mlx5"
	"vdpa/*"
	"vdpa/ifc"
	"regex/*"
	"vdpa/*"

[daniel-noland]

will fix

[Copilot]

There is a conflict between the disabled and enabled drivers lists. Line 173 disables "net/ixgbe" in the disabledDrivers list, but line 229 enables "net/intel/ixgbe" in the enabledDrivers list.

If these refer to different drivers (one being net/ixgbe and the other net/intel/ixgbe), this is fine. However, if they refer to the same driver, this creates a conflict that should be resolved by removing one of the entries. Please verify which path is correct for the ixgbe driver in DPDK 25.11.

[daniel-noland]

will fix

[mvachhar]

I would love someone that knows nix better than I do to approve. I just have the comment about ethtool and iproute2.

Also, I presume most of the work this does gets cached so we don't have to repeated build?

Finally, it might be good to update the README.md with new build instructions when the time comes. I appears that this set of changes doesn't have any effect on the current build so I am not asking for it here. If I've misunderstood, then we need an updated README.md

[mvachhar]

I would ship these tools, the odds of needing them to debug something once dpdk rolls out is high.

[daniel-noland]

We can and will ship them. That said, I would attach them to the FRR container (which runs in the same netns)

[daniel-noland]

At the very minimum, they don't need to be in the build toolchain of DPDK

[mvachhar]

Have we submitted this patch upstream?

[daniel-noland]

No, nor should we.

The problem isn't in the rdma-core source code. It is just in the way they use nm to find symbols in .o files. When you enable lto you don't put object code in the .o file (usually), you just put in the LLVM IR. That isn't compatible with nm and so the build fails to generate .so files (because it can't find symbols) and fails.

The patch is just a hack to tell it to ignore this and move on with its life. That is harmless for us because we never use any of those .so files anyway. They go directly to the trash regardless of how they are produced. Symbol versioning isn't a thing for .a files, so this is just us side stepping the whole problem

[daniel-noland]

does't -> doesn't

[daniel-noland]

(gpt:120b ai review)

Review of PR #1142 – "nix build system part 1"

Overall impression

• The PR introduces a new Nix‑based build system for the dataplane, adding a fairly extensive overlay that customises the toolchain and dependencies for DPDK and related libraries.
• The approach of splitting the build into a debug and release overlay (with separate profiles.nix for compiler flags) is clean and follows the Hedgehog dataplane development guide.
• The use of npins for pinning external sources (dpdk, rdma‑core, nixpkgs) is appropriate and makes the build reproducible.

---

Specific observations

1. default.nix

• The top‑level default.nix creates two pkgs instances (debug and release). The final attribute set only returns { sources, pkgs }. Consider exposing the two pkgs directly (e.g. { debugPkgs, releasePkgs, ... }) to make downstream Nix expressions clearer.
• The final empty attribute set {} before the output set is unnecessary; you can remove the stray } after the in block.

2. Overlays (nix/overlays/*.nix)

• Nulling large packages (ethtool, iproute2, docs, etc.) is a pragmatic choice, but the rationale is documented only in comments. It would be helpful to add a short top‑level comment explaining why these are excluded from the dataplane image (e.g., size, unused, cross‑compilation concerns).
• The helpers.addToEnv function uses string concatenation to merge environment variables. This works but can lead to trailing spaces. Consider trimming or using builtins.concatStringsSep " " for clarity.
• The dataplane-dep helper overrides stdenv with the custom LLVM‑based environment. Ensure that downstream packages that rely on the default stdenv (e.g., glibc‑based tools) are not inadvertently affected.

3. Package customisations (libmd, libbsd, numactl, rdma‑core)

• All overridden packages add a static output and enable shared libraries (--enable-shared). This is required for DPDK but makes the final image larger. If static linking of the final binaries is the goal, you could consider stripping the shared output after the build to keep the image lean.
• The postInstall scripts copy static archives to $static/lib. Make sure that any downstream consumers that expect libraries in $out/lib also look in $static/lib (or add a dev output that points to the static archive).
• In rdma-core, the postInstall step moves *.a files to $static/lib/. However, the static output is not added to the outputs list. It is included, but you might want to verify that the resulting derivation contains the static output as intended.

4. DPDK package (nix/pkgs/dpdk/default.nix)

• The mesonFlags list is extensive and disables many drivers/libs. This is fine, but keep the list under version control as the upstream DPDK may add new drivers that silently get enabled. Periodically run meson configure on a clean source tree to verify the flag list is still accurate.
• The -Dmax_numa_nodes=8 flag is hard‑coded. If the target platform has fewer NUMA nodes, the build may emit warnings or errors. Consider exposing this as a parameter or defaulting to auto.
• The postInstall step removes documentation and moves static archives. Ensure that the share output is correctly populated; the share variable is not defined in the derivation (should be $out/share). The current script creates $share which resolves to an empty string, effectively placing the docs under $out/share. Adjust to share=$out/share if the intention is to keep a separate output.

5. profiles.nix

• The secure profile comments out several sanitizer flags. If you plan to enable them later, consider adding a sanitizers attribute set that can be merged into the profile on demand, rather than commenting/uncommenting.
• The debug profile disables FORTIFY_SOURCE. This is reasonable for debugging, but be aware that any release binaries built with the release profile will still benefit from the default FORTIFY_SOURCE=2 provided by the compiler unless explicitly overridden.

6. npins integration

• The npins/default.nix and sources.json are correctly generated. The mayOverride function respects the NPINS_OVERRIDE_<name> environment variable, which is useful for local development.
• The npins overlay currently does not expose the version attribute. If downstream Nix code ever needs to branch on the pins version, consider exposing it (e.g., inherit version;).

---

Recommendations / Action items

1. Expose both debug and release pkgs in the top‑level output for easier consumption.
2. Add a top‑level comment in nix/overlays/dataplane.nix summarising why certain packages are nulled.
3. Validate the static output for rdma-core and other overridden packages – ensure they appear in the derivation’s outputs list.
4. Fix the share handling in the DPDK postInstall script (use $out/share or define a proper share variable).
5. Consider making max_numa_nodes configurable via a parameter or default to auto.
6. Add a small test (e.g., a Nix expression that builds a trivial binary using the debug and release pkgs) to the CI pipeline to catch regressions early.
7. Document the profiles (debug, release) in README.md or the existing development guide, so contributors know which flags are applied.

---

Minor style notes

• Use a consistent indentation of two spaces (already present) but avoid trailing whitespace on lines.
• In npins/default.nix, the comment block uses /* … */. For consistency with the rest of the repo, you might switch to #‑style comments.
• The long comment blocks in dataplane.nix are helpful; consider wrapping them at 80‑column width for readability.

---

Conclusion
The PR lays a solid foundation for a reproducible Nix‑based build system. With the few adjustments above, the overlay will be more robust, easier to maintain, and clearer for future contributors.

Feel free to reach out if you’d like help implementing any of the suggested changes.