diff --git a/.agents/docs/2026-09-07-heterogeneous-cross-platform-ecosystem.md b/.agents/docs/2026-09-07-heterogeneous-cross-platform-ecosystem.md new file mode 100644 index 00000000..2edd06a3 --- /dev/null +++ b/.agents/docs/2026-09-07-heterogeneous-cross-platform-ecosystem.md @@ -0,0 +1,383 @@ +# 异构计算与图形的跨平台生态:完整矩阵与补齐方案 + +> 状态:设计,未实现。每一条「现状」都标注了它是**读**出来的还是**跑**出来的。 +> 伴随文档:`docs/20-heterogeneous-builds.md`(规范)、 +> `2026-09-07-package-identity-and-doc-alignment.md`(规则自带环境)。 + +## 0. 这份方案要解决的一句话 + +**引擎与规则层已经与平台无关,而生态只在 Linux 上是完整的。** 这不是一个工程问题: +四个设备编译器里有三个在 Windows 上、两个在 macOS 上**已经有上游产物**,缺的是把它们 +发出来。 + +## 1. 现状:分层看,缝在哪一层 + +### 1.1 引擎与规则层:与平台无关(**读** + **跑**) + +`accel` 轴、`cfg(accelerator = ...)`、受约束 glob、产物的 accel 标签、规则包的 +`feature-xlings` 声明 —— 没有一处按宿主分支。e2e 628(工具版本冲突的拒绝)不设能力门, +三平台都跑。引擎里零个厂商名,由 `test_core_vendor_probes.cpp` 按文件数做分母守着。 + +**这一层不需要补。** + +### 1.2 载荷层:linux-only(**跑** index 查询) + +`glslang` `shaderc` `cuda-nvcc` `cuda-cudart` `libcurand` `cuda-cccl` +`hip-nvidia` `dpcpp` `mesa-lavapipe` `cann-toolkit` —— **全部只有 `linux` 块**。 + +### 1.3 库层:三平台齐全(**跑**) + +`compat.vulkan` `glfw` `sdl2` `imgui` `opengl` `vulkan-memory-allocator` +`spirv-reflect` —— 都有 `linux macosx windows`。 + +**所以缝恰好在中间那一层**:一个 Windows 开发者今天能链接 Vulkan、开窗、画三角形, +只要他自己准备着色器编译器 —— 而那正是构建系统该管的那一件事。 + +### 1.4 适配器层:linux-only 是结构性的,不是缺口 + +`compat:vulkan-runtime` / `sycl-runtime` / `cuda-runtime` 修的是**mcpp 私有 loader +够不到宿主驱动**。那是 ELF/`PT_INTERP` 的问题;macOS 走 dyld、Windows 走 PE loader, +两边都没有这一层。**它们不该被"补到三平台",它们在那两个平台上不存在。** + +驱动本身在三个平台上是同一条规则:**驱动是宿主能力,设备是载荷**。 +`compat.vulkan` 已经把它建模成 `vulkan.icd.driver` 能力(macOS 上是 MoltenVK, +Windows 上是任何 GPU 驱动装的 `vulkan-1.dll`)。 + +## 2. 关键发现:多数空格是发布工作(**跑**出来的) + +逐个核过上游产物的可得性: + +| 后端 / 工具 | linux-x86_64 | linux-aarch64 | windows-x86_64 | macos-arm64 | +|---|---|---|---|---| +| CUDA(nvcc, cudart, curand, cccl) | 已发布 | **redist 有**(重打包) | **redist 有**(重打包) | **不可能** | +| SYCL(`dpcpp`) | 已发布 | 未发布 | **`sycl_windows.tar.gz`**(重打包) | 上游不发 | +| SPIR-V(`shaderc`/glslc) | 已发布 | 上游有 | **Google 产物桶有**(重打包) | **Google 产物桶有**(重打包) | +| `glslang` | 已发布(本生态自建) | — | 上游无二进制 | 上游无二进制 | +| HIP(NVIDIA 平台,仅头文件) | 已发布 | 头文件与平台无关 | 头文件与平台无关 | 不适用 | +| Ascend(`cann-toolkit`) | 已发布 | `.run` 内含 aarch64 | 无 | 无 | +| 软件设备(`mesa-lavapipe`) | 已发布(自建 Mesa) | — | 需 Mesa-on-Windows 构建 | 不适用 | +| Metal | 不适用 | 不适用 | 不适用 | **仅 Xcode 内**,不可再分发 | + +核过的三条直链: + +- NVIDIA `redistrib_12.9.1.json` 里 `cuda_nvcc` / `cuda_cudart` / `libcurand` / + `cuda_cccl` 四个都列着 `linux-x86_64 linux-sbsa windows-x86_64 linux-aarch64` + —— **索引今天用的就是这棵 redist 树**,只取了其中一个平台; +- `intel/llvm` 的同一个 release 里同时有 `sycl_linux.tar.gz` 与 + **`sycl_windows.tar.gz`**; +- Google 的 shaderc 产物桶前缀:`linux/` `macos/` `windows-vs2022-amd64-release/`, + macOS 与 Linux 的 `install.tgz` 直链已核 200。 + +**结论:CUDA 上 Windows、SYCL 上 Windows、着色器编译器上 Windows 与 macOS,都是重新 +打包而不是从源码构建。** 而 macOS 的 CUDA 与 SYCL 是永久的洞(NVIDIA 与 Intel 都不发), +这是要设计**绕过**的事实,不是要补的缺口。 + +## 3. 可移植性分层:给使用者的模型 + +生态可用的前提是使用者能**预期**哪条路能走多远。三层: + +| 层 | 后端 | 平台 | 使用者写什么 | +|---|---|---|---| +| **T1 可移植** | Vulkan compute、Vulkan graphics、OpenCL | linux / macos / windows | `accel = "vulkan1.2"`,一份 manifest 三平台通 | +| **T2 双平台** | CUDA、SYCL | linux / windows | 同一份 manifest;macOS 上该后端的 `cfg` 不激活,走 CPU 接缝 | +| **T3 单平台** | Metal(macOS)、Ascend(linux)、ROCm(linux) | 各一 | 同上 | + +**跨这三层不需要新机制** —— `accel` 是集合、`cfg(accelerator = ...)` 是成员判定、 +接缝把实现藏在模块后面,这些已经有了。一个想要「到处能跑」的工程写: + +```toml +[build] +accel = "vulkan1.2" +sources = [ "src/*.cppm", "src/*.cpp", + { glob = "shaders/*.comp", accel = "vulkan1.2" } ] + +[target.'cfg(accelerator = "none")'.build] +sources = ["src/cpu/*.cpp"] +``` + +而一个想要「有 NVIDIA 就用 CUDA」的工程再加一块 `cfg(accelerator = "cuda")`,并在 +macOS 上自然退回 T1 或 CPU。 + +**T1 是这份方案要保证的那一层。** T2/T3 是能力,T1 是承诺。 + +## 4. CI 能断言到哪:软件设备决定上限 + +「生态级可用」的判据不是构建通过,是**有东西在设备上跑过**。而无头 runner 上有没有 +设备,取决于有没有**软件设备**: + +| 平台 | 软件设备 | CI 上限 | +|---|---|---| +| linux | `xim:mesa-lavapipe`(已有,CI 每次构建都跑到它并断言设备名) | **运行** | +| windows | 需要 Mesa-on-Windows 的 lavapipe,或 SwiftShader | 今天是**构建** | +| macos | MoltenVK 之上是否有可用 Metal 设备,**未实测** | 今天是**构建** | + +规则写死:**没有软件设备的平台,CI 的上限就是构建,而这一条要写进 README 而不是被 +后来的人发现**。把「构建通过」当成「能用」正是本轮反复付过代价的形状。 + +第四期(§6)专门抬这条上限,因为它是「生态级可用」与「能编译」的分界。 + +## 5. 图形管线:第一条要补的,而判据不是窗口 + +### 5.1 现状(**读** + **跑**) + +`rules-spirv` 的 `stage_of` 覆盖十四个阶段(`.comp .vert .frag .geom .tesc .tese +.mesh .task` 与五个光追阶段),产物形状对每个阶段相同:一个 shader 一个头,符号 +`__spv`。而全树 `.vert`/`.frag`/`.mesh` **零个文件** —— +**声明覆盖、执行为零**。 + +### 5.2 主示例必须是离屏渲染 + +直觉写法是 glfw + swapchain。**那样 CI 断言不了任何东西**:无头 runner 没有 surface, +而「构建通过」对图形管线几乎零信息量 —— `.frag` 编译成常量色、顶点输入接错,构建同样 +通过。 + +离屏渲染走的是**完整图形管线**(顶点输入 → 光栅化 → 片元输出 → render pass),不需要 +窗口/surface/swapchain,结果是确定像素,而且跑在**CI 里已经在工作的那个设备上**。 + +`examples/10-graphics/offscreen`:渲染顶点色三角形到 image,拷回主机内存,断言像素。 + +判据: + +| # | 断言 | 为什么不空转 | +|---|---|---| +| G1 | `triangle_vert.h` 与 `triangle_frag.h` 都存在 | 只断言其一,只处理第一个源的规则也通过 | +| G2 | 两个头的 SPIR-V magic 都是 `07230203` | 空文件也「存在」 | +| G3 | 程序打印它用的设备名,且是 lavapipe | 悄悄回落 CPU 腿会打印同样的像素 | +| G4 | **中心像素**在三顶点色的插值范围内,**四角**是清除色 | `.frag` 写成常量、顶点输入接错 → 这条红而 G1–G3 全绿 | +| G5 | `--no-accel` 走 CPU 腿并退 0 | 反向腿 | + +窗口那一层放进 `--features window`,CI 只构建不运行。**能被断言的和不能被断言的分开 +摆** —— 混在一起,CI 只能整个跳过。 + +### 5.3 已从代码读出、待证实的一处缺陷 + +着色器输出路径是 `gen / (stem + "_" + stage)`,**不含目录分量**。于是 +`shaders/ui/text.vert` 与 `shaders/world/text.vert` 落在同一个 `text_vert.h`。 +compute 示例每个工程只有一个 `.comp`,碰不到;真实图形工程按用途分目录是常态。 + +修法与引擎侧 mcpp#239/#240 的对象路径消歧同形,并且两个源映射到同一输出时必须 +**拒绝并点名两者** —— 静默覆盖是唯一不可接受的形态。一期的示例里**故意放两个同名 +不同目录的 shader,先看它坏**。 + +## 6. 补齐路线与分期 + +| 期 | 内容 | 依赖 | 它买到什么 | +|---|---|---|---| +| 一 | `examples/10-graphics/offscreen`(linux)+ G1–G5 + 同名 stem 修复 | 无 | 图形那一半第一次被执行 | +| 二 | `shaderc` 发 macOS/Windows;`rules-spirv` 按平台声明默认;图形示例三平台**构建** | 一 | **T1 的着色器编译在三平台成立** | +| 三 | CUDA 发 `windows-x86_64` 与 `linux-aarch64`;SYCL 发 `windows-x86_64` | 无(与一、二并行) | **T2 从"仅 linux"变成"linux+windows"** | +| 四 | Windows 软件 Vulkan 设备(Mesa-on-Windows 或 SwiftShader);macOS Metal 设备实测 | 二 | 把 CI 上限从构建抬到**运行** | +| 五 | ROCm / Metal 的准入判定 | 四 | 按需 | + +一期与二期顺序不可交换:**先有一个会失败的判据,再去扩平台。** 反过来做,扩平台那次 +改动没有任何东西能证明它是对的。三期与一、二期无依赖,可并行。 + +### 6.1 二期的具体形状 + +`glslang` 没有上游二进制,三平台自建它买不到任何 shaderc 买不到的东西。**因此不为 +macOS/Windows 发 glslang**,而让规则按平台选: + +```toml +[target.'cfg(accelerator = "vulkan")'.feature-xlings.rules-spirv] +"xim:shaderc" = ">=2026.3" + +[target.'cfg(all(accelerator = "vulkan", linux))'.feature-xlings.rules-spirv] +"xim:glslang" = ">=15.1.0" +``` + +规则的选择顺序(`options::compiler` → 环境变量 → glslang → shaderc → PATH)**已经** +会在没有 glslang 时退到 glslc,并自己写出声明抹平两者输出形状的差异。 +**跨平台一致性由规则的既有选择能力提供,不由载荷的完全对齐提供。** + +代价写明:Linux 与非 Linux 默认编译器不同,同一份着色器经由不同前端。这是可观测的 +(规则把用了哪个作为 fact 报出),而**判据落在产物上**(G2 的 magic、G4 的像素)在三个 +平台都要成立。 + +P4:**Linux 上的读数不变**(仍走 glslang)—— 这是「无感升级」那一条。 + +### 6.2 三期的具体形状 + +CUDA 与 SYCL 的 Windows 载荷来自索引**已经在用的那棵树**,所以是同一条流水线多跑 +几个平台: + +1. 下载 `windows-x86_64` 组件 / `sycl_windows.tar.gz`,核对能跑、版本对齐; +2. 按 xim 载荷布局重新打包,上传 `xlings-res/*`,gtc 镜像 GitCode,**两端逐字节比对**; +3. xim recipe 补 `windows` 块; +4. `rules-cuda` / `rules-sycl` 的 `feature-xlings` 声明去掉 linux 限定; +5. 夹具的 CI 从 linux 扩到 windows **构建**。 + +macOS 产物的架构要先核(本生态 macOS 是 arm64,而 Google 的 `macos` 前缀可能是 +x86_64 或 universal)。**查不清就不要发 —— 发一个跑不起来的载荷比不发更坏。** + +### 6.3 五期的准入条件,而不是排期 + +**ROCm/AMD**:规则已存在并按名字拒绝 AMD 平台。两个条件:(a) ROCm 运行时能否作为载荷 +分发(许可 + 与内核模块的 ABI 耦合决定它是载荷还是宿主能力);(b) **有没有一台能跑的 +机器或一个模拟器**。Ascend 那条之所以可接受,正是因为工具包自带 38 个 SoC 模拟器。 +**没有 (b) 就不开这条 lane** —— 只能编译不能运行的后端,判据上限就是编译。 + +**Metal**:形状不同。`.metal → .air → metallib` 是**两步且第二步是链接**,正是 +docs/20 列在「未实现」里的 device link;工具只在 Xcode 里、不可再分发,与「不依赖 +Host」相反,需要一次明确的例外裁决。建议**先不做,把它当作 device link 这条通用能力的 +第一个消费者来设计**。今天 `llamacpp` 的 `backend-metal` feature 已经能用(上游自己 +构建 Metal 部分),说明 Metal 在**依赖层**是通的,缺的只是把 `.metal` 变成设备源。 + +## 7. 跨平台一致性:四条可执行的规则 + +从上面抽出来,写下来是为了下一条 lane 不用重新推: + +1. **一致性由规则的选择能力提供,不由载荷的完全对齐提供。** 判据落在产物上,不落在 + 路线上。 +2. **能被断言的和不能被断言的分开摆。** 离屏进默认目标,窗口进 feature;有软件设备的 + 平台跑,没有的只构建。 +3. **驱动是宿主能力,设备是载荷。** 适配器层是 Linux 私有 loader 的产物,不该被补到 + 三平台 —— 它在那两个平台上不存在。 +4. **平台的空格分三种,处理方式不同**:上游有产物(发布工作)、上游只有源码(自建, + 要论证值得)、上游不发(结构性,要设计绕过)。**把三者混为一谈是这份方案最想避免的 + 错误** —— macOS 的 CUDA 不是"还没做",是不会有。 + +## 8. 判据总表 + +| # | 期 | 断言 | +|---|---|---| +| G1–G5 | 一 | 图形示例:两个头存在、magic 正确、跑到 lavapipe、像素在插值范围、`--no-accel` 反向腿 | +| S1 | 一 | 两个同名不同目录的 shader:**拒绝并点名两者**(今天静默覆盖) | +| P1 | 二 | 三平台各产出两个 SPIR-V 头,magic 正确 | +| P2 | 二 | 规则报出的编译器 fact 与平台表一致 | +| P3 | 二 | 工程侧 `[xlings.workspace]` 为空 | +| P4 | 二 | **Linux 读数不变** | +| C1 | 三 | Windows 上 `rules-cuda` 夹具构建通过,工程侧零声明 | +| C2 | 三 | 两端镜像逐字节比对,一个 sha256 命名两者 | +| D1 | 四 | Windows/macOS 上**跑到设备**并断言设备名 —— 这一条成立,CI 上限才从构建变成运行 | + +## 9. 自我 review:最可能错的三处 + +| 判断 | 为什么可能错 | 怎么发现 | +|---|---|---| +| 「离屏渲染在 lavapipe 上像素确定」 | 光栅化允许实现差异,插值精度未必逐位确定。**而 G4 是这个示例存在的理由** | 一期先跑一次读真实值,再把断言写成**区间**;区间写不出来就退到「中心像素明显不等于清除色」。**不要先写断言再调实现** | +| 「同名 stem 会覆盖」 | 从代码读出来的,没跑过 | 一期示例里放两个同名不同目录的 shader,先看它坏 | +| 「Windows 的 CUDA 载荷装上就能用」 | redist 的 Windows 组件是 `.zip` 且布局与 Linux 不同;规则的路径推导按 Linux 布局写的 | 三期先在一台 Windows 上手工解开、跑 `nvcc --version`,再动 recipe | + +第一条最该被推翻。第三条最容易被低估:**「上游有产物」不等于「重打包就能用」**,规则 +侧的路径推导是按一个平台的布局写的,而那正是本轮 `xpkg_dir` 缺陷的同一种形状 —— +一个函数只在它被写的那个环境里被验证过。 + +--- + +## 10. 实施回填(2026-09-07) + +这一节把上面的判断与实际发生的事对齐。**跑**出来的写在这里,**读**出来的如果被推翻 +也写在这里。分期没有被采纳:一二三期在同一天一起做了,四五期落成了记录在案的决定。 + +### 10.1 §9 的三条自我 review,读数 + +| 判断 | 结果 | +|---|---| +| 「离屏渲染像素确定」 | **比预期强**。lavapipe 与自写的软件光栅器给出**逐字节相同**的中心像素 `(124, 70, 62, 255)`,断言不需要写成区间。写法仍按 §9 的要求:先跑一次读真实值,再写断言 | +| 「同名 stem 会静默覆盖」 | **被推翻,但缺陷是真的**。ninja 在加载图时就报 `multiple rules generate .../triangle_vert.h` —— 从来不是静默的。真正的缺陷是它报的是**生成的文件**而不是那两个源、以图加载失败的形态到达、并且不说出路。规则现在自己先查一遍并点名两者 | +| 「Windows 的 CUDA 载荷装上就能用」 | **成立,而且是这一轮里最贵的一条**。见 10.2 | + +### 10.2 「上游有产物」与「重打包就能用」之间的四处 + +Windows 的组件确实只需要一个 `windows` 段,但那四个配方与另外二十个共用一套 +install/config 形状,而那套形状是按 Linux 写的: + +- `payload_root()` 只剥 `.tar.xz`。Windows 组件是 zip,按名字找不到解包目录,回落到 + 扫描 —— 而被扫的是**共享**下载目录,谁先在那里解出一个 `bin/` 就装谁。 +- `scan_dir()` 用 `find` 加 GNU 参数。Windows 上 `find` 是 System32 那个按内容搜 + 字符串的命令:它拒绝这些参数、往 stderr 写一行用法、返回空。**一个什么都没注册的 + 组件,和一个本来就没有程序的组件,读数完全一样。** +- `reunite_backend()` 的 `ln` 在 Windows 上不存在。12.x 线自带后端所以到不了,但一个 + 13.x 的 Windows 条目会到 —— 它现在拒绝而不是执行。 +- 版本齐平检查(`tests/test_platform_version_parity.py`)的入口是 + `content.find("xpm")` —— 文件里第一次出现这三个字母,而这些配方的注释里写着 + `xpm..deps`。**十五个配方对这条规则完全不可见**,其中 node.lua 声明了三个平台。 + 修好之后补了一条分母判据。 + +### 10.3 三处只在 Linux 之外成立的缺陷,由「每条规则都为本宿主编译一次」抓到 + +`tests/all-rules-compile` 是这一轮加的夹具:不点名任何 accelerator(每条规则立刻 +返回,一个字节都不下载),只问六个模块编不编得过。它在第一批运行里抓到: + +1. **macOS 14:`std::println` 不是 header-only。** 它的两个重载都要到 libc++ + **dylib** 里取 `__is_posix_terminal` 与 `__get_ostream_file`,而这两个符号是在 + macOS 14 不带的那一版里加进去的。规则改用 `std::format` 再 stream。 + + **引擎侧那一半试过了,而显而易见的修法更糟。** 直觉是让 + `host_link_tokens` 的「信任 cfg」出口也发 `-L<载荷>/lib`,于是 `-lc++` 找到工具链 + 自己那份 —— 而那正是 `dist::mechanism_for` 在 Mach-O 上明确拒绝的 + ToolchainCoupled:LLVM 的 macOS libc++abi 与 libunwind dylib **向上链接** + `/usr/lib/libc++`,系统 libc++ 与工具链的那份同时载入,跨两份释放的对象在 libmalloc + 里 abort(#202)。CI 报的正是这条路的第一步 —— 链接停在 `__cxa_end_catch` 与其余 + 那些系统 libc++ 会再导出、载荷那份不会的 ABI 符号上。 + + **这条是这一轮里唯一一个「我的修复本身被实测推翻」的。** 它被推翻的方式值得记: + 本地全部单测绿、判据是我自己写的那条正向断言,而**红在一个我没想到会受影响的对象上** + —— 图形示例的构建程序,在 macOS 15 上。改回去,把不对称与它的理由写进代码,并把判据 + 改成陈述这条决定(`OnlyTheSpelledOutExitNamesTheToolchainRuntimeDirs`)。 + **限制照实写下来:macOS 14 上构建程序不能用 `std::print` / `std::println`。** +2. **Windows:`popen` 拼作 `_popen`。** `rules-spirv` 的宿主模块在那里编译失败。 +3. **Windows:版本约束里的 `>` 被 cmd.exe 读成重定向。** mcpp 把供给请求作为 JSON + 参数放在 shell 命令行上;`shell::quote` 回答的是子进程的 argv 解析(`\"`),而 + cmd 不认这个转义,于是走到 `>` 时引号数是偶数。`>=` 正是每个规则包声明下界用的 + 形态,而**在此之前没有任何一条能在 Windows 上生效的声明带过 `>`**。已修(先按子 + 进程规则引用,再给每个 cmd 元字符前缀 `^`),判据是两个解析器的模拟器加一条反向腿。 + +三条的共同形状:**一段代码的正确性依赖于宿主,而 CI 只在其中一个宿主上执行它**。 +这正是 §9 第三行说的那个形状,只是它出现的次数比预期多。 + +### 10.4 分期没有采纳,四期与五期落成决定 + +- **一期(图形离屏)**:`examples/10-graphics/offscreen`,判据是像素。已实现。 +- **二期(shaderc 三平台)**:xim-pkgindex #778,规则按平台选编译器。已实现。 +- **三期(CUDA/SYCL 上 Windows)**:xim-pkgindex #779(五个包),规则侧的路径推导与 + 路线判断。已实现;端到端由 `windows-test` job 装卸五个包并断言注册的程序验证。 +- **四期(软件设备把 CI 上限抬到运行)**:**未实现,记录为发布工作**。Windows 上需要 + 一个 Mesa-on-Windows 的 `vulkan_lvp` 构建;macOS 上 runner 自带 GPU 与 MoltenVK, + 所以那一侧不需要软件设备,需要的是把 `compat.vulkan` 的 macOS 腿接到示例上。 + 今天的上限:三平台**构建**,Linux **运行**。 +- **五期(ROCm / Metal 准入)**:**决定是不收**,理由写在 §6.3,这里只补两条读数 —— + Metal 的编译器只在 Xcode 内、不可再分发,所以它不是一个打包问题;ROCm 的运行时可 + 再分发,但它需要的是一个 `rules-hip` 的 AMD 平台实现,而不是一个包。 + +### 10.4b 第四处「读出来的写法是错的」:依赖不能被 layer 条件化 + +图形示例最初把 Vulkan loader、运行时适配器与软件设备三条都门控在 +`cfg(accelerator = "vulkan")` 上,理由是「`--no-accel` 一个字节都不装」。**这条做不 +到,而且它失败的方式是安静的一半**:`accelerator` 是**从依赖图里解出来的**,所以一个 +由它选出的依赖会决定它自己正在问的那个答案 —— mcpp 因此忽略这个谓词并给出警告,而 +**同一个谓词下的 `[build]` 源照常生效**。结果是包被丢掉、包含它的源被留下: + + src/vulkan/render.cpp:20:10: fatal error: vulkan/vulkan.h: No such file or directory + +判据是构建本身。谓词现在按平台写(`cfg(linux)` 用于只有 Linux 有的两项),包是无条件 +的,accelerator 只选 `[build]` 源 —— 也就是既有的 `examples/09-heterogeneous/vulkan` +一直在用的形状。示例 README 把这一条写成了正文,因为它是一条使用者会撞上的规则。 + +顺带一条读数:两条腿的中心像素**逐字节相同**(`(124, 70, 62, 255)`),所以 CI 的反向 +腿从「CPU 腿跑起来了」加强成「两条腿报出同一个像素、不同的设备名」。 + +### 10.4c 图形示例在三个平台上都构建通过,而先红的是判据自己 + +macOS 与 Windows 上的构建都成功了 —— 规则供给了 `xim:shaderc@2026.3`,两个着色器阶段 +都编译了,Vulkan 那一半也链接上了 loader 包。**红的是我写的那条断言**: + + target/.build-mcpp/out/spirv/triangle_vert.h carries no SPIR-V magic + +因为规则会**选**编译器,而两个编译器把声明拆得不一样:glslang 写出完整的 +`const uint32_t ...[] = {...}`(magic 在 `.h` 里),glslc 写出初始化列表、由规则在外面 +补声明(magic 在 `.inc` 里)。**一条只点名 `.h` 的断言,是一条关于某一个编译器的断言** +—— 正是这个 job 存在的理由所要抓的那种形状,只不过这次它抓到的是自己。 + +顺带一个可迁移的小陷阱:改成 `grep -qs '' a.h a.inc` 是错的。**GNU grep 在被 +点名的文件不存在时返回 2,即使它在前一个文件里匹配到了,也即使加了 `-s`** —— `-s` 压的 +是消息不是状态。于是这条判据在「只产出 header」的那条路上永远失败,而失败的原因与被测 +的性质无关。正确写法是逐个文件测。 + +### 10.5 一条留下的不一致,以及它什么时候消失 + +`mcpp:plugins` 0.2.5 里 `xim:shaderc` 在 macOS 与 Windows 上是**精确版本**,而 +Linux 上的 `xim:glslang` 是**下界**。这不是形态判断的差异,是 10.3 第 3 条的后果: +发布中的引擎(2026.9.6.6)传不过去一个 `>`。引擎修复发布之后,那两处改回 +`>=2026.3`,而**那时 plugins 的 Windows job 就是这个引擎修复的端到端判据** —— +今天它只有单元级的两个模拟器。 diff --git a/.github/tools/build_examples.sh b/.github/tools/build_examples.sh index d1ef0b6e..ffcec76e 100755 --- a/.github/tools/build_examples.sh +++ b/.github/tools/build_examples.sh @@ -51,6 +51,7 @@ SKIP=( "examples/09-heterogeneous/hip/app|same, for the HIP payloads" "examples/09-heterogeneous/sycl/app|needs the dpcpp payload (over a gigabyte) and a device its runtime accepts" "examples/09-heterogeneous/vulkan/app|built AND RUN by the next step of this job, on the lavapipe payload, which needs no GPU" + "examples/10-graphics/offscreen|built AND RUN by its own step of this job, on the same lavapipe payload. It renders a real graphics pipeline offscreen and asserts the pixels, which is why it runs there rather than here" "examples/09-heterogeneous/cann/app|its device leg needs the Ascend DRIVER, which a runner does not have: the kernel compiles and the object links, and then `libascend_hal.so` is missing -- and only that one, since 2026.9.6.6 models DT_RPATH inheritance. Correct on a machine with no NPU. Its CPU leg does build, and is not built here only because it would make this job resolve a fifth rule package for one example. Covered by the measurements in its README" ) diff --git a/.github/workflows/ci-linux.yml b/.github/workflows/ci-linux.yml index 7d29c825..c87d5c45 100644 --- a/.github/workflows/ci-linux.yml +++ b/.github/workflows/ci-linux.yml @@ -285,6 +285,65 @@ jobs: export MCPP_VENDORED_XLINGS="$XLINGS_BIN" bash .github/tools/build_examples.sh + - name: "Graphics example: render offscreen on lavapipe and assert the pixels" + run: | + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + cd examples/10-graphics/offscreen + "$MCPP" build + # THE SHADERS ARE THE FIRST ASSERTION, and both of them: a rule that + # compiled only the first source would leave the second header absent + # and everything after this would still pass. + for f in triangle_vert triangle_frag; do + d="target/.build-mcpp/out/spirv" + test -f "$d/$f.h" || { echo "missing $d/$f.h"; exit 1; } + # Either file: which of the two carries the words is a property of + # the shader compiler the rule chose, not of the shader. See the + # cross-platform jobs, where that choice differs. + # ONE FILE AT A TIME, because `grep -qs a b` exits 2 when `b` + # does not exist -- even on a match in `a`, and even with `-s`, + # which suppresses the message and not the status. Written as one + # grep over both names, this criterion fails whenever the route + # that produces only a header is taken, which is a failure about + # the criterion and not about the shader. + found="" + for g in "$d/$f.h" "$d/$f.inc"; do + [ -f "$g" ] && grep -q '0x07230203' "$g" && found=1 + done + [ -n "$found" ] \ + || { echo "$f carries no SPIR-V magic in either $f.h or $f.inc"; exit 1; } + done + icd=$(find "${MCPP_HOME:-$HOME/.mcpp}/registry/data/xpkgs/xim-x-mesa-lavapipe" \ + "$HOME/.xlings/data/xpkgs/xim-x-mesa-lavapipe" \ + -name 'lvp_icd.x86_64.json' -print -quit 2>/dev/null || true) + [ -n "$icd" ] || { echo "no lavapipe ICD in either store"; exit 1; } + out=$(VK_DRIVER_FILES="$icd" "$MCPP" run 2>&1) || { echo "$out"; exit 1; } + echo "$out" + # The program asserts the corners and the centre itself and exits + # non-zero on either. What CI adds is that the run reached the DEVICE: + # the software rasteriser produces the same pixels by construction, so + # the image cannot distinguish them and the device name is what does. + echo "$out" | grep -q 'llvmpipe' \ + || { echo "the run did not reach the lavapipe device"; exit 1; } + echo "$out" | grep -qE 'centre pixel: \([0-9]+, [0-9]+, [0-9]+, 255\)' \ + || { echo "no centre pixel was reported"; exit 1; } + # THE REVERSE LEG, AND IT COMPARES THE PIXELS RATHER THAN JUST + # RUNNING. The claim this example makes is that the image is a + # contract two independent rasterisers satisfy, not a property of one + # device -- so the criterion is that the two legs report the SAME + # centre pixel while reporting different devices. Measured: both give + # (124, 70, 62, 255), byte for byte. + gpu_px=$(echo "$out" | grep -m1 '^centre pixel:') + "$MCPP" build --no-accel + "$MCPP" run --no-accel | tee cpu.log + grep -q 'cpu rasteriser' cpu.log || { echo "the CPU leg did not run"; exit 1; } + cpu_px=$(grep -m1 '^centre pixel:' cpu.log) + [ -n "$gpu_px" ] && [ "$gpu_px" = "$cpu_px" ] || { + echo "the two legs disagree about the image:" + echo " device: $gpu_px" + echo " cpu: $cpu_px" + exit 1; } + echo "ok: both legs agree on $cpu_px" + - name: "Vulkan example: build the device half and run it on lavapipe" run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" diff --git a/.github/workflows/ci-macos.yml b/.github/workflows/ci-macos.yml index 7e143ef3..35513b78 100644 --- a/.github/workflows/ci-macos.yml +++ b/.github/workflows/ci-macos.yml @@ -310,6 +310,57 @@ jobs: "$MCPP" build "$MCPP" --version + + # GRAPHICS ON THIS HOST, BUILD ONLY, AND THAT IS THE WHOLE CLAIM. + # + # This runner has no Vulkan device, so what is asserted is what this + # platform decides: that the shader compiler THIS platform uses -- the + # rule declares `xim:shaderc` here and `xim:glslang` on Linux -- produces + # both SPIR-V headers, and that the Vulkan half compiles and links + # against the loader package. Running it is the Linux job's criterion, + # where a software device (`xim:mesa-lavapipe`) is published and the two + # legs' pixels are compared. + # + # The example is otherwise built only on Linux (`build_examples.sh` runs + # there), which is exactly the shape this change exists to remove: the + # half of a lane written for a host is the half that host never + # exercises. + - name: "Graphics: the offscreen example builds on this host" + shell: bash + run: | + set -e + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + cd "$GITHUB_WORKSPACE/examples/10-graphics/offscreen" + # The toolchain is NAMED rather than inherited: leaving it to whatever + # a neighbouring step happened to select makes this step's subject + # depend on step order, which is not a property anybody reads. + "/tmp/mcpp-fresh" build --toolchain "llvm@${MCPP_LLVM_VER}" + for f in triangle_vert triangle_frag; do + d="target/.build-mcpp/out/spirv" + test -f "$d/$f.h" || { echo "missing $d/$f.h"; exit 1; } + # THE MAGIC IS NOT ALWAYS IN THE HEADER, AND THAT IS THE POINT OF + # THIS JOB. The rule chooses the shader compiler this platform + # publishes -- glslang on Linux, glslc here -- and the two split + # the declaration differently: glslang writes a complete `const + # uint32_t ...[] = {...}`, glslc an initialiser list the rule + # declares around, so the words land in `.inc`. An assertion + # naming only the header is an assertion about ONE compiler, which + # is exactly the shape this step exists to catch. + # ONE FILE AT A TIME, because `grep -qs a b` exits 2 when `b` + # does not exist -- even on a match in `a`, and even with `-s`, + # which suppresses the message and not the status. Written as one + # grep over both names, this criterion fails whenever the route + # that produces only a header is taken, which is a failure about + # the criterion and not about the shader. + found="" + for g in "$d/$f.h" "$d/$f.inc"; do + [ -f "$g" ] && grep -q '0x07230203' "$g" && found=1 + done + [ -n "$found" ] \ + || { echo "$f carries no SPIR-V magic in either $f.h or $f.inc"; exit 1; } + done + echo "ok: both shader stages compiled and the Vulkan half linked" + # Integration: the mcpp built from THIS PR's source (the self-host binary, # $MCPP = /tmp/mcpp-fresh) builds & runs a real external C++ project — # xlings (openxlings/xlings ships its own mcpp.toml). diff --git a/.github/workflows/ci-windows.yml b/.github/workflows/ci-windows.yml index a96089fa..b362d233 100644 --- a/.github/workflows/ci-windows.yml +++ b/.github/workflows/ci-windows.yml @@ -405,6 +405,57 @@ jobs: # restore the LLVM default for the remaining steps "$MCPP_SELF" toolchain default llvm@20.1.7 + + # GRAPHICS ON THIS HOST, BUILD ONLY, AND THAT IS THE WHOLE CLAIM. + # + # This runner has no Vulkan device, so what is asserted is what this + # platform decides: that the shader compiler THIS platform uses -- the + # rule declares `xim:shaderc` here and `xim:glslang` on Linux -- produces + # both SPIR-V headers, and that the Vulkan half compiles and links + # against the loader package. Running it is the Linux job's criterion, + # where a software device (`xim:mesa-lavapipe`) is published and the two + # legs' pixels are compared. + # + # The example is otherwise built only on Linux (`build_examples.sh` runs + # there), which is exactly the shape this change exists to remove: the + # half of a lane written for a host is the half that host never + # exercises. + - name: "Graphics: the offscreen example builds on this host" + shell: bash + run: | + set -e + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + cd "$GITHUB_WORKSPACE/examples/10-graphics/offscreen" + # The toolchain is NAMED rather than inherited: leaving it to whatever + # a neighbouring step happened to select makes this step's subject + # depend on step order, which is not a property anybody reads. + "$MCPP_SELF" build --toolchain "llvm@20.1.7" + for f in triangle_vert triangle_frag; do + d="target/.build-mcpp/out/spirv" + test -f "$d/$f.h" || { echo "missing $d/$f.h"; exit 1; } + # THE MAGIC IS NOT ALWAYS IN THE HEADER, AND THAT IS THE POINT OF + # THIS JOB. The rule chooses the shader compiler this platform + # publishes -- glslang on Linux, glslc here -- and the two split + # the declaration differently: glslang writes a complete `const + # uint32_t ...[] = {...}`, glslc an initialiser list the rule + # declares around, so the words land in `.inc`. An assertion + # naming only the header is an assertion about ONE compiler, which + # is exactly the shape this step exists to catch. + # ONE FILE AT A TIME, because `grep -qs a b` exits 2 when `b` + # does not exist -- even on a match in `a`, and even with `-s`, + # which suppresses the message and not the status. Written as one + # grep over both names, this criterion fails whenever the route + # that produces only a header is taken, which is a failure about + # the criterion and not about the shader. + found="" + for g in "$d/$f.h" "$d/$f.inc"; do + [ -f "$g" ] && grep -q '0x07230203' "$g" && found=1 + done + [ -n "$found" ] \ + || { echo "$f carries no SPIR-V magic in either $f.h or $f.inc"; exit 1; } + done + echo "ok: both shader stages compiled and the Vulkan half linked" + - name: "Toolchain: LLVM — build mcpp (self-host)" shell: bash run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index 930574dd..ead5aea6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,65 @@ ## [Unreleased] +## [2026.9.7.1] - 2026-09-07 + +### 两处只在 Linux 之外成立的缺陷,以及第一条图形管线 + +**引擎与规则层早就与平台无关,而生态只在 Linux 上完整。** 把规则包在另外两个平台上 +真跑一遍,暴露了两处引擎缺陷 —— 两处的共同形状是:一段代码的正确性依赖于宿主,而 CI +只在其中一个宿主上执行它。 + +**macOS 14 上构建程序用不了 `std::println`,而显而易见的修法更糟(已记录,未修)。** +`host_link_tokens` 的「信任 cfg」出口不给 `Toolchain::linkRuntimeDirs` 发 `-L`,于是 +macOS 上 `-lc++` 经 SDK 解析到**系统**那份,而头文件来自载荷。macos-14 上编译一个只有 +`import std` 的构建程序: + + ld64.lld: error: undefined symbol: std::__1::__is_posix_terminal(__sFILE*) + +`std::print` 不是 header-only 的,它的两个重载都要到 libc++ **dylib** 里取支持符号, +而那两个符号是在 macOS 14 不带的那一版里加进去的。macOS 15 有,所以这个项目用到的每 +一台 macOS runner 都是绿的。 + +在这一条上加 `-L<载荷>/lib` 试过了,**它买来一个更糟的问题**:那会让 `-lc++` 解析到 +工具链自己的 dylib,也就是 `dist::mechanism_for` 在 Mach-O 上明确拒绝的 +ToolchainCoupled —— LLVM 的 macOS libc++abi 与 libunwind dylib **向上链接** +`/usr/lib/libc++`,于是系统 libc++ 与工具链的那份同时载入,跨两份释放的对象在 +libmalloc 里 abort(#202)。CI 报的正是这条路的第一步:链接停在 `__cxa_end_catch` +与其余那些系统 libc++ 会再导出、而载荷那份不会的 ABI 符号上。 + +所以这个宿主上的 C++ 运行时就是系统那一份,而下限由分发契约选的**静态 libc++** 变成 +真的 —— 那是产物的机制,不属于一个 mcpp 编译、就地跑一次、然后丢掉的辅助程序。 +**限制照实写下来:macOS 14 上构建程序不能用 `std::print` / `std::println`。** +`std::format` 是 header-only 的,没有这个问题;本轮 mcpp:plugins 的六个模块因此全部 +改用它。判据 `HostFlags.OnlyTheSpelledOutExitNamesTheToolchainRuntimeDirs` 把这条 +不对称写成了一条会跑的断言。 + +**版本约束里的 `>` 被 cmd.exe 读成重定向(Windows)。** mcpp 把供给请求作为 JSON 参数 +放在 shell 命令行上;`shell::quote` 回答的是子进程的 argv 解析(`\"`),而 cmd 不认这 +个转义,于是走到 `>` 时引号数是偶数,`>` 成了重定向: + + Provisioning [xlings.workspace] entries declared by dependencies (xim:shaderc@>=2026.3) + The filename, directory name, or volume label syntax is incorrect. + +`>=` 正是每个规则包声明下界用的形态,而在此之前没有任何一条能在 Windows 上生效的声明 +带过 `>`。修法是标准的双重转义,判据是两个解析器的模拟器加一条反向腿。 + +### `examples/10-graphics/offscreen`:第一条图形管线,判据是像素 + +此前所有异构示例都是计算。这一个是图形:顶点与片段两个着色器阶段、一条 render pass、 +`vkCmdDraw` 一个三角形、`vkCmdCopyImageToBuffer` 取回像素。**离屏而不是开窗**,因为那 +是可断言的形态。同一道接缝背后是一个自己写的软件光栅器,而两条腿的中心像素**逐字节 +相同** —— 图像是契约,设备名是唯一区分它们的东西。 + +这个示例挖出一条使用者会撞上的规则:**依赖不能被 layer 条件化**。 +`cfg(accelerator = ...)` 下的 `[build]` 源生效而依赖被忽略,于是包被丢掉、包含它的源 +被留下。 + +### 文档 + +`docs/20` 新增「每条 lane 到得了哪些平台」:三件事同时为真才叫一条 lane 在某个平台上 +成立,而第三件(规则自己那段按宿主分岔的代码编译得过)是最容易被默认成立的那一件。 + ## [2026.9.6.6] - 2026-09-07 ### 一个包一个版本:规则自带环境,工程只写例外 diff --git a/docs/01-examples.md b/docs/01-examples.md index 8cb133b5..98285aec 100644 --- a/docs/01-examples.md +++ b/docs/01-examples.md @@ -38,6 +38,8 @@ examples. | 09d | [`…/hip`](../examples/09-heterogeneous/hip/) | The same computation in HIP, reaching an NVIDIA device | `mcpp.rules.hip`, HIP as a header layer over the CUDA runtime, a two-chunk `accel` | | 09e | [`…/multi-backend`](../examples/09-heterogeneous/multi-backend/) | Several backends in ONE artifact, chosen at run time — the library shape, not the program shape | `accel` as a set, `cfg(accelerator = "none")` and its negation, a dispatch chain, a module seam over a C island boundary | | 09f | [`…/cann`](../examples/09-heterogeneous/cann/) | An Ascend C kernel behind the same seam. **Does not build yet** — its README names the two missing pieces | the `.asc` device extension, `op_kernel`/`op_host` as an island CANN already has, `accelerator = "none"` for the fallback | +| 10 | [`examples/10-graphics`](../examples/10-graphics/) | Graphics rather than compute: a rendering pipeline whose result is pixels | `mcpp.rules.spirv` for the vertex and fragment stages, offscreen rendering as the assertable form | +| 10a | [`…/offscreen`](../examples/10-graphics/offscreen/) | A triangle rasterised by Vulkan into a buffer, and the same triangle by a software rasteriser behind the same seam | two shader stages from one glob, a render pass with no window or swapchain, a pixel as the criterion | ## Suggested Reading Order diff --git a/docs/20-heterogeneous-builds.md b/docs/20-heterogeneous-builds.md index 6165a4ac..722334ef 100644 --- a/docs/20-heterogeneous-builds.md +++ b/docs/20-heterogeneous-builds.md @@ -483,8 +483,8 @@ own denominator. |---|---|---|---|---| | `rules-cuda` | `mcpp.rules.cuda` | the project's own clang (`-x cuda`), or nvcc with a GCC toolchain | `xim:cuda-nvcc`, `xim:cuda-cudart`, `xim:libcurand`, `xim:cuda-cccl` | `cuda12.9+{sm_89} ptx>=89` | | `rules-hip` | `mcpp.rules.hip` | the project's own clang (`-x cuda`) on the NVIDIA platform | the above plus `xim:hip-nvidia` | `hip, cuda12.9+{sm_89}` | -| `rules-sycl` | `mcpp.rules.sycl` | the `xim:dpcpp` payload's clang (`-fsycl`) | `xim:dpcpp`, `xim:gcc`, `xim:cuda-nvcc` for an NVIDIA target | `sycl` or `sycl, cuda12.9+{sm_89}` | -| `rules-spirv` | `mcpp.rules.spirv` | `glslangValidator` or `glslc` | `xim:glslang` or `xim:shaderc` | `vulkan1.2` | +| `rules-sycl` | `mcpp.rules.sycl` | the `xim:dpcpp` payload's clang (`-fsycl`) | `xim:dpcpp`; on Linux also `xim:gcc`, `xim:glibc`, `xim:linux-headers`; `xim:cuda-nvcc` for an NVIDIA target | `sycl` or `sycl, cuda12.9+{sm_89}` | +| `rules-spirv` | `mcpp.rules.spirv` | `glslangValidator` or `glslc` | `xim:glslang` on Linux, `xim:shaderc` on macOS and Windows | `vulkan1.2` | | `rules-ascendc` | `mcpp.rules.ascendc` | `bisheng` (`-x asc`) from the CANN toolkit | `xim:cann-toolkit` | `ascend8.5+{dav-c220}` | The payload column is what each rule declares for itself under @@ -512,6 +512,71 @@ unwinder symbols they share. Nothing may cross the seam: a SYCL exception is caught in the device translation unit and returned as a code, because the runtime that threw it is not the one the caller would unwind with. +## Which platforms each lane reaches + +A lane reaches a platform when three things hold there: the device compiler is +published for it, the runtime the produced artifact needs can be reached, and +the rule's own host-dependent code compiles for it. The third is the one that +is easy to assume. `mcpp:plugins` compiles every rule for every platform in its +CI matrix (`tests/all-rules-compile`, a fixture that names no accelerator, so +it downloads nothing and asks only whether the modules compile); that fixture +turned three latent host differences into compile errors on the runners that +had them, and none of the three had been visible to a Linux build. + +| lane | Linux | macOS | Windows | what decides it | +|---|---|---|---|---| +| `rules-spirv` | yes | yes | yes | the shader compiler is published for all three: `xim:glslang` on Linux, `xim:shaderc` on macOS arm64 and Windows x86_64 | +| `rules-cuda` | yes | no | yes | NVIDIA publishes the redistributable components for Linux and Windows and has published no macOS toolkit since CUDA 10.2 | +| `rules-sycl` | yes | no | Level Zero and OpenCL only | Intel publishes `sycl_linux` and `sycl_windows` from one tag and nothing for macOS; upstream states that the CUDA and HIP plugins are not built for Windows, and the Windows asset carries only the Level Zero and OpenCL adapters | +| `rules-hip` | yes | no | no | the NVIDIA-platform header package is published for Linux alone; the AMD platform needs a ROCm runtime this ecosystem does not publish anywhere | +| `rules-ascendc` | yes | no | no | the CANN toolkit is published for Linux alone | + +**"Reaches" is not the same claim on every row, and the difference is stated +rather than left to be inferred.** What has been RUN on all three platforms is +`rules-spirv`: the shader compiler produces both SPIR-V stages and the artifact +links, and on Linux it also renders and the pixels are compared against a +software rasteriser. What has been INSTALLED and COMPILED on Windows is the +CUDA and SYCL lane: the components install and register their programs, and the +rules compile for that host, but no CI runner has yet driven `nvcc` or `dpcpp` +there end to end. A row saying "yes" therefore means the three conditions above +hold; it does not mean a runner has executed that lane. + +**A vendor that does not publish for a platform ends the question.** No amount +of engine work makes a CUDA toolkit exist for macOS. What the ecosystem can do +is state the boundary at the point where a build asks to cross it, which is +what each lane does: the SYCL rule refuses an ahead-of-time NVIDIA target on +Windows and names the upstream release note that decides it, rather than +compiling something the runtime cannot load. + +**Where a lane reaches a platform, it reaches it the same way.** Four +differences are the whole of what a rule does differently per host, and each is +a property of the host rather than of the device: + +- **The suffix on a program name.** `nvcc` and `nvcc.exe` are the same tool. +- **Where the libraries are.** `lib` and `lib64` on ELF hosts, `lib/x64` in + NVIDIA's Windows layout. +- **Which host compiler the device compiler drives.** On Windows the CUDA rule + takes its clang route whatever the project's compiler is: the nvcc route + compiles the host half through a compiler named by `-ccbin`, and on that host + the only one it accepts is MSVC's `cl.exe`, whose location is found by asking + the machine about its Visual Studio installation. The clang route drives no + second compiler and locates the MSVC headers itself. +- **Which of the host's libraries have to be kept out.** On Linux the SYCL rule + names `xim:gcc`, `xim:glibc` and `xim:linux-headers` because dpcpp's clang is + not the clang mcpp resolved and is configured with neither library. On + Windows there is one C++ runtime, MSVC's, and both compilers use it, so those + three declarations do not exist there -- requiring them would refuse a build + over three packages that this ecosystem does not publish for that platform + and that its compiler does not need. + +**The runtime adapters are a Linux construction.** `compat:cuda-runtime`, +`compat:sycl-runtime` and `compat:vulkan-runtime` exist because an mcpp +artifact on Linux runs behind a private loader that does not consult +`/usr/lib`, so a vendor library installed by a driver package has to be brought +onto the artifact's own search path. macOS (dyld) and Windows (the PE loader) +have no such layer by construction, and a project targeting them declares no +adapter. + ## What a framework looks like on top of this The five lanes prove a rule package can drive five compilers, the newest of @@ -558,6 +623,20 @@ past it is the framework's own selector rather than a change to the packaging. ## Not implemented Device targets and the device linking they imply for the island shape, OpenMP -offload and stdpar, the AMD platform of HIP, and Metal. See +offload and stdpar, the AMD platform of HIP, and Metal. + +Two further gaps are per-platform rather than per-model, and both are +publishing work rather than engine work. **HIP on Windows** needs Windows +sections for the NVIDIA-platform header package and for `cuda-profiler-api`, +plus a Windows form for the header-declaration step that today writes into a +Linux SubOS view. **The 13.x CUDA line on Windows** needs a Windows form for +the back-end reunification: on that line upstream splits `nvvm/` and `crt/` out +of `cuda_nvcc` into four separately published components, which the index +reunites with symlinks, and `ln` is not a command on that host. The 12.x line +keeps its back end inside the component and needs none of it, so that is the +line Windows carries, declared as a deliberate divergence in each recipe rather +than left to be read off the file. + +See `.agents/docs/2026-09-05-heterogeneous-build-ecosystem-design-v2.md` for the design these follow from and the reason each is open. diff --git a/docs/zh/01-examples.md b/docs/zh/01-examples.md index a7670518..46ac445c 100644 --- a/docs/zh/01-examples.md +++ b/docs/zh/01-examples.md @@ -35,6 +35,8 @@ mcpp build && mcpp run | 09d | [`…/hip`](../../examples/09-heterogeneous/hip/) | 同一个计算写成 HIP,够到一台 NVIDIA 设备 | `mcpp.rules.hip`、HIP 作为 CUDA 运行时之上的一层头文件、两段式的 `accel` | | 09e | [`…/multi-backend`](../../examples/09-heterogeneous/multi-backend/) | 多个后端进**同一个产物**,运行期选择 —— 这是库的形态,不是程序的形态 | `accel` 作为集合、`cfg(accelerator = "none")` 及其否定、分发链、C 岛边界之上的模块接缝 | | 09f | [`…/cann`](../../examples/09-heterogeneous/cann/) | 同一道接缝背后的 Ascend C kernel。**目前还构建不了** —— README 里点明了缺的两块 | `.asc` 设备扩展名、CANN 本来就有的 `op_kernel`/`op_host` 岛、回退用 `accelerator = "none"` | +| 10 | [`examples/10-graphics`](../../examples/10-graphics/) | 图形而不是计算:一条渲染管线,它的结果是像素 | `mcpp.rules.spirv` 编译顶点与片段两个阶段、离屏渲染作为可断言的形态 | +| 10a | [`…/offscreen`](../../examples/10-graphics/offscreen/) | Vulkan 把一个三角形光栅化进缓冲区,同一道接缝背后是同一个三角形的软件光栅器 | 一个 glob 带出两个着色器阶段、无窗口无交换链的 render pass、判据是一个像素 | ## 推荐阅读顺序 diff --git a/docs/zh/20-heterogeneous-builds.md b/docs/zh/20-heterogeneous-builds.md index 3bbbae6a..11e30d30 100644 --- a/docs/zh/20-heterogeneous-builds.md +++ b/docs/zh/20-heterogeneous-builds.md @@ -403,8 +403,8 @@ sources = ["src/cpu/*.cpp"] |---|---|---|---|---| | `rules-cuda` | `mcpp.rules.cuda` | 工程自己的 clang(`-x cuda`),或 GCC 工具链下的 nvcc | `xim:cuda-nvcc`、`xim:cuda-cudart`、`xim:libcurand`、`xim:cuda-cccl` | `cuda12.9+{sm_89} ptx>=89` | | `rules-hip` | `mcpp.rules.hip` | NVIDIA 平台上是工程自己的 clang(`-x cuda`) | 上面那些,再加 `xim:hip-nvidia` | `hip, cuda12.9+{sm_89}` | -| `rules-sycl` | `mcpp.rules.sycl` | `xim:dpcpp` 载荷里的 clang(`-fsycl`) | `xim:dpcpp`、`xim:gcc`,NVIDIA 目标另加 `xim:cuda-nvcc` | `sycl` 或 `sycl, cuda12.9+{sm_89}` | -| `rules-spirv` | `mcpp.rules.spirv` | `glslangValidator` 或 `glslc` | `xim:glslang` 或 `xim:shaderc` | `vulkan1.2` | +| `rules-sycl` | `mcpp.rules.sycl` | `xim:dpcpp` 载荷里的 clang(`-fsycl`) | `xim:dpcpp`;Linux 上另有 `xim:gcc`、`xim:glibc`、`xim:linux-headers`;NVIDIA 目标另加 `xim:cuda-nvcc` | `sycl` 或 `sycl, cuda12.9+{sm_89}` | +| `rules-spirv` | `mcpp.rules.spirv` | `glslangValidator` 或 `glslc` | Linux 上 `xim:glslang`,macOS 与 Windows 上 `xim:shaderc` | `vulkan1.2` | | `rules-ascendc` | `mcpp.rules.ascendc` | CANN 工具包里的 `bisheng`(`-x asc`) | `xim:cann-toolkit` | `ascend8.5+{dav-c220}` | 载荷那一列是每条规则在 `cfg(accelerator = ...)` 之下**为自己**声明的东西,列出来是为了 @@ -424,6 +424,55 @@ CUDA 入口点的内联包装,所以目标文件链接的是 CUDA 运行时,机 重复符号检查会报出它们共有的那些 unwinder 符号。任何东西都不得穿过接缝:SYCL 异常在 设备编译单元里被捕获并转成返回码,因为抛出它的那个运行时不是调用方会用来展开的那个。 +## 每条 lane 到得了哪些平台 + +一条 lane 在某个平台上成立,要三件事同时为真:设备编译器为它发布了、产物需要的运行时 +在那里够得到、以及这条规则自己那段按宿主分岔的代码在那里编译得过。第三件是最容易被默认 +成立的那一件。`mcpp:plugins` 会为矩阵里的每个平台编译每一条规则 +(`tests/all-rules-compile` —— 一个不点名任何 accelerator 的夹具,因此一个字节都不 +下载,只问六个模块编不编得过);它把三处潜伏的宿主差异变成了对应 runner 上的编译错误, +而三处没有一处是 Linux 构建看得见的。 + +| lane | Linux | macOS | Windows | 由什么决定 | +|---|---|---|---|---| +| `rules-spirv` | 是 | 是 | 是 | 着色器编译器三个平台都有发布:Linux 上 `xim:glslang`,macOS arm64 与 Windows x86_64 上 `xim:shaderc` | +| `rules-cuda` | 是 | 否 | 是 | NVIDIA 为 Linux 与 Windows 发布可再分发组件,而自 CUDA 10.2 之后没有为 macOS 发布过工具包 | +| `rules-sycl` | 是 | 否 | 仅 Level Zero 与 OpenCL | Intel 从同一个 tag 发布 `sycl_linux` 与 `sycl_windows`,macOS 一个都没有;上游写明 Windows 不构建 CUDA 与 HIP 插件,而那份资产也确实只带 Level Zero 与 OpenCL 两个适配器 | +| `rules-hip` | 是 | 否 | 否 | NVIDIA 平台的头文件包只为 Linux 发布;AMD 平台要一个本生态在任何平台上都还没发布的 ROCm 运行时 | +| `rules-ascendc` | 是 | 否 | 否 | CANN 工具包只为 Linux 发布 | + +**「到得了」在每一行上不是同一个断言,而这个差别写出来而不是留给人推。** 三个平台上 +真的**跑过**的是 `rules-spirv`:着色器编译器产出两个 SPIR-V 阶段、产物链接得上,而 +Linux 上还渲染出来并把像素与一个软件光栅器逐字节比对。Windows 上**装上并编译过**的是 +CUDA 与 SYCL 那条 lane:组件装得上、注册出程序,规则也为那个宿主编译过,但还没有任何 +一台 runner 在那里端到端驱动过 `nvcc` 或 `dpcpp`。所以一行写「是」的意思是上面三个条件 +成立,不是说有 runner 执行过那条 lane。 + +**厂商没有为某个平台发布,这个问题就到此为止。** 再多的引擎工作也变不出一个 macOS 的 +CUDA 工具包。生态能做的是在一次构建请求跨过那条边界的地方把它说出来,而每条 lane 正是 +这样做的:SYCL 规则在 Windows 上拒绝一个提前编译的 NVIDIA 目标,并点名决定这件事的那 +条上游发布说明,而不是编出一个运行时装不进去的东西。 + +**一条 lane 到得了的平台,它到达的方式是同一个。** 一条规则按宿主区别对待的全部内容 +就是下面四处,而每一处都是宿主的性质,不是设备的: + +- **程序名上的后缀。** `nvcc` 与 `nvcc.exe` 是同一个工具。 +- **库在哪里。** ELF 宿主上是 `lib` 与 `lib64`,NVIDIA 的 Windows 布局里是 `lib/x64`。 +- **设备编译器驱动的是哪个宿主编译器。** Windows 上 CUDA 规则一律走 clang 路线,与 + 项目的编译器无关:nvcc 路线把设备单元的宿主那一半交给 `-ccbin` 命名的编译器,而在那 + 个宿主上它只接受 MSVC 的 `cl.exe`,找到它要问机器的 Visual Studio 安装。clang 路线 + 不驱动第二个编译器,它自己就会定位 MSVC 的头文件。 +- **要把宿主的哪些库挡在外面。** Linux 上 SYCL 规则点名 `xim:gcc`、`xim:glibc` 与 + `xim:linux-headers`,因为 dpcpp 的 clang 不是 mcpp 解析的那个 clang,两个库都没有 + 被配置过。Windows 上只有一个 C++ 运行时 —— MSVC 的 —— 两个编译器用的是同一个,所以 + 那三条声明在那里根本不存在:要求它们会让构建因为三个本生态不为那个平台发布、而那个 + 平台的编译器也不需要的包而失败。 + +**运行时适配层是 Linux 的构造。** `compat:cuda-runtime`、`compat:sycl-runtime` 与 +`compat:vulkan-runtime` 存在,是因为 Linux 上 mcpp 的产物跑在一个不查 `/usr/lib` 的 +私有 loader 后面,于是驱动包装的厂商库必须被搬到产物自己的搜索路径上。macOS(dyld) +与 Windows(PE loader)按构造就没有这一层,面向它们的项目一条适配声明都不写。 + ## 在此之上,一个框架是什么形状 五条 lane 证明了规则包能驱动五个编译器,其中最新的一个来自 NVIDIA 与 Khronos 两个 @@ -459,5 +508,15 @@ Mesa 的 lavapipe 仅因类型就被排除 —— 尽管它声明了后端要求 ## 尚未实现 岛形态的 device target 及其隐含的 device link、OpenMP offload 与 stdpar、HIP 的 AMD -平台,以及 Metal。这些所依据的设计,以及每一项仍然开着的理由,见 +平台,以及 Metal。 + +另有两处缺口是按平台而不是按编程模型分的,而且两处都是打包工作不是引擎工作。 +**HIP 上 Windows** 需要给 NVIDIA 平台的头文件包与 `cuda-profiler-api` 补 Windows 段, +以及给那一步「把头文件声明进去」补一个 Windows 形态 —— 它今天写进的是一个 Linux 的 +SubOS 视图。**CUDA 的 13.x 线上 Windows** 需要给后端的重新拼合补一个 Windows 形态: +那条线上,上游把 `nvvm/` 与 `crt/` 从 `cuda_nvcc` 里拆成四个单独发布的组件,索引用符号 +链接把它们拼回去,而 `ln` 不是那个宿主上的命令。12.x 线自带后端,一样都不需要,所以 +Windows 承载的是那条线,并在每个配方里显式声明为有意的分歧,而不是留给人从文件里读。 + +这些所依据的设计,以及每一项仍然开着的理由,见 `.agents/docs/2026-09-05-heterogeneous-build-ecosystem-design-v2.md`。 diff --git a/examples/10-graphics/README.md b/examples/10-graphics/README.md new file mode 100644 index 00000000..11635acd --- /dev/null +++ b/examples/10-graphics/README.md @@ -0,0 +1,87 @@ +# 10 — Graphics: a pipeline whose output can be asserted + +`offscreen/` renders a triangle through a real Vulkan graphics pipeline — +vertex input, rasterisation, fragment output, render pass — into an image, reads +the pixels back, and checks them. It is the graphics counterpart of +[`09-heterogeneous/vulkan`](../09-heterogeneous/vulkan), which does the same for +compute. + +``` +cd examples/10-graphics/offscreen +mcpp run # on whatever Vulkan device the loader finds +mcpp run --no-accel # the same image, from a software rasteriser +``` + +## Why offscreen rather than a window + +A swapchain needs a surface, and a surface needs a window system. On a headless +machine there is none, so an example built around one can only be **built** +there — and "it built" says almost nothing about a graphics pipeline. A fragment +shader that ignores its input, a pipeline whose vertex stage never runs, an +image never rendered into: all three compile and link. + +Rendering into an image makes the result assertable, and it exercises the same +pipeline a windowed application uses. What a window would add is presentation, +which is the one part that cannot be checked without one. + +## What the assertions are + +| | | +|---|---| +| the four corners | exactly the clear colour, with no tolerance — nothing interpolates there, and a tolerance would hide an image that was never rendered into | +| the centre | inside the triangle, so all three vertex colours contribute. **Every channel must be non-zero** | +| the device name | printed after the run, never before | + +The centre is the assertion this example exists for. A fragment shader writing a +constant would put 255 in one channel and 0 in the other two; a pipeline whose +vertex stage never ran would leave the clear colour. Requiring all three +channels to be non-zero separates those from an interpolated result **without +depending on a rasteriser's exact rounding**. + +Measured: both legs produce `(124, 70, 62, 255)` at the centre — the same bytes +from llvmpipe and from the software rasteriser in `src/cpu/`. + +## The two implementations behind one seam + +`src/vulkan/render.cpp` and `src/cpu/render.cpp` define the same three +`extern "C"` entry points and are never in one link. They produce the same +image on purpose: **the pixel test is the contract**, and the CPU leg exists to +show the contract is satisfiable without a GPU. `render_device_name()` is what +tells the two apart, which is why it is printed. + +## What the build system contributes + +```toml +[build-dependencies.mcpp] +plugins = { version = "0.2.5", features = ["rules-spirv"], host-module = true } + +[build] +accel = "vulkan1.2" +sources = [ + "src/*.cppm", "src/*.cpp", + { glob = "shaders/*.vert", accel = "vulkan1.2" }, + { glob = "shaders/*.frag", accel = "vulkan1.2" }, +] +``` + +That is the whole of it. `mcpp.rules.spirv` declares the shader compiler it +drives, so this project names no payload for it; the constrained globs route the +shaders to the build program rather than to the C++ compiler; and the generated +headers land on the include path, so `src/vulkan/render.cpp` writes +`#include "triangle_vert.h"`. + +**A dependency cannot be conditioned on the accelerator, and this project is +where that shows.** `accelerator` is resolved from the dependency graph, so a +dependency chosen by it would decide the answer it is asking for. mcpp says so +and ignores the predicate. An earlier revision of this manifest gated the Vulkan +loader on `cfg(accelerator = "vulkan")` and the build failed on +`vulkan/vulkan.h: No such file or directory` — the header's package had been +dropped while the source that includes it, selected by the same predicate, was +kept. Packages are therefore unconditional or conditioned on the platform; +`[build]` sources are what the accelerator selects. + +**One shader per stem.** The generated name is the shader's stem and its stage, +so `ui/text.vert` and `world/text.vert` would both produce `text_vert.h` +declaring `text_vert_spv`. The rule refuses that and names both files. The +directory cannot be part of the name: two headers reaching one translation unit +would still collide on the symbol. diff --git a/examples/10-graphics/offscreen/build.mcpp b/examples/10-graphics/offscreen/build.mcpp new file mode 100644 index 00000000..31461b7f --- /dev/null +++ b/examples/10-graphics/offscreen/build.mcpp @@ -0,0 +1,13 @@ +import std; +import mcpp; +import mcpp.rules.spirv; + +int main() { + mcpp::rerun_if_changed_glob("shaders/**/*.vert"); + mcpp::rerun_if_changed_glob("shaders/**/*.frag"); + mcpp::rerun_if_changed_glob("shaders/**/*.glsl"); + + mcpp::rules::spirv::options opt; + opt.includes = { "shaders" }; + return mcpp::rules::spirv::compile(opt) ? 0 : 1; +} diff --git a/examples/10-graphics/offscreen/include/render/render.h b/examples/10-graphics/offscreen/include/render/render.h new file mode 100644 index 00000000..6a08369f --- /dev/null +++ b/examples/10-graphics/offscreen/include/render/render.h @@ -0,0 +1,29 @@ +#pragma once + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +// Render one frame into `rgba`, which holds `w * h` pixels of four bytes each +// in R, G, B, A order. Returns 0 on success. +// +// The interface is a raw buffer for the reason every device seam in this +// repository uses one: the memory the device writes is not the program's, and +// the module above turns the result back into C++. +int render_offscreen(unsigned w, unsigned h, unsigned char* rgba); + +// The clear colour the implementation uses, so the assertion and the renderer +// cannot disagree about it. Four bytes, R G B A. +void render_clear_color(unsigned char* rgba4); + +// WHICH DEVICE THE LAST SUCCESSFUL `render_offscreen` RAN ON, or "" if none +// has. Both implementations produce the same image, so the image alone does not +// distinguish a device run from the software one -- which is the confusion an +// example about GPU rendering must not teach. +const char* render_device_name(void); + +#ifdef __cplusplus +} +#endif diff --git a/examples/10-graphics/offscreen/mcpp.toml b/examples/10-graphics/offscreen/mcpp.toml new file mode 100644 index 00000000..39d953f5 --- /dev/null +++ b/examples/10-graphics/offscreen/mcpp.toml @@ -0,0 +1,69 @@ +[package] +name = "offscreen-triangle" +namespace = "example" +version = "0.1.0" +description = "A Vulkan graphics pipeline rendering offscreen, with a software rasteriser behind the same seam" +accelerators = ["vulkan"] + +[language] +standard = "c++23" +modules = true +import_std = true + +# One edge. `mcpp.rules.spirv` declares the shader compiler it drives, so this +# project names no payload for it. +[build-dependencies.mcpp] +plugins = { version = "0.2.5", features = ["rules-spirv"], host-module = true } + +# The Khronos loader. Unconditional, and the reason is a rule of the engine +# rather than a preference: A DEPENDENCY CANNOT BE CONDITIONED ON A LAYER. +# `accelerator` is resolved FROM the dependency graph, so a dependency chosen by +# it would decide the answer it is asking. mcpp says so and ignores the +# predicate -- measured here, where an earlier revision of this file gated these +# three entries on `cfg(accelerator = "vulkan")` and the build failed on +# `vulkan/vulkan.h: No such file or directory`, because the header's package had +# been silently dropped while the source that includes it was kept. +# +# `[build]` sources under the same predicate DO apply, which is what makes the +# seam work: the CPU leg below is selected by the accelerator, and only the +# packages are unconditional. +[dependencies.compat] +vulkan = "1.4.357.0" + +# The adapter that makes the host's own ICDs reachable from a binary running +# under mcpp's private loader. An OS predicate, which IS allowed, and a Linux +# concern by construction: macOS resolves through dyld and Windows through the +# PE loader, and neither has a private loader to work around. +[target.'cfg(linux)'.dependencies.compat] +vulkan-runtime = "2026.09.07" + +# A Vulkan device that needs no GPU, so this example runs on a machine that has +# none. It is a DEVICE and therefore a payload; the drivers a real GPU needs are +# the host's, and `compat.vulkan` models those as a capability. Published for +# Linux alone, so the predicate names the platform rather than the accelerator. +[target.'cfg(linux)'.xlings.workspace] +"xim:mesa-lavapipe" = "26.2.1" + +[build] +accel = "vulkan1.2" +sources = [ + "src/*.cppm", + "src/*.cpp", + # Two globs rather than one over `shaders/*`: the stages are different things + # and the manifest is where a reader learns which ones this project has. + { glob = "shaders/*.vert", accel = "vulkan1.2" }, + { glob = "shaders/*.frag", accel = "vulkan1.2" }, +] +include_dirs = ["include"] + +[target.'cfg(accelerator = "vulkan")'.build] +sources = ["src/vulkan/*.cpp"] + +# The same triangle, rasterised in software. The pixel assertions are the +# contract and both legs satisfy it; `device_name()` is what tells them apart. +[target.'cfg(not(accelerator = "vulkan"))'.build] +sources = ["src/cpu/*.cpp"] + +[targets.offscreen-triangle] +kind = "bin" +main = "src/main.cpp" diff --git a/examples/10-graphics/offscreen/shaders/triangle.frag b/examples/10-graphics/offscreen/shaders/triangle.frag new file mode 100644 index 00000000..f81f29ad --- /dev/null +++ b/examples/10-graphics/offscreen/shaders/triangle.frag @@ -0,0 +1,11 @@ +#version 450 + +// The interpolated colour, and nothing else. A fragment shader that ignored its +// input and wrote a constant would produce an image with one non-zero channel +// at the centre; the assertion in `main.cpp` is written to tell those apart. +layout(location = 0) in vec3 v_color; +layout(location = 0) out vec4 o_color; + +void main() { + o_color = vec4(v_color, 1.0); +} diff --git a/examples/10-graphics/offscreen/shaders/triangle.vert b/examples/10-graphics/offscreen/shaders/triangle.vert new file mode 100644 index 00000000..a12d934e --- /dev/null +++ b/examples/10-graphics/offscreen/shaders/triangle.vert @@ -0,0 +1,22 @@ +#version 450 + +// No vertex buffer, no vertex input state: the three positions are indexed by +// `gl_VertexIndex`. That is deliberate, and it is what makes the pixel +// assertion in `main.cpp` a statement about the SHADERS rather than about a +// buffer upload -- there is no vertex data for a bug to hide in. +layout(location = 0) out vec3 v_color; + +vec2 positions[3] = vec2[]( + vec2( 0.0, -0.5), + vec2( 0.5, 0.5), + vec2(-0.5, 0.5)); + +vec3 colors[3] = vec3[]( + vec3(1.0, 0.0, 0.0), + vec3(0.0, 1.0, 0.0), + vec3(0.0, 0.0, 1.0)); + +void main() { + gl_Position = vec4(positions[gl_VertexIndex], 0.0, 1.0); + v_color = colors[gl_VertexIndex]; +} diff --git a/examples/10-graphics/offscreen/src/app.cppm b/examples/10-graphics/offscreen/src/app.cppm new file mode 100644 index 00000000..b4057fb2 --- /dev/null +++ b/examples/10-graphics/offscreen/src/app.cppm @@ -0,0 +1,40 @@ +// The seam. Underneath is a Vulkan graphics pipeline on one build and a +// software rasteriser on another, and no importer of this module can tell. +// +// The same shape `examples/09-heterogeneous/vulkan` uses for compute. What +// differs is only what the device does: there the device answered with numbers, +// here it answers with an image. +module; +#include "render/render.h" +export module app.render; +import std; + +export namespace app { + +struct image { + unsigned w = 0, h = 0; + std::vector rgba; // w * h * 4 + + std::array at(unsigned x, unsigned y) const { + const std::size_t i = (static_cast(y) * w + x) * 4; + return { rgba[i], rgba[i + 1], rgba[i + 2], rgba[i + 3] }; + } +}; + +std::optional render(unsigned w, unsigned h) { + image out{ w, h, std::vector(static_cast(w) * h * 4) }; + if (render_offscreen(w, h, out.rgba.data()) != 0) return std::nullopt; + return out; +} + +std::array clear_color() { + std::array c{}; + render_clear_color(c.data()); + return c; +} + +// Answered by the seam, because the seam is the only place that knows which +// implementation was linked. +std::string_view device_name() { return render_device_name(); } + +} // namespace app diff --git a/examples/10-graphics/offscreen/src/cpu/render.cpp b/examples/10-graphics/offscreen/src/cpu/render.cpp new file mode 100644 index 00000000..6bfc5d5c --- /dev/null +++ b/examples/10-graphics/offscreen/src/cpu/render.cpp @@ -0,0 +1,81 @@ +// The software rasteriser behind the same seam, compiled when the build names +// no accelerator (`mcpp build --no-accel`). +// +// It draws the SAME triangle the shaders draw, which is the point: the pixel +// assertions in `main.cpp` are the contract, and this file exists to show the +// contract is satisfiable without a GPU. What distinguishes the two builds is +// `render_device_name`, never the image. +#include "render/render.h" + +// `` FOR `std::size_t`, AND IT IS NOT PEDANTRY. +// +// A standard header is entitled to bring in whichever others it needs, and +// which ones it brings differs between implementations. This file compiled +// against libstdc++ and then failed against libc++ on the same machine: +// +// src/cpu/render.cpp:56:52: error: no type named 'size_t' in namespace 'std' +// +// A translation unit that names a type has to include the header that declares +// it, whatever the last implementation happened to hand it for free. +#include +#include + +namespace { + +const char* g_ran_on = ""; + +// Clipspace positions and colours, byte for byte what `shaders/triangle.vert` +// declares. Two copies of one fact, and the assertion that keeps them honest is +// that both legs must satisfy the same pixel test. +constexpr float kPos[3][2] = { { 0.0f, -0.5f }, { 0.5f, 0.5f }, { -0.5f, 0.5f } }; +constexpr float kCol[3][3] = { { 1, 0, 0 }, { 0, 1, 0 }, { 0, 0, 1 } }; + +constexpr unsigned char kClear[4] = { 16, 16, 16, 255 }; + +float edge(const float a[2], const float b[2], float px, float py) { + return (px - a[0]) * (b[1] - a[1]) - (py - a[1]) * (b[0] - a[0]); +} + +unsigned char quantise(float v) { + const float c = v < 0.0f ? 0.0f : (v > 1.0f ? 1.0f : v); + return static_cast(c * 255.0f + 0.5f); +} + +} // namespace + +extern "C" void render_clear_color(unsigned char* rgba4) { + for (int i = 0; i < 4; ++i) rgba4[i] = kClear[i]; +} + +extern "C" int render_offscreen(unsigned w, unsigned h, unsigned char* rgba) { + if (w == 0 || h == 0 || rgba == nullptr) return 1; + + const float area = edge(kPos[0], kPos[1], kPos[2][0], kPos[2][1]); + if (area == 0.0f) return 1; + + for (unsigned y = 0; y < h; ++y) { + for (unsigned x = 0; x < w; ++x) { + // Vulkan's framebuffer origin is top-left and its clip space has + // +Y down, so the pixel centre maps straight through. + const float px = 2.0f * (static_cast(x) + 0.5f) / static_cast(w) - 1.0f; + const float py = 2.0f * (static_cast(y) + 0.5f) / static_cast(h) - 1.0f; + + const float w0 = edge(kPos[1], kPos[2], px, py) / area; + const float w1 = edge(kPos[2], kPos[0], px, py) / area; + const float w2 = edge(kPos[0], kPos[1], px, py) / area; + + unsigned char* p = rgba + (static_cast(y) * w + x) * 4; + if (w0 < 0.0f || w1 < 0.0f || w2 < 0.0f) { + for (int i = 0; i < 4; ++i) p[i] = kClear[i]; + continue; + } + for (int c = 0; c < 3; ++c) + p[c] = quantise(w0 * kCol[0][c] + w1 * kCol[1][c] + w2 * kCol[2][c]); + p[3] = 255; + } + } + g_ran_on = "cpu rasteriser (this build names no accelerator)"; + return 0; +} + +extern "C" const char* render_device_name(void) { return g_ran_on; } diff --git a/examples/10-graphics/offscreen/src/main.cpp b/examples/10-graphics/offscreen/src/main.cpp new file mode 100644 index 00000000..c4ad7c0d --- /dev/null +++ b/examples/10-graphics/offscreen/src/main.cpp @@ -0,0 +1,62 @@ +import std; +import app.render; + +namespace { + +constexpr unsigned kW = 64, kH = 64; + +bool same(std::array a, std::array b) { + return a == b; +} + +std::string show(std::array c) { + return std::format("({}, {}, {}, {})", c[0], c[1], c[2], c[3]); +} + +} // namespace + +int main() { + auto img = app::render(kW, kH); + if (!img) { std::println("render unavailable"); return 1; } + const auto clear = app::clear_color(); + + // The four corners are outside the triangle whichever way it is rasterised, + // so they are the clear colour exactly. No tolerance: nothing interpolates + // there, and a tolerance would hide an image that was never rendered into. + const std::array, 4> corners{{ + {0, 0}, {kW - 1, 0}, {0, kH - 1}, {kW - 1, kH - 1}}}; + for (auto [x, y] : corners) { + const auto got = img->at(x, y); + if (!same(got, clear)) { + std::println("corner ({}, {}) is {}, expected the clear colour {}", + x, y, show(got), show(clear)); + return 1; + } + } + + // THE CENTRE IS THE ASSERTION THIS EXAMPLE EXISTS FOR. + // + // It lies inside the triangle, where all three vertex colours contribute. + // A fragment shader that ignored its input and wrote a constant would put + // 255 in one channel and 0 in the other two; a pipeline whose vertex stage + // never ran would leave the clear colour. Requiring all three channels to + // be non-zero separates those from an interpolated result, and it does so + // without depending on a rasteriser's exact rounding. + const auto centre = img->at(kW / 2, kH / 2); + std::println("centre pixel: {}", show(centre)); + if (same(centre, clear)) { + std::println("the centre is the clear colour: nothing was drawn"); + return 1; + } + if (centre[0] == 0 || centre[1] == 0 || centre[2] == 0) { + std::println("the centre has a zero channel: the vertex colours were not interpolated"); + return 1; + } + if (centre[3] != 255) { + std::println("the centre is not opaque: alpha is {}", centre[3]); + return 1; + } + + std::println("device: {}", app::device_name()); + return 0; +} diff --git a/examples/10-graphics/offscreen/src/vulkan/render.cpp b/examples/10-graphics/offscreen/src/vulkan/render.cpp new file mode 100644 index 00000000..e44ce018 --- /dev/null +++ b/examples/10-graphics/offscreen/src/vulkan/render.cpp @@ -0,0 +1,347 @@ +// The Vulkan graphics island: a full pipeline -- vertex input, rasterisation, +// fragment output, render pass -- rendering into an image this program then +// reads back. +// +// OFFSCREEN, AND THAT IS THE DESIGN RATHER THAN A LIMITATION. +// +// A swapchain needs a surface, and a surface needs a window system. On a +// headless runner there is none, so an example built around one can only be +// BUILT in CI -- and "it built" is close to no information about a graphics +// pipeline: a fragment shader that ignores its input, a pipeline with the wrong +// vertex stage, an image never rendered into, all compile and link. Rendering +// into an image and reading the pixels back makes the result assertable, and it +// exercises the same pipeline a windowed application uses. +// +// Nothing here was compiled by the C++ toolchain. `triangle_vert.h` and +// `triangle_frag.h` are `const uint32_t[]` that `mcpp.rules.spirv` produced +// from `shaders/triangle.vert` and `shaders/triangle.frag`. +#include "render/render.h" + +#include + +#include +#include +#include +#include + +#include "triangle_vert.h" +#include "triangle_frag.h" + +namespace { + +const char* g_ran_on = ""; + +constexpr unsigned char kClear[4] = { 16, 16, 16, 255 }; + +int find_memory_type(VkPhysicalDevice phys, std::uint32_t bits, + VkMemoryPropertyFlags want) { + VkPhysicalDeviceMemoryProperties props{}; + vkGetPhysicalDeviceMemoryProperties(phys, &props); + for (std::uint32_t i = 0; i < props.memoryTypeCount; ++i) + if ((bits & (1u << i)) && (props.memoryTypes[i].propertyFlags & want) == want) + return static_cast(i); + return -1; +} + +// The first device with a GRAPHICS queue. Not "the fastest": what this example +// demonstrates is that it runs wherever a driver exists, and a machine whose +// only driver is lavapipe is the case it is written for. +bool pick_device(VkInstance inst, VkPhysicalDevice& out, std::uint32_t& family) { + std::uint32_t n = 0; + vkEnumeratePhysicalDevices(inst, &n, nullptr); + if (n == 0) return false; + std::vector devices(n); + vkEnumeratePhysicalDevices(inst, &n, devices.data()); + for (auto d : devices) { + std::uint32_t qn = 0; + vkGetPhysicalDeviceQueueFamilyProperties(d, &qn, nullptr); + std::vector qs(qn); + vkGetPhysicalDeviceQueueFamilyProperties(d, &qn, qs.data()); + for (std::uint32_t i = 0; i < qn; ++i) + if (qs[i].queueFlags & VK_QUEUE_GRAPHICS_BIT) { + out = d; family = i; return true; + } + } + return false; +} + +VkShaderModule make_module(VkDevice dev, const std::uint32_t* code, std::size_t bytes) { + VkShaderModuleCreateInfo ci{ VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO }; + ci.codeSize = bytes; + ci.pCode = code; + VkShaderModule m = VK_NULL_HANDLE; + return vkCreateShaderModule(dev, &ci, nullptr, &m) == VK_SUCCESS ? m : VK_NULL_HANDLE; +} + +// Everything this frame owns, released in reverse. A single scope with one exit +// path: an example that leaked on its error paths would be teaching the wrong +// thing about a C API behind a C++ seam. +struct frame { + VkInstance instance = VK_NULL_HANDLE; + VkDevice device = VK_NULL_HANDLE; + VkImage image = VK_NULL_HANDLE; + VkDeviceMemory imageMem = VK_NULL_HANDLE; + VkImageView view = VK_NULL_HANDLE; + VkRenderPass pass = VK_NULL_HANDLE; + VkFramebuffer fb = VK_NULL_HANDLE; + VkShaderModule vs = VK_NULL_HANDLE; + VkShaderModule fs = VK_NULL_HANDLE; + VkPipelineLayout layout = VK_NULL_HANDLE; + VkPipeline pipeline = VK_NULL_HANDLE; + VkBuffer readback = VK_NULL_HANDLE; + VkDeviceMemory readbackMem= VK_NULL_HANDLE; + VkCommandPool pool = VK_NULL_HANDLE; + VkFence fence = VK_NULL_HANDLE; + + ~frame() { + if (device != VK_NULL_HANDLE) { + if (fence) vkDestroyFence(device, fence, nullptr); + if (pool) vkDestroyCommandPool(device, pool, nullptr); + if (readback) vkDestroyBuffer(device, readback, nullptr); + if (readbackMem) vkFreeMemory(device, readbackMem, nullptr); + if (pipeline) vkDestroyPipeline(device, pipeline, nullptr); + if (layout) vkDestroyPipelineLayout(device, layout, nullptr); + if (fs) vkDestroyShaderModule(device, fs, nullptr); + if (vs) vkDestroyShaderModule(device, vs, nullptr); + if (fb) vkDestroyFramebuffer(device, fb, nullptr); + if (pass) vkDestroyRenderPass(device, pass, nullptr); + if (view) vkDestroyImageView(device, view, nullptr); + if (image) vkDestroyImage(device, image, nullptr); + if (imageMem) vkFreeMemory(device, imageMem, nullptr); + vkDestroyDevice(device, nullptr); + } + if (instance != VK_NULL_HANDLE) vkDestroyInstance(instance, nullptr); + } +}; + +} // namespace + +extern "C" void render_clear_color(unsigned char* rgba4) { + for (int i = 0; i < 4; ++i) rgba4[i] = kClear[i]; +} + +extern "C" int render_offscreen(unsigned w, unsigned h, unsigned char* rgba) { + if (w == 0 || h == 0 || rgba == nullptr) return 1; + frame f; + + VkApplicationInfo app{ VK_STRUCTURE_TYPE_APPLICATION_INFO }; + app.pApplicationName = "mcpp-offscreen"; + app.apiVersion = VK_API_VERSION_1_2; + VkInstanceCreateInfo ici{ VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO }; + ici.pApplicationInfo = &app; + if (vkCreateInstance(&ici, nullptr, &f.instance) != VK_SUCCESS) return 1; + + VkPhysicalDevice phys = VK_NULL_HANDLE; + std::uint32_t family = 0; + if (!pick_device(f.instance, phys, family)) return 1; + + const float prio = 1.0f; + VkDeviceQueueCreateInfo qci{ VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO }; + qci.queueFamilyIndex = family; + qci.queueCount = 1; + qci.pQueuePriorities = &prio; + VkDeviceCreateInfo dci{ VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO }; + dci.queueCreateInfoCount = 1; + dci.pQueueCreateInfos = &qci; + if (vkCreateDevice(phys, &dci, nullptr, &f.device) != VK_SUCCESS) return 1; + + VkQueue queue = VK_NULL_HANDLE; + vkGetDeviceQueue(f.device, family, 0, &queue); + + // ── the colour attachment, and the buffer it is copied into ────────────── + VkImageCreateInfo img{ VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO }; + img.imageType = VK_IMAGE_TYPE_2D; + img.format = VK_FORMAT_R8G8B8A8_UNORM; + img.extent = { w, h, 1 }; + img.mipLevels = 1; + img.arrayLayers = 1; + img.samples = VK_SAMPLE_COUNT_1_BIT; + img.tiling = VK_IMAGE_TILING_OPTIMAL; + img.usage = VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT; + img.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED; + if (vkCreateImage(f.device, &img, nullptr, &f.image) != VK_SUCCESS) return 1; + + VkMemoryRequirements req{}; + vkGetImageMemoryRequirements(f.device, f.image, &req); + int type = find_memory_type(phys, req.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT); + if (type < 0) type = find_memory_type(phys, req.memoryTypeBits, 0); + if (type < 0) return 1; + VkMemoryAllocateInfo mai{ VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO }; + mai.allocationSize = req.size; + mai.memoryTypeIndex = static_cast(type); + if (vkAllocateMemory(f.device, &mai, nullptr, &f.imageMem) != VK_SUCCESS) return 1; + if (vkBindImageMemory(f.device, f.image, f.imageMem, 0) != VK_SUCCESS) return 1; + + VkImageViewCreateInfo vci{ VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO }; + vci.image = f.image; + vci.viewType = VK_IMAGE_VIEW_TYPE_2D; + vci.format = VK_FORMAT_R8G8B8A8_UNORM; + vci.subresourceRange = { VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1 }; + if (vkCreateImageView(f.device, &vci, nullptr, &f.view) != VK_SUCCESS) return 1; + + // ── the render pass: clear, draw, leave it ready to be copied ──────────── + VkAttachmentDescription att{}; + att.format = VK_FORMAT_R8G8B8A8_UNORM; + att.samples = VK_SAMPLE_COUNT_1_BIT; + att.loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR; + att.storeOp = VK_ATTACHMENT_STORE_OP_STORE; + att.stencilLoadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE; + att.stencilStoreOp = VK_ATTACHMENT_STORE_OP_DONT_CARE; + att.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED; + // The final layout is what removes an explicit barrier before the copy. + att.finalLayout = VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL; + VkAttachmentReference ref{ 0, VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL }; + VkSubpassDescription sub{}; + sub.pipelineBindPoint = VK_PIPELINE_BIND_POINT_GRAPHICS; + sub.colorAttachmentCount = 1; + sub.pColorAttachments = &ref; + VkRenderPassCreateInfo rp{ VK_STRUCTURE_TYPE_RENDER_PASS_CREATE_INFO }; + rp.attachmentCount = 1; + rp.pAttachments = &att; + rp.subpassCount = 1; + rp.pSubpasses = ⊂ + if (vkCreateRenderPass(f.device, &rp, nullptr, &f.pass) != VK_SUCCESS) return 1; + + VkFramebufferCreateInfo fbi{ VK_STRUCTURE_TYPE_FRAMEBUFFER_CREATE_INFO }; + fbi.renderPass = f.pass; + fbi.attachmentCount = 1; + fbi.pAttachments = &f.view; + fbi.width = w; fbi.height = h; fbi.layers = 1; + if (vkCreateFramebuffer(f.device, &fbi, nullptr, &f.fb) != VK_SUCCESS) return 1; + + // ── the pipeline ──────────────────────────────────────────────────────── + f.vs = make_module(f.device, triangle_vert_spv, sizeof triangle_vert_spv); + f.fs = make_module(f.device, triangle_frag_spv, sizeof triangle_frag_spv); + if (!f.vs || !f.fs) return 1; + + VkPipelineShaderStageCreateInfo stages[2]{}; + stages[0].sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO; + stages[0].stage = VK_SHADER_STAGE_VERTEX_BIT; + stages[0].module = f.vs; + stages[0].pName = "main"; + stages[1].sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO; + stages[1].stage = VK_SHADER_STAGE_FRAGMENT_BIT; + stages[1].module = f.fs; + stages[1].pName = "main"; + + // EMPTY vertex input: the vertex shader indexes its own arrays. + VkPipelineVertexInputStateCreateInfo vin{ VK_STRUCTURE_TYPE_PIPELINE_VERTEX_INPUT_STATE_CREATE_INFO }; + VkPipelineInputAssemblyStateCreateInfo ia{ VK_STRUCTURE_TYPE_PIPELINE_INPUT_ASSEMBLY_STATE_CREATE_INFO }; + ia.topology = VK_PRIMITIVE_TOPOLOGY_TRIANGLE_LIST; + + VkViewport vp{ 0.0f, 0.0f, static_cast(w), static_cast(h), 0.0f, 1.0f }; + VkRect2D sc{ { 0, 0 }, { w, h } }; + VkPipelineViewportStateCreateInfo vps{ VK_STRUCTURE_TYPE_PIPELINE_VIEWPORT_STATE_CREATE_INFO }; + vps.viewportCount = 1; vps.pViewports = &vp; + vps.scissorCount = 1; vps.pScissors = ≻ + + VkPipelineRasterizationStateCreateInfo rs{ VK_STRUCTURE_TYPE_PIPELINE_RASTERIZATION_STATE_CREATE_INFO }; + rs.polygonMode = VK_POLYGON_MODE_FILL; + // NO CULLING, so the winding of the three vertices is not a second fact the + // software rasteriser has to agree about. + rs.cullMode = VK_CULL_MODE_NONE; + rs.frontFace = VK_FRONT_FACE_COUNTER_CLOCKWISE; + rs.lineWidth = 1.0f; + + VkPipelineMultisampleStateCreateInfo ms{ VK_STRUCTURE_TYPE_PIPELINE_MULTISAMPLE_STATE_CREATE_INFO }; + ms.rasterizationSamples = VK_SAMPLE_COUNT_1_BIT; + + VkPipelineColorBlendAttachmentState cba{}; + cba.colorWriteMask = VK_COLOR_COMPONENT_R_BIT | VK_COLOR_COMPONENT_G_BIT + | VK_COLOR_COMPONENT_B_BIT | VK_COLOR_COMPONENT_A_BIT; + VkPipelineColorBlendStateCreateInfo cb{ VK_STRUCTURE_TYPE_PIPELINE_COLOR_BLEND_STATE_CREATE_INFO }; + cb.attachmentCount = 1; + cb.pAttachments = &cba; + + VkPipelineLayoutCreateInfo pli{ VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO }; + if (vkCreatePipelineLayout(f.device, &pli, nullptr, &f.layout) != VK_SUCCESS) return 1; + + VkGraphicsPipelineCreateInfo gp{ VK_STRUCTURE_TYPE_GRAPHICS_PIPELINE_CREATE_INFO }; + gp.stageCount = 2; + gp.pStages = stages; + gp.pVertexInputState = &vin; + gp.pInputAssemblyState = &ia; + gp.pViewportState = &vps; + gp.pRasterizationState = &rs; + gp.pMultisampleState = &ms; + gp.pColorBlendState = &cb; + gp.layout = f.layout; + gp.renderPass = f.pass; + if (vkCreateGraphicsPipelines(f.device, VK_NULL_HANDLE, 1, &gp, nullptr, &f.pipeline) + != VK_SUCCESS) return 1; + + // ── readback buffer ───────────────────────────────────────────────────── + const VkDeviceSize bytes = static_cast(w) * h * 4; + VkBufferCreateInfo bci{ VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO }; + bci.size = bytes; + bci.usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT; + if (vkCreateBuffer(f.device, &bci, nullptr, &f.readback) != VK_SUCCESS) return 1; + VkMemoryRequirements breq{}; + vkGetBufferMemoryRequirements(f.device, f.readback, &breq); + const int btype = find_memory_type(phys, breq.memoryTypeBits, + VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT); + if (btype < 0) return 1; + VkMemoryAllocateInfo bmai{ VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO }; + bmai.allocationSize = breq.size; + bmai.memoryTypeIndex = static_cast(btype); + if (vkAllocateMemory(f.device, &bmai, nullptr, &f.readbackMem) != VK_SUCCESS) return 1; + if (vkBindBufferMemory(f.device, f.readback, f.readbackMem, 0) != VK_SUCCESS) return 1; + + // ── record and submit ─────────────────────────────────────────────────── + VkCommandPoolCreateInfo cpi{ VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO }; + cpi.queueFamilyIndex = family; + if (vkCreateCommandPool(f.device, &cpi, nullptr, &f.pool) != VK_SUCCESS) return 1; + VkCommandBufferAllocateInfo cbi{ VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO }; + cbi.commandPool = f.pool; + cbi.level = VK_COMMAND_BUFFER_LEVEL_PRIMARY; + cbi.commandBufferCount = 1; + VkCommandBuffer cmd = VK_NULL_HANDLE; + if (vkAllocateCommandBuffers(f.device, &cbi, &cmd) != VK_SUCCESS) return 1; + + VkCommandBufferBeginInfo begin{ VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO }; + begin.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT; + if (vkBeginCommandBuffer(cmd, &begin) != VK_SUCCESS) return 1; + + VkClearValue clear{}; + for (int i = 0; i < 4; ++i) + clear.color.float32[i] = static_cast(kClear[i]) / 255.0f; + VkRenderPassBeginInfo rpb{ VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO }; + rpb.renderPass = f.pass; + rpb.framebuffer = f.fb; + rpb.renderArea = sc; + rpb.clearValueCount = 1; + rpb.pClearValues = &clear; + vkCmdBeginRenderPass(cmd, &rpb, VK_SUBPASS_CONTENTS_INLINE); + vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_GRAPHICS, f.pipeline); + vkCmdDraw(cmd, 3, 1, 0, 0); + vkCmdEndRenderPass(cmd); + + VkBufferImageCopy region{}; + region.imageSubresource = { VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1 }; + region.imageExtent = { w, h, 1 }; + vkCmdCopyImageToBuffer(cmd, f.image, VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, + f.readback, 1, ®ion); + if (vkEndCommandBuffer(cmd) != VK_SUCCESS) return 1; + + VkFenceCreateInfo fci{ VK_STRUCTURE_TYPE_FENCE_CREATE_INFO }; + if (vkCreateFence(f.device, &fci, nullptr, &f.fence) != VK_SUCCESS) return 1; + VkSubmitInfo si{ VK_STRUCTURE_TYPE_SUBMIT_INFO }; + si.commandBufferCount = 1; + si.pCommandBuffers = &cmd; + if (vkQueueSubmit(queue, 1, &si, f.fence) != VK_SUCCESS) return 1; + if (vkWaitForFences(f.device, 1, &f.fence, VK_TRUE, ~0ull) != VK_SUCCESS) return 1; + + void* mapped = nullptr; + if (vkMapMemory(f.device, f.readbackMem, 0, bytes, 0, &mapped) != VK_SUCCESS) return 1; + std::memcpy(rgba, mapped, static_cast(bytes)); + vkUnmapMemory(f.device, f.readbackMem); + + VkPhysicalDeviceProperties props{}; + vkGetPhysicalDeviceProperties(phys, &props); + static char name[VK_MAX_PHYSICAL_DEVICE_NAME_SIZE]; + std::memcpy(name, props.deviceName, sizeof name); + g_ran_on = name; + return 0; +} + +extern "C" const char* render_device_name(void) { return g_ran_on; } diff --git a/mcpp.toml b/mcpp.toml index 6655a311..65752d67 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.9.6.6" +version = "2026.9.7.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/platform/src/shell.cppm b/modules/platform/src/shell.cppm index e37bc9e9..5da75324 100644 --- a/modules/platform/src/shell.cppm +++ b/modules/platform/src/shell.cppm @@ -27,6 +27,46 @@ std::string quote(std::string_view s); std::string quote_windows(std::string_view s); std::string quote_posix(std::string_view s); +// An argument that must survive TWO parsers: cmd.exe's, and then the child +// program's own argv parsing. +// +// `quote_windows` answers only the second. Its `\"` escape belongs to the +// MSVCRT argv rules, and cmd.exe does not know it: to cmd every `"` simply +// toggles a quote state, so a payload carrying an EVEN number of them before a +// metacharacter leaves that character unquoted. Measured on windows-2022 with +// a JSON argument -- `{"targets":["xim:shaderc@>=2026.3"],"yes":true}` puts +// four quotes before the `>`, and cmd read it as a REDIRECTION, answering +// +// The filename, directory name, or volume label syntax is incorrect. +// +// which arrived as a package-provisioning failure naming a package. +// +// The answer escapes exactly what needs it: quote for the child, then walk the +// result tracking the quote state CMD sees -- every `"` toggles it, because cmd +// does not know MSVCRT's `\"` -- and prefix `^` to any metacharacter that falls +// outside a quoted region. Inside one it is already inert, and `^` there is a +// literal character rather than an escape. +// +// ESCAPING EVERY METACHARACTER INCLUDING THE QUOTES WAS TRIED FIRST AND +// BROKE EVERY PAYLOAD. It is defensible on paper -- with no bare `"`, cmd never +// enters a quoted region and each metacharacter is escaped rather than quoted +// -- and Windows CI answered `exit 1` for every package fetch, including the +// ones whose JSON contains no metacharacter at all. So the rule here is the +// conservative one: a payload with nothing to escape comes out byte-identical +// to `quote_windows`, and only the character that is actually unprotected +// acquires a caret. +// +// `%` IS NOT ESCAPED AND CANNOT BE. Variable expansion happens before caret +// processing, and the batch-file escape (`%%`) is not available on a command +// line. Callers passing text that may contain `%` need a different mechanism; +// the JSON arguments this exists for do not. +std::string quote_windows_through_cmd(std::string_view s); + +// Host-selecting: `quote_windows_through_cmd` on Windows, `quote_posix` +// elsewhere. Use this wherever an argument reaches a shell and may contain +// metacharacters -- notably JSON, and any version constraint spelled `>=`. +std::string quote_through_shell(std::string_view s); + // Silent redirect — stdout + stderr → /dev/null (or NUL on Windows). // stdin is NOT touched here; that's the responsibility of // mcpp::platform::process::seal_stdin, which is auto-applied by capture / @@ -67,6 +107,33 @@ std::string quote_posix(std::string_view s) { return out; } +std::string quote_windows_through_cmd(std::string_view s) { + const std::string inner = quote_windows(s); + std::string out; + out.reserve(inner.size() + 8); + // cmd's quote state, which is toggled by EVERY `"` -- it does not know + // MSVCRT's `\"`. Inside a quoted region a metacharacter is already inert + // and `^` is a literal character, so only the characters that fall OUTSIDE + // one are escaped. That keeps the result byte-identical to `quote_windows` + // for every payload with nothing to escape, which is nearly all of them. + bool inQuotes = false; + for (char c : inner) { + if (c == '"') { inQuotes = !inQuotes; out.push_back(c); continue; } + if (!inQuotes) { + switch (c) { + case '<': case '>': case '&': case '|': + case '^': case '(': case ')': + out.push_back('^'); + break; + default: + break; + } + } + out.push_back(c); + } + return out; +} + std::string quote(std::string_view s) { #if defined(_WIN32) return quote_windows(s); @@ -75,4 +142,12 @@ std::string quote(std::string_view s) { #endif } +std::string quote_through_shell(std::string_view s) { +#if defined(_WIN32) + return quote_windows_through_cmd(s); +#else + return quote_posix(s); +#endif +} + } // namespace mcpp::platform::shell diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index f41fd6e1..692e8bbe 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.9.6.6"; +inline constexpr std::string_view MCPP_VERSION = "2026.9.7.1"; } // namespace mcpp diff --git a/src/toolchain/hostflags.cppm b/src/toolchain/hostflags.cppm index f01e8730..5ab3fcb0 100644 --- a/src/toolchain/hostflags.cppm +++ b/src/toolchain/hostflags.cppm @@ -49,7 +49,14 @@ struct HostFlagOptions { // needs_explicit_libcxx path owns; duplicating that for a // host compile produced undefined __cxa_* / // __gxx_personality_v0 (build_program.cppm, pre-existing). - enum class CfgBypass { Always, LinuxOnly }; + // Never — always trust the cfg. No caller selects it; it completes + // the enum, and it is what makes the cfg-trusting branch of + // `host_link_tokens` reachable from a test on a Linux + // runner. `LinuxOnly` folds into `Always` there, so without + // this value that branch could only be exercised on the two + // hosts it was written for -- which is how it came to be + // missing the runtime-directory tokens in the first place. + enum class CfgBypass { Always, LinuxOnly, Never }; CfgBypass cfgBypass = CfgBypass::Always; // binutils `-B` so the driver finds as/ld. A GCC/libstdc++ payload @@ -206,7 +213,8 @@ std::vector host_compile_tokens(const Toolchain& tc, const bool bypassCfg = dm.hasCfg && (opt.cfgBypass == HostFlagOptions::CfgBypass::Always - || mcpp::platform::is_linux); + || (opt.cfgBypass == HostFlagOptions::CfgBypass::LinuxOnly + && mcpp::platform::is_linux)); // Trusting the cfg means contributing no include paths, stdlib selection // or runtime choices — it already carries them. It does NOT mean @@ -297,6 +305,20 @@ std::vector bmi_reference_tokens(std::string_view usePrefix, std::string(p.substr(sp + 1)) + bmi.string() }; } +// The toolchain's own runtime directories, on both exits of the function +// below. `-L` is link-time and wanted everywhere; rpath is an ELF and Mach-O +// concept. A PE target reaches here too, where the rpath flag is inert and +// self-containment comes from the static link instead (#299). +void append_runtime_lib_dirs(const Toolchain& tc, const HostFlagOptions& opt, + const PathEscape& esc, std::vector& out) { + if (!opt.runtimeLibDirs) return; + for (auto& d : tc.linkRuntimeDirs) { + out.push_back("-L" + esc(d)); + if constexpr (mcpp::platform::supports_rpath) + out.push_back("-Wl,-rpath," + esc(d)); + } +} + std::vector host_link_tokens(const Toolchain& tc, const HostFlagOptions& opt, const PathEscape& esc) { @@ -308,7 +330,8 @@ std::vector host_link_tokens(const Toolchain& tc, const bool bypassCfg = dm.hasCfg && (opt.cfgBypass == HostFlagOptions::CfgBypass::Always - || mcpp::platform::is_linux); + || (opt.cfgBypass == HostFlagOptions::CfgBypass::LinuxOnly + && mcpp::platform::is_linux)); if (bypassCfg) { for (auto& t : dm.link_tokens(esc)) out.push_back(t); @@ -336,6 +359,42 @@ std::vector host_link_tokens(const Toolchain& tc, // lld ships with the very toolchain doing the compile, so it cannot // be diverted to a libc++ it was not built against. if constexpr (mcpp::platform::is_macos) out.push_back("-fuse-ld=lld"); + // AND DELIBERATELY NOT THE TOOLCHAIN'S RUNTIME DIRECTORIES. THIS WAS + // TRIED, AND WHAT IT COSTS IS RECORDED HERE RATHER THAN REDISCOVERED. + // + // The motivation is real. Trusting the cfg decides WHICH runtimes are + // linked and never decides WHERE they are found, so on macOS `-lc++` + // resolves through the SDK to /usr/lib/libc++.tbd -- the system copy, + // whose version floats with the host OS -- while the headers come from + // the payload. On macOS 14 those two disagree, measured on a build + // program that does nothing but `import std`: + // + // ld64.lld: error: undefined symbol: + // std::__1::__is_posix_terminal(__sFILE*) + // >>> referenced by std::__1::__print::__is_terminal(__sFILE*) + // + // `std::print` is not header-only; that support symbol arrived in a + // libc++ macOS 14 does not ship, and macOS 15's copy has it, which is + // why every macOS runner this project uses was green. + // + // ADDING `-L/lib` HERE FIXES THAT AND BUYS A WORSE PROBLEM. + // It makes `-lc++` resolve to the toolchain's own dylib, which is the + // ToolchainCoupled contract that `dist::mechanism_for` REFUSES on + // Mach-O for a measured reason: LLVM's macOS libc++abi and libunwind + // dylibs upward-link /usr/lib/libc++, so the system libc++ loads + // alongside the toolchain's and an object freed across the two aborts + // in libmalloc (#202). The first step of that path is what CI reported + // when this was tried -- the link stopped on `__cxa_end_catch`, + // `std::runtime_error::~runtime_error()` and the rest of the ABI + // surface the system libc++ re-exports and the payload's does not. + // + // So the C++ runtime on this host is the system one, and the + // deployment floor is made real by the STATIC libc++ the distribution + // contract selects -- a mechanism that belongs to an artifact and not + // to a helper mcpp compiles, runs here, and throws away. The + // consequence, stated because it is a real limit: on macOS 14 a build + // program cannot use `std::print` or `std::println`. `std::format` is + // header-only and has none of this. return out; } @@ -346,16 +405,7 @@ std::vector host_link_tokens(const Toolchain& tc, out.push_back("-B" + esc(ar.parent_path())); } - if (opt.runtimeLibDirs) { - // -L is link-time and wanted everywhere; rpath is an ELF-only concept. - // A PE target reaches here too, where the flag is inert and - // self-containment comes from the static link instead (#299). - for (auto& d : tc.linkRuntimeDirs) { - out.push_back("-L" + esc(d)); - if constexpr (mcpp::platform::supports_rpath) - out.push_back("-Wl,-rpath," + esc(d)); - } - } + append_runtime_lib_dirs(tc, opt, esc, out); return out; } diff --git a/src/xlings/xlings.cppm b/src/xlings/xlings.cppm index aa7dfc97..37526099 100644 --- a/src/xlings/xlings.cppm +++ b/src/xlings/xlings.cppm @@ -259,6 +259,10 @@ namespace paths { // Shell-escape (single-quote) a string for the command line. std::string shq(std::string_view s); +// `shq` for an argument that may carry a shell metacharacter (JSON, `>=`). +// See `shell::quote_windows_through_cmd`. +std::string shq_meta(std::string_view s); + // ─── Shell command builders ───────────────────────────────────────── // Build the standard xlings command prefix with proper env vars. @@ -772,6 +776,13 @@ std::string shq(std::string_view s) { return mcpp::platform::shell::quote(s); } +// The same, for an argument that may carry a shell metacharacter -- which the +// JSON payloads below do, and which a version constraint spelled `>=` does. +// See `quote_windows_through_cmd` for what `shq` alone leaves unprotected. +std::string shq_meta(std::string_view s) { + return mcpp::platform::shell::quote_through_shell(s); +} + // ─── Path helpers ─────────────────────────────────────────────────── namespace paths { @@ -1157,7 +1168,7 @@ std::string build_interface_command(const Env& env, std::string_view capability, std::string_view argsJson) { return std::format("{} interface {} --args {} {}", - build_command_prefix(env), capability, shq(argsJson), + build_command_prefix(env), capability, shq_meta(argsJson), mcpp::platform::null_redirect); } @@ -1417,7 +1428,7 @@ int install_with_progress(const Env& env, std::string_view target, if constexpr (mcpp::platform::is_windows) { return std::format("{} interface install_packages --args {} {} .cfg`. +struct FakeClangPayload { + std::filesystem::path root; + explicit FakeClangPayload(std::string_view name) { + root = std::filesystem::temp_directory_path() + / ("mcpp-hostflags-" + std::string(name)); + std::filesystem::remove_all(root); + std::filesystem::create_directories(root / "bin"); + std::filesystem::create_directories(root / "lib"); + std::ofstream{root / "bin" / "clang++"} << ""; + std::ofstream{root / "bin" / "clang++.cfg"} << ""; + } + ~FakeClangPayload() { std::error_code ec; std::filesystem::remove_all(root, ec); } +}; + +bool names_dir(const std::vector& tokens, std::string_view dir) { + return std::ranges::any_of(tokens, [&](auto const& t) { + return t == std::string("-L") + std::string(dir); + }); +} + +} // namespace + +TEST(HostFlags, OnlyTheSpelledOutExitNamesTheToolchainRuntimeDirs) { + FakeClangPayload payload{"runtime-dirs"}; + auto tc = tc_for(CompilerId::Clang); + tc.binaryPath = payload.root / "bin" / "clang++"; + tc.linkRuntimeDirs = { payload.root / "lib" }; + + HostFlagOptions opt; + opt.runtimeLibDirs = true; + + // Spelled out: the payload's own runtime directories are named. + opt.cfgBypass = HostFlagOptions::CfgBypass::Always; + auto spelled = mcpp::toolchain::host_link_tokens(tc, opt, mcpp::toolchain::no_escape); + EXPECT_TRUE(names_dir(spelled, (payload.root / "lib").string())); + + // Trusting the cfg: they are NOT, and the comment above says what naming + // them costs. + opt.cfgBypass = HostFlagOptions::CfgBypass::Never; + auto trusting = mcpp::toolchain::host_link_tokens(tc, opt, mcpp::toolchain::no_escape); + EXPECT_FALSE(names_dir(trusting, (payload.root / "lib").string())) + << "the cfg-trusting exit now points `-lc++` at the toolchain's own " + "dylib; on Mach-O that link fails on the C++ ABI symbols the system " + "libc++ re-exports"; + + // And the option is still an option on the exit that honours it. + opt.runtimeLibDirs = false; + opt.cfgBypass = HostFlagOptions::CfgBypass::Always; + auto off = mcpp::toolchain::host_link_tokens(tc, opt, mcpp::toolchain::no_escape); + EXPECT_FALSE(names_dir(off, (payload.root / "lib").string())); +} + TEST(HostFlags, DeploymentTargetOnlyOnMacos) { auto tc = tc_for(CompilerId::GCC); HostFlagOptions opt; diff --git a/tests/unit/test_windows_command_line.cpp b/tests/unit/test_windows_command_line.cpp index a9c3656f..f1acde0c 100644 --- a/tests/unit/test_windows_command_line.cpp +++ b/tests/unit/test_windows_command_line.cpp @@ -118,3 +118,152 @@ TEST(WindowsCommandLine, PosixQuotingIsUnaffected) { "'/home/my dir'"); EXPECT_EQ(mcpp::platform::shell::quote_posix("it's"), "'it'\\''s'"); } + + +// ── An argument that must survive cmd.exe AND the child's argv parser ─────── +// +// THE DEFECT. mcpp hands xlings its provisioning request as a JSON argument on +// a shell command line. `shell::quote` answers the CHILD's parser -- MSVCRT, +// whose escape for an embedded quote is `\"` -- and cmd.exe does not know that +// escape: to cmd every `"` toggles a quote state. A JSON payload therefore +// arrives at a `>` with an EVEN number of quotes behind it, cmd reads the `>` +// as a redirection, and the redirection target is the rest of the JSON: +// +// Provisioning [xlings.workspace] entries declared by dependencies +// (xim:shaderc@>=2026.3) +// The filename, directory name, or volume label syntax is incorrect. +// +// Measured on windows-2022. The `>=` shape is what every rule package uses to +// state a floor, and no declaration reachable on Windows had ever carried one, +// so the whole shape was unexercised on that host. +// +// The two simulators below are the criterion. Neither asserts on the SPELLING +// of the escape -- they replay what each parser does and compare the argument +// the child would receive against the one that was meant. +namespace { + +// cmd.exe, from `/d /s /c ""` to what CreateProcess receives. +// Returns the command line, and reports whether any redirection or piping +// metacharacter survived unquoted -- which is what actually broke. +struct CmdParse { + std::string passedOn; + bool sawActiveMetacharacter = false; +}; + +CmdParse simulate_cmd_c(std::string_view wrapped) { + // `/s`: strip the first character and the last quote character. + std::string line{wrapped}; + if (!line.empty() && line.front() == '"') line.erase(0, 1); + if (auto last = line.rfind('"'); last != std::string::npos) line.erase(last, 1); + + CmdParse out; + bool inQuotes = false; + for (std::size_t i = 0; i < line.size(); ++i) { + char c = line[i]; + // `^` escapes the next character OUTSIDE a quoted region; inside one + // it is an ordinary character. Modelling only the first half would let + // this simulator accept a shape cmd does not. + if (c == '^' && !inQuotes) { + if (i + 1 < line.size()) out.passedOn.push_back(line[++i]); + continue; + } + if (c == '"') { inQuotes = !inQuotes; out.passedOn.push_back(c); continue; } + if (!inQuotes && (c == '<' || c == '>' || c == '&' || c == '|')) + out.sawActiveMetacharacter = true; + out.passedOn.push_back(c); + } + return out; +} + +// The MSVCRT argv rules, over the command line cmd passed on. +std::vector msvcrt_argv(std::string_view line) { + std::vector argv; + std::string cur; + bool inQuotes = false, any = false; + std::size_t i = 0; + auto flush = [&] { if (any) { argv.push_back(cur); cur.clear(); any = false; } }; + while (i < line.size()) { + char c = line[i]; + if (c == '\\') { + std::size_t n = 0; + while (i < line.size() && line[i] == '\\') { ++n; ++i; } + if (i < line.size() && line[i] == '"') { + cur.append(n / 2, '\\'); + if (n % 2 == 0) inQuotes = !inQuotes; + else cur.push_back('"'); + any = true; + ++i; + } else { + cur.append(n, '\\'); + any = any || n > 0; + } + continue; + } + if (c == '"') { inQuotes = !inQuotes; any = true; ++i; continue; } + if (!inQuotes && (c == ' ' || c == '\t')) { flush(); ++i; continue; } + cur.push_back(c); + any = true; + ++i; + } + flush(); + return argv; +} + +constexpr std::string_view kJsonWithAFloor = + R"({"targets":["xim:shaderc@>=2026.3"],"yes":true})"; + +} // namespace + +TEST(WindowsCommandLine, PlainQuotingLetsCmdSeeARedirection) { + // The state before the fix, stated so the fix below is not asserting + // against nothing. This is `shell::quote_windows`, which is correct for + // the child and incomplete for cmd. + auto line = "xlings.exe interface install_packages --args " + + mcpp::platform::shell::quote_windows(kJsonWithAFloor); + auto parsed = simulate_cmd_c(proc::windows_wrap_for_cmd_c(line)); + EXPECT_TRUE(parsed.sawActiveMetacharacter) + << "if this ever becomes false the simulator stopped modelling cmd, " + "and the test below proves nothing"; +} + +TEST(WindowsCommandLine, MetacharacterQuotingSurvivesBothParsers) { + auto line = "xlings.exe interface install_packages --args " + + mcpp::platform::shell::quote_windows_through_cmd(kJsonWithAFloor); + auto parsed = simulate_cmd_c(proc::windows_wrap_for_cmd_c(line)); + + EXPECT_FALSE(parsed.sawActiveMetacharacter) + << "cmd would still read the `>` in the version floor as a redirection"; + + // xlings.exe / interface / install_packages / --args / + auto argv = msvcrt_argv(parsed.passedOn); + ASSERT_EQ(argv.size(), 5u) << parsed.passedOn; + EXPECT_EQ(argv[4], kJsonWithAFloor) + << "the child received something other than the JSON that was meant"; +} + +TEST(WindowsCommandLine, NothingToEscapeMeansByteIdenticalToPlainQuoting) { + // The conservative half of the rule, and it is the half that was measured + // the hard way: an earlier version escaped every metacharacter INCLUDING + // the quotes, which is defensible on paper and broke every package fetch on + // Windows -- including the ones whose JSON contains no metacharacter. A + // payload with nothing to escape must come out exactly as before. + for (std::string_view plain : { + R"({"targets":["mcpplibs:tpl-demo@1.0.0"],"yes":true})", + R"({"targets":["compat:widget@1.38.1"],"yes":true})", + R"(a plain path C:\Program Files\x)" }) { + EXPECT_EQ(mcpp::platform::shell::quote_windows_through_cmd(plain), + mcpp::platform::shell::quote_windows(plain)) + << "a payload with no metacharacter acquired an escape: " << plain; + } +} + +TEST(WindowsCommandLine, MetacharacterQuotingIsUnchangedForPlainText) { + // A payload with nothing to escape must not acquire carets, so the common + // case stays legible in a log. + constexpr std::string_view plain = R"({"targets":["xim:shaderc@2026.3"]})"; + auto quoted = mcpp::platform::shell::quote_windows_through_cmd(plain); + auto parsed = simulate_cmd_c(proc::windows_wrap_for_cmd_c("prog.exe --args " + quoted)); + auto argv = msvcrt_argv(parsed.passedOn); + ASSERT_EQ(argv.size(), 3u) << parsed.passedOn; + EXPECT_EQ(argv[2], plain); +}