Zcu: refactor File

This commit makes some big changes to how we track state for Zig source
files. In particular, it changes:

* How `File` tracks its path on-disk
* How AstGen discovers files
* How file-level errors are tracked
* How `builtin.zig` files and modules are created

There is also one breaking change here, which is that `@import` of a
non-existent module is now a compile error even if the import is not
semantically analyzed.

The original motivation here was to address incremental compilation bugs
with the handling of files, such as #22696. To fix this, a few changes
are necessary.

Just like declarations may become unreferenced on an incremental update,
meaning we suppress analysis errors associated with them, it is also
possible for all imports of a file to be removed on an incremental
update, in which case file-level errors for that file should be
suppressed. As such, after AstGen, the compiler must traverse files
(starting from analysis roots) and discover the set of "live files" for
this update.

Additionally, the compiler's previous handling of retryable file errors
was not very good; the source location the error was reported as was
based only on the first discovered import of that file. This source
location also disappeared on future incremental updates. So, as a part
of the file traversal above, we also need to figure out the source
locations of imports which errors should be reported against.

Another observation I made is that the "file exists in multiple modules"
error was not implemented in a particularly good way (I get to say that
because I wrote it!). It was subject to races, where the order in which
different imports of a file were discovered affects both how errors are
printed, and which module the file is arbitrarily assigned, with the
latter in turn affecting which other files are considered for import.
The thing I realised here is that while the AstGen worker pool is
running, we cannot know for sure which module(s) a file is in; we could
always discover an import later which changes the answer.

So, here's how the AstGen workers have changed. We initially ensure that
`zcu.import_table` contains the root files for all modules in this Zcu,
even if we don't know any imports for them yet. Then, the AstGen
workers do not need to be aware of modules. Instead, they simply ignore
module imports, and only spin off more workers when they see a by-path
import. During this phase, we work exclusively in absolute paths, to
ensure consistency in terms of `NameTooLong` errors.

After the AstGen workers all complete, we know that any file which might
be imported is definitely in `import_table` and up-to-date. So, we
perform a single-threaded graph traversal; similar to what
`resolveReferences` plays for `AnalUnit`s, but for files instead. We
figure out which files are alive, and which module each file is in. If a
file turns out to be in multiple modules, we set a field on `Zcu` to
indicate this error. If a file is in a different module to a prior
update, we set a flag instructing `updateZirRefs` to invalidate all
dependencies on the file. This traversal also discovers "import errors";
these are errors associated with a specific `@import`. There are two
possible errors here: "module not found" when a module import uses an
unmapped name, or "import outside of module root" when importing a file
using `..` past the module root path. These errors have to be identified
during this traversal because they depend on which module the file is
in, which we don't know until now.

For simplicity, `failed_files` now just maps to `?[]u8`, since the
source location is always the whole file. In fact, this allows removing
`LazySrcLoc.Offset.entire_file` completely, slightly simplifying some
error reporting logic. File-level errors are now directly built in the
`std.zig.ErrorBundle.Wip`. If the payload is not `null`, it is the
message for a retryable error (i.e. an error loading the source file),
and will be reported with a "file imported here" note pointing to the
import site discovered during the single-threaded file traversal.

The last piece of fallout here is how `Builtin` works. Rather than
constructing "builtin" modules when creating `Package.Module`s, they are
now constructed on-the-fly by `Zcu`. The map `Zcu.builtin_modules` maps
from digests to `*Package.Module`s. These digests are abstract hashes of
the `Builtin` value; i.e. all of the options which are placed into
"builtin.zig". During the file traversal, we populate `builtin_modules`
as needed, so that when we see this imports in Sema, we just grab the
relevant entry from this map. This eliminates a bunch of awkward state
tracking during construction of the module graph. It's also now clearer
exactly what options the builtin module has, since previously it
inherited some options arbitrarily from the first-created module with
that "builtin" module!

The actual effects of this commit are:
* retryable file errors are now consistently reported against the whole
  file, with a note pointing to a live import of that file
* incremental updates do not print retryable file errors differently
  between updates
* incremental updates support files changing modules
* incremental updates support files becoming unreferenced

Resolves: #22696

Author: mlugg

build.zig

@@ -8,12 +8,24 @@ const io = std.io;
 const fs = std.fs;
 const InstallDirectoryOptions = std.Build.InstallDirectoryOptions;
 const assert = std.debug.assert;
-const DevEnv = @import("src/dev.zig").Env;
 const ValueInterpretMode = enum { direct, by_name };
 
 const zig_version: std.SemanticVersion = .{ .major = 0, .minor = 15, .patch = 0 };
 const stack_size = 46 * 1024 * 1024;
 
+/// Keep in sync with `Env` in `src/dev.zig`.
+const DevEnv = enum {
+    bootstrap,
+    core,
+    full,
+    c_source,
+    ast_gen,
+    sema,
+    @"x86_64-linux",
+    @"riscv64-linux",
+    wasm,
+};
+
 pub fn build(b: *std.Build) !void {
     const only_c = b.option(bool, "only-c", "Translate the Zig compiler to C code, with only the C backend enabled") orelse false;
     const target = b.standardTargetOptions(.{

lib/std/Build/Cache.zig

@@ -276,11 +276,13 @@ pub const HashHelper = struct {
     }
 
     pub fn oneShot(bytes: []const u8) [hex_digest_len]u8 {
+        return binToHex(oneShotBin(bytes));
+    }
+
+    pub fn oneShotBin(bytes: []const u8) BinDigest {
         var hasher: Hasher = hasher_init;
         hasher.update(bytes);
-        var bin_digest: BinDigest = undefined;
-        hasher.final(&bin_digest);
-        return binToHex(bin_digest);
+        return hasher.finalResult();
     }
 };
 

src/Builtin.zig

@@ -19,6 +19,26 @@ code_model: std.builtin.CodeModel,
 omit_frame_pointer: bool,
 wasi_exec_model: std.builtin.WasiExecModel,
 
+/// Compute an abstract hash representing this `Builtin`. This is *not* a hash
+/// of the resulting file contents.
+pub fn hash(opts: @This()) [std.Build.Cache.bin_digest_len]u8 {
+    var h: Cache.Hasher = Cache.hasher_init;
+    inline for (@typeInfo(@This()).@"struct".fields) |f| {
+        if (comptime std.mem.eql(u8, f.name, "target")) {
+            // This needs special handling.
+            std.hash.autoHash(&h, opts.target.cpu);
+            std.hash.autoHash(&h, opts.target.os.tag);
+            std.hash.autoHash(&h, opts.target.os.versionRange());
+            std.hash.autoHash(&h, opts.target.abi);
+            std.hash.autoHash(&h, opts.target.ofmt);
+            std.hash.autoHash(&h, opts.target.dynamic_linker);
+        } else {
+            std.hash.autoHash(&h, @field(opts, f.name));
+        }
+    }
+    return h.finalResult();
+}
+
 pub fn generate(opts: @This(), allocator: Allocator) Allocator.Error![:0]u8 {
     var buffer = std.ArrayList(u8).init(allocator);
     try append(opts, &buffer);
@@ -263,22 +283,49 @@ pub fn append(opts: @This(), buffer: *std.ArrayList(u8)) Allocator.Error!void {
     }
 }
 
-pub fn populateFile(comp: *Compilation, mod: *Module, file: *File) !void {
-    if (mod.root.statFile(mod.root_src_path)) |stat| {
+/// This essentially takes the place of `Zcu.PerThread.updateFile`, but for 'builtin' modules.
+/// Instead of reading the file from disk, its contents are generated in-memory.
+pub fn populateFile(opts: @This(), gpa: Allocator, file: *File) Allocator.Error!void {
+    assert(file.is_builtin);
+    assert(file.status == .never_loaded);
+    assert(file.source == null);
+    assert(file.tree == null);
+    assert(file.zir == null);
+
+    file.source = try opts.generate(gpa);
+
+    log.debug("parsing and generating 'builtin.zig'", .{});
+
+    file.tree = try std.zig.Ast.parse(gpa, file.source.?, .zig);
+    assert(file.tree.?.errors.len == 0); // builtin.zig must parse
+
+    file.zir = try AstGen.generate(gpa, file.tree.?);
+    assert(!file.zir.?.hasCompileErrors()); // builtin.zig must not have astgen errors
+    file.status = .success;
+}
+
+/// After `populateFile` succeeds, call this function to write the generated file out to disk
+/// if necessary. This is useful for external tooling such as debuggers.
+/// Assumes that `file.mod` is correctly set to the builtin module.
+pub fn updateFileOnDisk(file: *File) !void {
+    assert(file.is_builtin);
+    assert(file.status == .success);
+    assert(file.source != null);
+
+    if (file.mod.?.root.statFile("builtin.zig")) |stat| {
         if (stat.size != file.source.?.len) {
             std.log.warn(
-                "the cached file '{}{s}' had the wrong size. Expected {d}, found {d}. " ++
+                "the cached file '{}{c}builtin.zig' had the wrong size. Expected {d}, found {d}. " ++
                     "Overwriting with correct file contents now",
-                .{ mod.root, mod.root_src_path, file.source.?.len, stat.size },
+                .{ file.mod.?.root, std.fs.path.sep, file.source.?.len, stat.size },
             );
-
-            try writeFile(file, mod);
         } else {
             file.stat = .{
                 .size = stat.size,
                 .inode = stat.inode,
                 .mtime = stat.mtime,
             };
+            return;
         }
     } else |err| switch (err) {
         error.BadPathName => unreachable, // it's always "builtin.zig"
@@ -287,26 +334,13 @@ pub fn populateFile(comp: *Compilation, mod: *Module, file: *File) !void {
         error.NoDevice => unreachable, // it's not a pipe
         error.WouldBlock => unreachable, // not asking for non-blocking I/O
 
-        error.FileNotFound => try writeFile(file, mod),
+        error.FileNotFound => {},
 
         else => |e| return e,
     }
 
-    log.debug("parsing and generating '{s}'", .{mod.root_src_path});
-
-    file.tree = try std.zig.Ast.parse(comp.gpa, file.source.?, .zig);
-    assert(file.tree.?.errors.len == 0); // builtin.zig must parse
-
-    file.zir = try AstGen.generate(comp.gpa, file.tree.?);
-    assert(!file.zir.?.hasCompileErrors()); // builtin.zig must not have astgen errors
-    file.status = .success;
-    // Note that whilst we set `zir` here, we populated `path_digest`
-    // all the way back in `Package.Module.create`.
-}
-
-fn writeFile(file: *File, mod: *Module) !void {
     var buf: [std.fs.max_path_bytes]u8 = undefined;
-    var af = try mod.root.atomicFile(mod.root_src_path, .{ .make_path = true }, &buf);
+    var af = try file.mod.?.root.atomicFile("builtin.zig", .{ .make_path = true }, &buf);
     defer af.deinit();
     try af.file.writeAll(file.source.?);
     af.finish() catch |err| switch (err) {
@@ -331,6 +365,7 @@ fn writeFile(file: *File, mod: *Module) !void {
 const builtin = @import("builtin");
 const std = @import("std");
 const Allocator = std.mem.Allocator;
+const Cache = std.Build.Cache;
 const build_options = @import("build_options");
 const Module = @import("Package/Module.zig");
 const assert = std.debug.assert;