Parse Starlark files as raw bytes for Bzlmod

[fmeum]

As long as Bazel internally represents strings as raw bytes "encoded" in Latin-1, the same must be true for all Starlark files that may contain file system paths.

Also includes changes to the Python test setup:

• ScratchFile now always writes files as UTF-8
• RunProgram encodes and decodes stdin/stderr/stdout as UTF-8
• download no longer leaks a file

[fmeum]

This needs #24010 to get its test to pass.

[fmeum]

@bazel-io fork 8.0.0

[fmeum]

Ready for review now.

@tjgq FYI, this is the other Unicode-related PR I am planning to get into 8.0.0. If you see anything else that looks incompatible, please let me know.

[fmeum]

Turns out that there is an interesting failure, switching back to draft.

[fmeum]

@Wyverald CI should be green now, I just had to fix some issues in the Python test setup code.

[Wyverald]

hmm, actually -- does this require the "re-land" PR first? if so, we should hold back on the import.

[fmeum]

hmm, actually -- does this require the "re-land" PR first? if so, we should hold back on the import.

I think it does for the test to pass.

[fmeum]

@Wyverald The reland has happened.

[Wyverald]

Do I understand correctly that, while this PR fixes the use case of non-ASCII characters in file paths, it "breaks" the case where non-ASCII characters are passed in e.g. string attrs in tags?

[fmeum]

Do I understand correctly that, while this PR fixes the use case of non-ASCII characters in file paths, it "breaks" the case where non-ASCII characters are passed in e.g. string attrs in tags?

It actually fixes both: The .bzl files containing the module extension source are read as raw bytes, module and repo context functions read and write raw bytes from files, ... With this change, strings attrs in tags will also line up with e.g. string literals in extension code.

[Wyverald]

I see!

module and repo context functions read and write raw bytes from files

I just double-checked, and it seems that we try to write files in UTF-8 in mctx/rctx: code

It's been broken like this forever, so presumably nobody has been relying on it working...

[Wyverald]

Also, presumably this means print(some_tag.some_attr) could yield gibberish, but that's nothing new. (To be clear, the import is already under way -- this is just me trying to understand the problem better)

[fmeum]

I just double-checked, and it seems that we try to write files in UTF-8 in mctx/rctx: code
It's been broken like this forever, so presumably nobody has been relying on it working...

That's being fixed in #24182 :-)

Also, presumably this means print(some_tag.some_attr) could yield gibberish, but that's nothing new.

It doesn't, since the Bazel client ultimately outputs raw bytes as well. That's being tested in #24243.

To be clear, the import is already under way -- this is just me trying to understand the problem better

In case it is not too late, I pushed a new test to demonstrate this end-to-end for tags.

[Wyverald]

Ahh, thanks for the explanation. It sounds like we're actually just treating Starlark strs as byte arrays everywhere now, right?

In case it is not too late, I pushed a new test to demonstrate this end-to-end for tags.

Ack -- I'll make sure that's imported too

[fmeum]

Ahh, thanks for the explanation. It sounds like we're actually just treating Starlark strs as byte arrays everywhere now, right?

Yes, that's correct. It's actually not the worst in a world where you increasingly deal with binary data and UTF-8 strings only.

[fmeum]

@bazel-io fork 8.0.0