`core:os` tracking issue

[laytan]

We are nearing the replacement of core:os, this issue outlines todos and nice to haves.

Todos:

• core:os/os2 read_entire_file creates infinite loop #3279
• os2.pipe() -> Assert error #4712
• @gingerBill needs to look at the env API, env_posix.odin has a bunch of NOTE(laytan): comments outlining potential improvements, same thing on the directory iterator, these comments are mostly about surfacing errors instead of ignoring them or returning just a boolean (which core:os famously does a lot and we should avoid) (b3bbb00 & 7994405)
• the IMPORTANT NOTE saying the package is a mockup needs to be removed from doc.odin
• write on linux needs to loop until everything is written, just like windows and posix implementations (os/os2: bring Linux to other impls standards by looping writes and maxing one shot RW sizes #4730)
• linux does not do the MAX_RW thing other implementations do but afaik it does have the same restriction (os/os2: bring Linux to other impls standards by looping writes and maxing one shot RW sizes #4730)
• remove on posix needs to handle directories, just like the other implementations (it already did, whoops)
• unused File_Flag cases should be removed, .Sparse, .Unbuffered_IO
• core:path needs to be looked over and possibly removed/merged into os2 (Add new path API for os2 #4954)
• Flysand: create() does not allow to set the permission bits or specify the behavior when the file exists (.Trunc or .Excl). I would like to see truncate parameter to that procedure that is false by default. If it's not specified, .Excl is the behavior on existing files, otherwise existing file is truncated.
• atomic bulk operations
• bulk operations errors with more info on what actually went wrong and where, so instead of Permission_Denied, get a path back or something too
• Errors are not detailed enough, often getting an .Unknown, this is because it is built on io.Error

Targets supported in core:os but not yet in core:os/os2:

• wasi (os/os2: wasi target support #4716)
• js - just os.write is implemented, maybe it should just be dropped entirely for os2
• haiku - should be pretty easy to use the posix implementation at first, can go "native" later

Non blocking but can have api consequences:

• buffered file mode/flag (needs testing to see if it's beneficial)
• Jeroen: Virtual File System - file possibly backed by not actual files

Packages that use core:os and need to be updated: (Started in #5858)

• core:compress/gzip
• core:crypto
• core:encoding/csv
• core:encoding/hxa
• core:encoding/ini
• core:encoding/xml
• core:flags
• core:fmt
• core:image/bmp
• core:image/jpeg
• core:image/netpbm
• core:image/png
• core:image/qoi
• core:image/tga
• core:image
• core:log
• core:math/big
• core:mem/virtual
• core:net
• core:odin/parser
• core:path/filepath (todo: only rel and dir to look at adopting or not in core:os/os2)
• core:prof/spall
• core:terminal
• core:testing
• core:text/i18n
• core:text/regex
• core:text/table
• core:time/timezone
• core:unicode/tools
• tests/core/encoding/hxa
• tests/core/flags
• tests/core/io
• tests/core/os
• tests/documentation
• vendor:fontstash
• vendor:libc
• vendor:opengl
• examples/all (for all of the odin checks at the moment, later also for documentation)
• odin-lang/examples repo needs to be updated

Nice to have, but not blocking the replacement of core:os:

• JS impl of core:fmt uses a 1k stack buffer to format into, add a way to grow if that's insufficient? Someone who uses wasm may want to look at this.
• more tests
• docs additions, dir.odin, file.odin, etc. don't have any docs
• docs unification, doc style differs between files and procs
• linux process_info should query /proc/pid/exe instead of the way it's done now to get executable paths
• netbsd, openbsd, freebsd targets implementation for _process_info_by_pid, _process_list, _process_open, _process_handle_still_valid, _process_state_update_times for full process API coverage (decided to make non blocking because it is a new API)
• current_executable_path(allocator: runtime.Allocator) -> (string, Error) returning the full absolute executable path of the program, useful for loading things relative to this path. You can already do this in a round about way with: current_process_info({.Executable_Path}, context.allocator) but a more direct way with the native APIs for each target would be nice (os/os2: add get_executable_path and get_executable_directory #4733)
• wrapper api over the process API that handles the details of processes and just allows easy communication, similar to this idea: https://gist.github.com/laytan/68be38614d9274663a48fd3d710fefc2
• a heap allocator that does not rely on any system libraries, ideally this goes into base: though, so it can be the default allocator in general (Add native lock-free dynamic heap allocator #4749)
• core:testing to replace os2.stdout and os2.stderr with an implementation that synchronises between tests (currently writing to stdout or stderr in tests doesn't work well because the terminal renders progress and clears it out)
• Flysand: There is no procedure to free a file handle without closing it (which can be used in tandem with new_file, if i.e. a library returned a handle, and that library is responsible for closing the file).
• Flysand: It would be nice to have unlink :: remove alias.

After replacement the following can be closed:

• Spawning a process #3325
• core:os package read_entire_file_xxx procedures don't work on /proc filesystem even if on a Linux system #4264
• Implement os.environ() on Linux #3154
• copy_file implemented for Windows, Linux and Darwin #3186

[Kelimion]

Virtual File System, working similarly to allocators, in that you call it with a Mode, instead of having to have vtable with all of stat , open , close , read ,write, etc.

[flysand7]

doc style differs between files and procs

Pretty sure I messed up files a little bit. I (or someone else) will have to rewrite the documentation at some point for files. process.odin should be the most recent and the most correct style, but still not ideal.

[laytan]

Virtual File System, working similarly to allocators, in that you call it with a Mode, instead of having to have vtable with all of stat , open , close , read ,write, etc.

I put it on the list under "non blocking but can have api consequences"

[Feoramund]

I have a private branch that has a chunk of what's outlined here already done (for os->os2 conversion), including a test suite.

It's been a while since I touched it (~2 months), so I'll have to revisit what's changed since. The main blocker for me was rewriting @jasonKercher's heap allocator to be thread-safe, which took quite some time, debugging, research, and learning of new skills.

I'm aiming to get a draft PR of the new allocator up in a couple days. I'll need some help with testing and reviewing non-Linux system calls with regards to virtual memory, in particular.

I'm currently working through integrating it into base and fixing some cyclic import issues. I had hoped to avoid exposing too many new symbols in base:runtime by using a sub-package such as base:runtime/heap, but fixing the cyclic import would've been too much of a pain, so I'm just putting everything in base now. I'll be using a liberal application of prefixes so it's clear what's only for the heap API. @(private="file") wouldn't work either, because my allocator test bench uses the internals, and I think it's just as well for a low-level API to be available.

EDIT: To be perfectly clear, when I say draft PR, I mean that the allocator is largely finished from a design and implementation standpoint, but it's untested outside of microbenchmarks, my test bench, and the Odin core: test suite. If someone's willing to hook it up to a real application once the PR's up, that'd be great.

[laytan]

Sounds great @Feoramund , excited to see it!

[jasonKercher]

Me too, and I'm very sorry ;]

[flysand7]

Some things I've noted when looking at os2/file API:

1. create() does not allow to set the permission bits or specify the behavior when the file exists (.Trunc or .Excl). I would like to see truncate parameter to that procedure that is false by default. If it's not specified, .Excl is the behavior on existing files, otherwise existing file is truncated.
2. open_buffered() is still commented out and .Unbuffered_IO flag is unused.
3. There is no procedure to free a file handle without closing it (which can be used in tandem with new_file, if i.e. a library returned a handle, and that library is responsible for closing the file).
4. It would be nice to have unlink :: remove alias.

[laytan]

Some things I've noted when looking at os2/file API:

1. create() does not allow to set the permission bits or specify the behavior when the file exists (.Trunc or .Excl). I would like to see truncate parameter to that procedure that is false by default. If it's not specified, .Excl is the behavior on existing files, otherwise existing file is truncated.
2. open_buffered() is still commented out and .Unbuffered_IO flag is unused.
3. There is no procedure to free a file handle without closing it (which can be used in tandem with new_file, if i.e. a library returned a handle, and that library is responsible for closing the file).
4. It would be nice to have unlink :: remove alias.

I added 1, 3 and 4 to the list, 2 was already there

[Feoramund]

The heap allocator PR is up at #4749

[Feoramund]

I've been working on this again lately, specifically reimplementing some of the functionality that filepath had into os2 directly in order to work around the cyclic import problem. I think filepath should be removed, possibly in favor of platform-agnostic procedures for converting between systems.

So far I've simplified the path splitting API into split_path(path) -> (dir, filename) and split_filename(filename) -> (base, ext), with a separate proc split_filename_all which returns the longest extension possible tar.gz instead of gz in the case of foo.tar.gz.

[laytan]

Updated with the addition of #4954 🔥

[flysand7]

@laytan This issue also needs to be resolved for os2: #4999. I've written down the gist of it, and looked at os2 code for reading from console, and well, it is still there hidden behind a TODO.

Odin/core/os/os2/file_windows.odin

Line 323 in 9516084

// TODO(bill): should this be moved to `File_Impl` instead?

[deepankarsharma]

Experienced systems engineer but new Odin user here. Is there anyway I can help out with this effort?