Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify + Sustainability #334

Merged
merged 8 commits into from
Dec 15, 2023
Merged

Simplify + Sustainability #334

merged 8 commits into from
Dec 15, 2023

Conversation

fosslinux
Copy link
Owner

@fosslinux fosslinux commented Nov 24, 2023 •
edited
Loading

This is still fairly rough around the edges; but all of the core functionality I wanted has been implemented.

This does not create any meaningful change to the bootstrap path used. Rather this is a fairly significant refactor of live-bootstrap to make it easier to reason about, more sustainable to make changes to in the future, and fix a number of long-standing issues.

  • The most significant change made is the removal of the sys* system introduced very early in the project. That abstraction has become increasingly meaningless and ultimately created many issues jumping between syses, and sysa/sysc had so many different run scripts (run.kaem, run.sh, run-after-fiwix.sh, run2.sh), which didn't actually convey any meaning. There were so many varied cases depending on chroot/kernel bootstrap/etc. I believe that this is the biggest barrier to understanding the project. Technical details;
    • /init's are generated files now
    • The filesystem is persistent across the entire bootstrap; ie, packages are not reinstalled, rather everything is moved, after transitions. There are benefits and drawbacks to this, but for now, I think it is OK.
      • With the exception of builder-hex0 to fiwix, which needs a hardcoded list of files, that I would like to remove at some point, but I'm not sure if that will happen before 1.0.
  • Now, we have a human AND machine readable manifest that details the progression of the bootstrap, stored in steps/manifest. The M2-Planet program seed/script-generator.c transforms the manifest dynamically into a series of kaem/shell scripts that invoke one another. The syntax of the manifest is detailed in the manifest. All associated scripts required to get to the point where the manifest can be used (which is almost immediately after stage0-posix), live in seed/.
    • I think this will prove to be much more sustainable and sensible than the sys* system.
    • Further documentation will be created regarding what script-generator creates, but it is expected that the average contributor will not need to understand the abstraction of the manifest.
  • Another significant change is the clear separation of scripts and other sources external to live-bootstrap.
    • Everything that is NOT part of, or an artifact of, live-bootstrap, is stored in /external in the rootfs.
  • When using kernel bootstrap/Linux kernel, there is a new propagation mechanism for data between kernels.
    • In builder-hex0, everything exists on its srcfs.
    • In fiwix, /external is mounted from a disk, everything else comes from an initrd. Once util-linux is built, everything transfers into that disk, moving off the initrd.
    • In Linux as a seed (replacing fiwix and builder-hex0), the system boots from a single disk.
    • In Linux built by live-bootstrap, the system boots from a single disk - either the same one if Linux was a seed, or the one created in fiwix.
    • The primary benefit of this is we are no longer restricted by size of initrd - this was a requirement of this PR for me, as that was the primary cause of the sysa/sysc split.
    • This was achieved by adding a very janky and hampered mknod and mount to mes libc, and creating early command line utils for them named efsu (Early FS Utils)
  • The early kaem era has been modernized a bit, alongside environment variable handling.
    • The meaningless lowercase variables have been removed.
    • The /steps/env replaces .env. /steps/env propagates throughout the entire filesystem.
    • The system is musl-first in all cases, not mes-first. This means, that musl-era conforms largely to FHS, and mes-era largely does not conform to FHS. Practically, this means we have /usr/lib/mes and /usr/include/mes, and no /usr/include/musl.
    • I'm still uncertain about how to handle the mes environment variable spam. For now, I have tried to limit the scope of mes's environment variables as much as possible, and tcc & mes build scripts have been slightly decoupled - to be revisited at a later date.

TODOS

  • Lint
  • HELP NEEDED - kernel bootstrap mode is broken ?!. Consistently segfaults just into fiwix. statically compiled bash binary segfaults. memory mapping issue ?? not sure what's going on here
  • finish doc

@eduardosm
Copy link
Contributor

HELP NEEDED - kernel bootstrap mode is broken ?!. Consistently segfaults just into fiwix. statically compiled bash binary segfaults. memory mapping issue ?? not sure what's going on here

In order to make this a bit easier, I think some parts of this PR can be splitted to their own PRs, reducing the amount of changes introduced by this PR and hopefully making it easier to pinpoint the source of the problem.

However, that would require rebasing this PR and conflicts are expected, but I can help with that.

This was referenced Nov 28, 2023
Merged
Merged
Merged
@fosslinux
Copy link
Owner Author

fosslinux commented Nov 28, 2023 •
edited
Loading

I'm in the process of cleaning up the commit history for this PR, and fixing little bits and pieces.

I think I've split out everything from this PR that can be logically split out.

We have as dependencies for this PR;

@fosslinux
Copy link
Owner Author

Only large remaining error is fiwix -> linux not working. This is in the process of being fixed.

This was referenced Dec 5, 2023
Merged
Merged
@fosslinux fosslinux force-pushed the simplify branch 3 times, most recently from 5189a92 to cdcc783 Compare December 7, 2023 10:28
@fosslinux
Copy link
Owner Author

I'm fairly happy with this now. CI is obviously broken but I would still like to limit the scope of this PR.

@fosslinux fosslinux marked this pull request as ready for review December 7, 2023 10:29
@fosslinux
Add script-generator
0907cfd
@fosslinux
Remove the notion of "sys*"
6ed2e09 
- This idea originates from very early in the project and was, at the
  time, a very easy way to categorise things.
- Now, it doesn't really make much sense - it is fairly arbitary, often
  occuring when there is a change in kernel, but not from builder-hex0
  to fiwix, and sysb is in reality completely unnecessary.
- In short, the sys* stuff is a bit of a mess that makes the project
  more difficult to understand.
- This puts everything down into one folder and has a manifest file that
  is used to generate the build scripts on the fly rather than using
  coded scripts.
- This is created in the "seed" stage.

stage0-posix -- (calls) --> seed -- (generates) --> main steps

Alongside this change there are a variety of other smaller fixups to the
general structure of the live-bootstrap rootfs.

- Creating a rootfs has become much simpler and is defined as code in
  go.sh. The new structure, for an about-to-be booted system, is

/
-- /steps (direct copy of steps/)
-- /distfiles (direct copy of distfiles/)
-- all files from seed/*
-- all files from seed/stage0-posix/*

- There is no longer such a thing as /usr/include/musl, this didn't
  really make any sense, as musl is the final libc used. Rather, to
  separate musl and mes, we have /usr/include/mes, which is much easier
  to work with.
- This also makes mes easier to blow away later.
- A few things that weren't properly in packages have been changed;
  checksum-transcriber, simple-patch, kexec-fiwix have all been given
  fully qualified package names.
- Highly breaking change, scripts now exist in their package directory
  but NOT WITH THE packagename.sh. Rather, they use pass1.sh, pass2.sh,
  etc. This avoids manual definition of passes.
  - Ditto with patches; default directory is patches, but then any patch
    series specific to a pass are named patches-passX.
@fosslinux
Modify rootfs.py infrastructure to support the new layout
05c13dd
@fosslinux
Create a proper Linux kernel package
e06a19f
@fosslinux
Update docs with new changes
ee77ef8
@fosslinux
Update all checksums
ff4f97a
@fosslinux
Linters & CI updates
ab47483
@fosslinux
Install heirloom-devtools utility files to /usr/lib
ac4a8c7 
Before they were just in / for no real reason
@fosslinux fosslinux merged commit 545bb42 into master Dec 15, 2023
5 of 6 checks passed
@cosinusoidally cosinusoidally mentioned this pull request Dec 18, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@fosslinux @eduardosm