-
Notifications
You must be signed in to change notification settings - Fork 189
feat(system): implement unified memory manager and tiered zram/swap core #2223
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: next
Are you sure you want to change the base?
Conversation
|
Note all features can be enabled/disabled at runtime via overide on /etc/swap.conf placed into /storage/.config/swap.conf |
|
Thanks, the scope of this PR looks good. |
This commit implements Kernel Samepage Merging (KSM) orchestration as a native component of the rocknix-memory-manager, building upon the foundational infrastructure established in PR ROCKNIX#2223 (ROCKNIX#2223). KSM provides transparent page deduplication across the system, allowing the kernel to merge identical memory pages into a single copy-on-write reference. This is critical for ARM-based emulation environments where multiple standalone emulator instances or large guest RAM mappings frequently contain redundant data structures. Technical Implementation: 1. Kernel Configuration: * CONFIG_KSM is enabled across all target device kernel configurations. 2. Manager Integration: * Added setup_ksm() to the rocknix-memory-manager to configure scan intensity and sleep intervals via /sys/kernel/mm/ksm/ sysfs nodes. * The manager now executes KSM initialization during the --setup phase if ENABLE_KSM is configured. 3. Observability and Metrics: * Enhanced the --status command to extract and report real-time deduplication metrics. * Metrics include current KSM run status, pages shared (merged), and total RAM saved (shared - unique). 4. Hardware-Specific Profiling: * Implements tiered scan rates based on SoC performance tiers: * High-end (RK3588, SM8xxx): Enabled with aggressive scan rates (200 pages/200ms) to maximize RAM density. * Mid-range (RK3399, S922X): Enabled with moderate rates (100 pages/500ms). * Low-end (H700, RK3326, RK3566): Disabled by default to reserve all CPU cycles for primary emulation threads. 5. Build System Integration: * Extends the util-linux package overlay to propagate KSM tuning variables (ENABLE_KSM, KSM_SCAN_PAGES, KSM_SLEEP_MS) into the final /etc/swap.conf. This deduplication layer acts as a secondary efficiency tier above ZRAM, increasing effective RAM capacity without the latency penalties associated with physical storage I/O.
|
Can remove |
cc3c6ae to
b8fb013
Compare
This commit integrates Kernel Samepage Merging (KSM) into the unified 'rocknix-memory-manager'. KSM scans memory for identical pages and merges them, releasing RAM back to the system. This is particularly effective for emulators that fork processes or load redundant assets. Features: - Orchestration: KSM logic is injected into rocknix-memory-manager setup. - Tuning: Configurable pages-to-scan and sleep-milliseconds via options. - Observability: `rocknix-memory-manager --status` now reports KSM merge efficiency and saved RAM. Defaults: - High-End (RK3588, RK3399, etc): Enabled by default (Aggressive scan). - Low-End (H700, RK3326): Disabled by default to save CPU cycles. ref# ROCKNIX#2223
|
You are still manually editing kernel configs and not updating them properly, we can't see what options are being enabled in your PR as a result of this. You need to run For example SM8550 had the following changes |
I'm calling the upstream kernel config tool with my patch apply scripts . Was not aware of the in tree tool. Will use it in future now I'm aware of it's existence and preference. |
|
unless I'm mistaken, but so it can't be changed |
|
|
||
| post_install () { | ||
| if [ "${SWAP_SUPPORT}" = "yes" ]; then | ||
| enable_service swap.service |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
swap.service file can be removed I guess
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually looking over this further I think we should remove the upstream/libreelec mount-swap install step entirely from the ROCKNIX packages/sysutils/util-linux/package.mk ; i've checked through the codebase and it's not used at all and since we have quite different set of critera than what's in that script and handle it all within the rocknix-memory-manager its presence is just going to be confusing
projects/ROCKNIX/packages/rocknix/sources/scripts/rocknix-memory-manager
Show resolved
Hide resolved
Gah - the link logic got dropped in my refactor for @porschemad911 suggestions last night accidentally. |
Given this change affects all platforms, as requested, when you are happy with the code, please do a CI run in your repo as per: https://rocknix.org/contribute/build-gh/ This will also create a PR in your repo with full updated kconfig changes as well as images for testing. I know you're not a fan of GH, neither am I but this is what the project uses so we have to work with it. |
Actually this is fine as is - the memory-manager checks both locations and the user config override takes precedence if it exists. I think this is desired behaviour as is, as the build time options persist in the immutable /etc/swap.conf which is what we want for resets etc. |
727af26 to
70d18b3
Compare
projects/ROCKNIX/packages/rocknix/sources/scripts/rocknix-memory-manager
Outdated
Show resolved
Hide resolved
aec0f83 to
13a3e6e
Compare
…figs
This commit introduces a comprehensive overhaul of memory management across all
supported ROCKNIX devices. It deprecates the legacy \`mount-swap\` script in
favor of a new, robust \`rocknix-memory-manager\`.
Key Changes:
* New Memory Manager (\`rocknix-memory-manager\`):
* Replaces \`mount-swap\` with a Tiered Memory Architecture.
* Smart "ZSTD" Interpretation (Hybrid Tiering):
* If \`ZRAM_COMPRESSION_ALGO\` is set to "zstd", the manager automatically
configures a hybrid pipeline:
1. Primary: LZO-RLE (Low latency) for immediate writes to prevent stutter.
2. Secondary: ZSTD (High compression) enabled via \`CONFIG_ZRAM_MULTI_COMP\`
for background recompression of idle pages.
* Auto-Configuration & Defaults:
* High-Performance Targets (RK3588, S922X, SM8x50): Default to "Auto" ZRAM
sizing and "zstd" profile (Hybrid Tiering enabled).
* Low-Power Targets (RK3326, RK3566, H700): Default to ZRAM disabled ("0")
to conserve system resources.
* Swap File: Defaults to disabled ("0") on all targets to preserve flash storage.
* Tuning: Dynamic sysctl application (\`swappiness=100\`, \`page-cluster=0\`)
optimized for the new hybrid zram setup.
* Runtime Configuration:
* Users on low-end targets can enable ZRAM by copying defaults:
\`cp /etc/swap.conf /storage/.config/swap.conf\`
and setting \`ZRAM_SIZE="Auto"\` or a specific value (e.g., "512").
* Configuration supports user overrides for algorithm, sizing, and swap files.
* Kernel Configuration Standardization:
* ZRAM: Converted to modular (\`=m\`) to allow runtime algorithm switching.
* Multi-Compression: Enabled \`CONFIG_ZRAM_MULTI_COMP\` on capable SoCs
(Rockchip, Qualcomm, Amlogic) to support the LZO-RLE -> ZSTD pipeline.
* Algorithms: Enabled \`LZO-RLE\` (default), \`ZSTD\`, and \`LZ4\` modules.
* ZSMALLOC: Modularized (\`=m\`) with chain size 8.
* THP: Standardized to \`madvise\` to reduce unneeded compaction latency.
* Cgroups: Enabled \`CONFIG_CGROUP_FREEZER\` for consistent suspend handling.
* Platform Specifics:
* SDM845/SM8250: Enabled \`CONFIG_PSI\` and fixed toolchain versioning.
* Rockchip/Amlogic: Enabled architecture-specific migration features.
* System Services:
* Added \`rocknix-memory-manager.service\` (oneshot, pre-shutdown).
* Updated \`util-linux\` to enable \`fallocate\` for instant swap generation
13a3e6e to
d140447
Compare
| # Check case-insensitive for "auto" | ||
| if [[ "${ZRAM_SIZE,,}" == "auto" ]]; then | ||
| ZRAM_SIZE_VAL=$((TOTAL_MEM_MB / 2)) | ||
| [ "$ZRAM_SIZE_VAL" -gt 4096 ] && ZRAM_SIZE_VAL=4096 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
reserving 4Gb on 8Gb seems a lot to me
feat(system) Memory Management Overhaul: ZRAM Tiering, Swap, and Kernel Configs
This commit introduces a comprehensive overhaul of memory management across all
supported ROCKNIX devices. It deprecates the legacy `mount-swap` script in
favor of a new, robust `rocknix-memory-manager`.
Key Changes:
New Memory Manager (`rocknix-memory-manager`):
configures a hybrid pipeline:
for background recompression of idle pages.
sizing and "zstd" profile (Hybrid Tiering enabled).
to conserve system resources.
optimized for the new hybrid zram setup.
Runtime Configuration:
`cp /etc/swap.conf /storage/.config/swap.conf`
and setting `ZRAM_SIZE="Auto"` or a specific value (e.g., "512").
Kernel Configuration Standardization:
(Rockchip, Qualcomm, Amlogic) to support the LZO-RLE -> ZSTD pipeline.
Platform Specifics:
System Services: