Skip to content

overlay: keep plausible time across reboots without an RTC - #2336

Merged
openipc-ai merged 2 commits into
masterfrom
overlay/fake-hwclock
Aug 30, 2026
Merged

openipc-ai merged 2 commits into
masterfrom
overlay/fake-hwclock

Conversation

@openipc-ai

Copy link
Copy Markdown
Collaborator

What this solves

A camera without a battery-backed clock boots at the firmware build stamp (TIME_STAMP in /etc/os-release, applied by S30customizer). On a camera with no network, every session therefore starts at the same moment: recordings collide with earlier ones, log timestamps repeat, and TLS validation trips over certificate validity windows. #1886 (@danielbanar) diagnosed this and proposed a fake-hwclock package; this PR supersedes it — the redesign below is why it is a new PR rather than a rework of that branch, and Daniel is credited as co-author.

Why not #1886 as it stood

  • Its daemon wrote /etc/fake-hwclock.data every 5 seconds. /etc lives on the overlay — jffs2 on NOR, where the partition is a handful of 64 KB erase blocks (as small as 704 KB) that jffs2 can only wear-level within. That is ~6.3M writes a year against a ~100k-cycle budget.
  • Its init script used the pidfile start-stop-daemon pattern the tree removed in init scripts: match daemons by name, not by a stale pidfile #2326.
  • It shipped as an opt-in package; time that is only plausible on cameras whose owner found a hidden knob does not fix the out-of-the-box recording behaviour. This PR enables it for every board — deliberately revisiting the "not by default" position from the fake-hwclock implementation #1886 discussion, with the wear numbers below as the argument that default-on is now safe.

Design

Surveyed OpenWrt sysfixtime, Debian/RPi fake-hwclock, systemd-timesyncd, and OpenRC swclock (mechanisms verified in their sources). This is a combination of the pieces that fit a hard-power-cut camera:

Restore (boot, S02) — forward-only, from the best of three candidates:

  • the checkpoint file /etc/fake-hwclock.data;
  • the newest file mtime under /etc (OpenWrt's trick: organic config writes keep this floor advancing for free, so even a camera that never checkpoints resumes from its last config write);
  • the SoC RTC via /sys/class/rtc/rtc0/since_epoch where one exists — read through sysfs rather than hwclock -s, because a stale battery-less RTC must lose to a newer checkpoint, never yank the clock backwards.

The clock is only ever moved forward, so this composes with S30customizer's build-stamp guard in either order.

Checkpoint (write to flash) — cadence bounded by what the overlay actually sits on:

medium interval writes/year
jffs2 (NOR) 3600 s ~8.8k
ubifs (NAND, UBI levels across the whole chip) 900 s ~35k
ext4/f2fs/vfat (eMMC/SD) 600 s ~53k
tmpfs (NFS/RAM root) never 0

Overridable via /etc/default/fake-hwclock (INTERVAL=, 0 disables). Each save also refuses to persist a clock at or below the build stamp (a clock nothing set yet would only launder garbage into the file), never moves the checkpoint backwards, and writes the RTC forward where present.

Three triggers share that one guarded path:

  • busybox ntpd's -S hook (S49ntpd now passes -S /usr/sbin/ntpd-script): stratum/step — the first real sync — checkpoints immediately; periodic (every 11 min while synced) is rate-limited to the medium interval; unsync is ignored.
  • a fallback loop for the offline camera fake-hwclock implementation #1886 was actually about — an unsynced ntpd fires no events at all;
  • clean shutdown (rcK), as a bonus, never as the mechanism.

Evidence (hi3516av300, NOR flash, jffs2 overlay, IMX415)

Method: the camera ran today's master build (2.6.08.29, master+fc923b7) with the four files from this branch copied onto its overlay — they are plain shell, so the copies are exactly what the image would ship (an hi3516av300_lite image was also built from this branch to verify installation and size, but was not flashed). NTP was made unreachable by pointing /etc/ntp.conf at 192.0.2.1 (TEST-NET). The camera was returned to stock afterwards and verified over a final reboot.

Hard reset with a stale RTC and no NTP — power-cut simulation (reboot -f, so no shutdown hook ran; RTC deliberately written back to the build era first):

== pre-reset:  checkpoint=1788038583  real=1788038583 (Sat Aug 29 21:23:03 UTC 2026)
   cold state: sys=1787975900 rtc=1787975900   ntp.conf: server 192.0.2.1
== after boot (uptime 51s):
   date: Sat Aug 29 21:23:51 UTC 2026   (wall-clock real time: 21:24:10)
   Aug 29 21:23:03 ... user.notice fake-hwclock: clock restored to 2026-08-29 21:23:03 UTC

Boot came up ~19 s behind real time — the checkpoint's age. Stock behaviour for this state is the build stamp, ~17.6 h behind.

Guards (ntpd stopped):

date -s @1787974800            # below the build stamp
fake-hwclock save              # -> rc=1, checkpoint untouched
fake-hwclock load              # 03:40 -> 21:11:44, forward to the checkpoint

Backwards-set never observed; a load with a correct clock is a no-op (delta 0 s), including with a deliberately stale RTC present.

ntpd hook: after S49ntpd restart with the -S flag, the stratum event checkpointed within 5 s of sync; periodic events did not write again before the interval elapsed (rate-limit observed).

Daemon: with INTERVAL=5 override, periodic saves observed; stop kills the pid (verified against /proc/<pid>/cmdline, not blindly) and writes a final checkpoint.

Image: hi3516av300_lite builds clean; rootfs 7624/8192 KB. The three scripts cost ~2.8 KB comment-stripped.

Not tested on NAND/ubifs or eMMC boards — the medium detection is a /proc/mounts fstype match and falls back to the most conservative (hourly) cadence for anything unrecognised.

Supersedes #1886.

Co-authored-by: Daniel Banar daniel.banar5@gmail.com

Cameras have no battery-backed clock, so every boot starts at the
firmware build stamp and an offline camera stamps each session's
recordings with the same times over and over (#1886).

Restore is forward-only at S02: the newer of a checkpoint file and the
newest mtime under /etc wins, and only ever moves the clock ahead, so
it composes with the build-stamp guard in S30customizer in either
order. The mtime floor means even a camera that dies mid-write or
never checkpoints resumes from its last organic config write, the way
OpenWrt's sysfixtime does.

Checkpointing is bounded by the flash the overlay sits on: hourly on
jffs2 (a NOR overlay is a handful of erase blocks that jffs2 can only
wear-level within), 15 min on ubifs (UBI levels across the whole
NAND), 10 min on eMMC/SD media, never on tmpfs. busybox ntpd's -S hook
reports sync events so the first sync is checkpointed immediately, and
a small fallback loop covers the offline camera an unsynced ntpd never
fires events for. A checkpoint below the build stamp is refused: a
clock nothing ever set only launders garbage into the file.

Co-authored-by: Daniel Banar <daniel.banar5@gmail.com>
@qodo-free-for-open-source-projects

Copy link
Copy Markdown

PR Summary by Qodo

Preserve plausible camera time across RTC-less reboots

🐞 Bug fix ✨ Enhancement ⚙️ Configuration changes 🕐 20-40 Minutes

Grey Divider

AI Description

• Restores boot time forward from persistent, filesystem, and SoC RTC candidates.
• Checkpoints time at storage-aware intervals to limit flash wear.
• Integrates early boot, NTP events, offline fallback, and clean shutdown by default.
Diagram

sequenceDiagram
    participant Init as S02 Init
    participant FHC as Fake Clock
    participant State as Overlay State
    participant RTC as SoC RTC
    participant Clock as System Clock
    participant NTP as BusyBox ntpd
    Init->>FHC: load
    FHC->>State: read checkpoint and mtimes
    FHC->>RTC: read epoch
    FHC->>Clock: advance to newest
    Init->>FHC: start fallback daemon
    NTP->>FHC: send sync event
    FHC->>State: checkpoint by cadence
    FHC->>RTC: write forward
    Init->>FHC: save on shutdown
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Adopt a standard fake-hwclock package
  • ➕ Uses an established implementation
  • ➕ Reduces custom maintenance
  • ➖ Generic save schedules may cause excessive NOR wear
  • ➖ Does not directly integrate overlay mtimes or BusyBox ntpd events
  • ➖ May remain optional instead of fixing default offline behavior
2. Checkpoint only during clean shutdown
  • ➕ Minimizes persistent writes
  • ➕ Requires no background process
  • ➖ Fails for common hard-power-cut shutdowns
  • ➖ Cannot preserve recent NTP or offline elapsed time

Recommendation: Keep the embedded-specific implementation: forward-only candidate selection and storage-aware writes directly address battery-less cameras and constrained flash. Standard packages and shutdown-only persistence do not adequately cover hard power loss, though automated shell tests for guards, event handling, and filesystem detection would reduce regression risk.

Files changed (4) +224 / -1

Enhancement (2) +166 / -0
fake-hwclockImplement forward-only software RTC persistence +161/-0

Implement forward-only software RTC persistence

• Adds checkpoint restore from saved time, '/etc' mtimes, and the optional SoC RTC without moving the clock backward. Saves are serialized, guarded by the firmware build timestamp, rate-limited by overlay medium, and triggered by NTP, an offline fallback loop, or shutdown.

general/overlay/usr/sbin/fake-hwclock

ntpd-scriptBridge ntpd events to software clock checkpoints +5/-0

Bridge ntpd events to software clock checkpoints

• Adds the BusyBox ntpd hook entry point that forwards event types to the fake-hwclock event handler.

general/overlay/usr/sbin/ntpd-script

Bug fix (1) +54 / -0
S02fakehwclockStart time restoration early and checkpoint during shutdown +54/-0

Start time restoration early and checkpoint during shutdown

• Adds an S02 init service that restores plausible time before later boot services and starts the offline checkpoint daemon. Shutdown validates the daemon PID against procfs before stopping it and saving a final checkpoint.

general/overlay/etc/init.d/S02fakehwclock

Other (1) +4 / -1
S49ntpdForward BusyBox NTP synchronization events +4/-1

Forward BusyBox NTP synchronization events

• Configures ntpd's '-S' hook so initial synchronization, clock steps, and periodic events can trigger guarded time checkpoints.

general/overlay/etc/init.d/S49ntpd

@qodo-free-for-open-source-projects

qodo-free-for-open-source-projects Bot commented Aug 29, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Upgrade leaves flash writer active ✓ Resolved 🐞 Bug ☼ Reliability
Description
The new daemon periodically writes /etc/fake-hwclock.data, but sysupgrade's fixed service-stop
path does not stop it before overlay erasure. If the overlay cannot be remounted read-only,
sysupgrade deliberately erases it live while this daemon can still write to the same flash,
recreating the concurrent-writer corruption race that quiesce_overlay() is meant to prevent.
Code

general/overlay/etc/init.d/S02fakehwclock[R28-29]

+	"$DAEMON" daemon &
+	echo "OK"
Evidence
The PR introduces an indefinite writer to an /etc file; sysupgrade stops other services but omits
it, and its documented fallback proceeds to raw overlay erasure even when remounting read-only
fails.

general/overlay/etc/init.d/S02fakehwclock[21-39]
general/overlay/usr/sbin/fake-hwclock[19-21]
general/overlay/usr/sbin/fake-hwclock[71-84]
general/overlay/usr/sbin/fake-hwclock[118-131]
general/overlay/usr/sbin/sysupgrade[338-370]
general/overlay/usr/sbin/sysupgrade[373-380]
general/overlay/usr/sbin/sysupgrade[474-487]
general/overlay/usr/sbin/sysupgrade[511-517]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Sysupgrade leaves the new persistent overlay writer running while it may erase rootfs_data live.
## Issue Context
Add fake-hwclock to the sysupgrade stop/restore lifecycle and ensure the process has actually exited before overlay quiescing and erase begin.
## Fix Focus Areas
- general/overlay/etc/init.d/S02fakehwclock[21-39]
- general/overlay/usr/sbin/fake-hwclock[71-84]
- general/overlay/usr/sbin/fake-hwclock[118-131]
- general/overlay/usr/sbin/sysupgrade[474-487]
- general/overlay/usr/sbin/sysupgrade[511-517]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Save lock outlives operation ✓ Resolved 🐞 Bug ☼ Reliability
Description
do_save() acquires flock on an exec-opened descriptor but never closes it, so the long-lived
daemon retains the lock throughout its next 60-second sleep. NTP-triggered checkpoints can therefore
block for nearly a minute instead of persisting the first valid sync immediately, extending the
hard-power-loss window this feature is intended to close.
Code

general/overlay/usr/sbin/fake-hwclock[R72-73]

+	exec 9> "$LOCK"
+	flock 9
Evidence
The save function returns without closing FD 9, while the same shell remains alive and sleeps
between saves; NTP hooks invoke a separate process that must acquire that retained lock.

general/overlay/usr/sbin/fake-hwclock[71-86]
general/overlay/usr/sbin/fake-hwclock[118-131]
general/overlay/usr/sbin/fake-hwclock[134-147]
general/overlay/usr/sbin/ntpd-script[1-5]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The daemon retains FD 9 and its flock after `do_save` returns, delaying concurrent NTP checkpoints.
## Issue Context
Shell functions run in the current shell, and `exec 9>` persists until the descriptor is explicitly closed, replaced, or the daemon exits.
## Fix Focus Areas
- general/overlay/usr/sbin/fake-hwclock[71-86]
- general/overlay/usr/sbin/fake-hwclock[118-131]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Daemon can escape stop ✓ Resolved 🐞 Bug ☼ Reliability
Description
start() backgrounds the daemon and returns before the child creates its pidfile, while stop()
only kills a process found through that file. A stop issued in this window misses the child, which
can then create the pidfile and continue running after stop or alongside a subsequently started
second daemon.
Code

general/overlay/etc/init.d/S02fakehwclock[R28-29]

+	"$DAEMON" daemon &
+	echo "OK"
Evidence
The init parent backgrounds the child without waiting, the child writes the pidfile only after it is
scheduled, and stop skips kill when that file is absent.

general/overlay/etc/init.d/S02fakehwclock[16-39]
general/overlay/usr/sbin/fake-hwclock[118-131]
general/overlay/etc/init.d/rcK[3-9]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The init script reports startup before the daemon has published its pidfile, allowing an immediate stop to miss the process.
## Issue Context
`running()` and `stop()` rely entirely on the pidfile written asynchronously inside `do_daemon`.
## Fix Focus Areas
- general/overlay/etc/init.d/S02fakehwclock[16-39]
- general/overlay/usr/sbin/fake-hwclock[118-131]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Tip of the day
💡 Did you know, you can enable the Remediation agent and Qodo fixes findings in a dedicated fix PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread general/overlay/usr/sbin/fake-hwclock Outdated
Comment thread general/overlay/etc/init.d/S02fakehwclock
Comment thread general/overlay/etc/init.d/S02fakehwclock
…p for sysupgrade

Three findings from the review of #2336, each verified on the
hi3516av300 before and after:

The save lock was taken on an exec-opened descriptor, which the daemon
then held across its whole next sleep - an ntpd first-sync checkpoint
could stall behind it for a minute, exactly the write the hook exists
to persist promptly. The lock now lives in a subshell and dies with
the write; a hook save alongside a running daemon returns in under a
second where it previously blocked.

start() returned before the daemon had written its pidfile, so a stop
issued in that window found nothing to kill and the daemon survived
it. The parent now writes the same pid itself before reporting OK.

sysupgrade stopped syslog, klogd, ntpd and cron before the flash but
would have left this daemon running - a live overlay writer during the
very erase quiesce_overlay() fences off. It is now on the stop list
(its stop doubles as a final checkpoint, so the camera returns from
the post-flash reboot with the freshest time it can have) and on the
restore list for upgrades that abort before flashing.
@openipc-ai
openipc-ai merged commit 689b37a into master Aug 30, 2026
114 checks passed
@openipc-ai
openipc-ai deleted the overlay/fake-hwclock branch August 30, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant