sysupgrade -n: camera can hang instead of rebooting after the overlay wipe, needing a physical power cycle

[widgetii]

sysupgrade -n completes and announces its reboot, but the camera sometimes never comes back — only a physical power cycle recovers it. Seen repeatedly; opening this to pin the mechanism down rather than to report a single event.

Distinct from the page-cache hazard fixed by the ramfs pivot in #2251: that one is about rewriting the rootfs the camera runs from, and it is excluded here — the run below pivoted into RAM successfully (Flashing from RAM (pid 1041)) and never touched mtd3. The only partition written was rootfs_data.

Environment

SoC	hi3516ev300 (NOR, 16 MB)
Build	master+2f558d4, 2026-08-21, BUILD_PLATFORM=hi3516ev300_lite, OPENIPC_VERSION=2.6.08.21
sysupgrade	v1.0.60
Overlay	mtd4 "rootfs_data", 008b0000, erasesize 00010000, jffs2 on /dev/mtdblock4 mounted at /overlay

Steps to reproduce

/usr/sbin/sysupgrade -n --web

The WebUI's Reset firmware button runs the same thing, so it reproduces from the browser too. --web is incidental — it only skips the killall -3 majestic in free_resources().

What happens

The run completes normally end to end, erase reaches 100%, and sysupgrade prints its reboot line:

OpenIPC System Updater v1.0.60

Vendor	hisilicon
SoC	hi3516ev300
Model	hi3516ev300
Kernel	22:58:06 2026-08-21
RootFS	master+2f558d4, 2026-08-21

Synchronizing time
Sun Aug 23 06:16:23 UTC 2026

Stop services, sync files, free up memory
Stopping crond: OK
Stopping ntpd: OK
Stopping klogd: OK
Stopping syslogd: OK

Uptime:
 06:16:24 up 22:40,  load average: 0.22, 0.17, 0.10

Memory:
              total        used        free      shared  buff/cache   available
Mem:         120.7M       53.7M       61.8M       20.0K        5.1M       64.5M
Swap:             0           0           0

Processes:
PID   USER     TIME  COMMAND
    1 root      0:01 init
  694 root      0:00 udhcpc -x hostname:hi3516ev300-imx335 -A 0 -T 1 -t 5 -R -b -O search -p /var/run/udhcpc.eth0.pid -i eth0
  705 root      0:00 dropbear -R -B -k -p 22 -K 300
  847 root      1h19 majestic -s
  854 root      0:00 /sbin/getty -L console 0 vt100
 1039 root      0:00 dropbear -R -B -k -p 22 -K 300
 1040 root      0:00 sh -c /usr/sbin/sysupgrade -n --web 2>&1 | cat
 1041 root      0:00 {sysupgrade} /bin/sh /usr/sbin/sysupgrade -n --web
 1042 root      0:00 cat
 1093 root      0:00 ps

Unmounting SD card

Protected: flashing continues even if this terminal disconnects.

Moving the flash phase into RAM
Flashing from RAM (pid 1041)

OverlayFS
Erase overlay partition
Erasing 64 Kibyte @ 8b0000 - 100% complete.Cleanmarker written at 8a0000.

Unconditional reboot

Then nothing. No ICMP, no :80, no :22 for ~45 minutes. A manual power cycle brought it straight back, healthy, with a correctly wiped overlay — so the flash contents were fine and the reset itself did what it was asked. It is the reboot that did not take.

Timing, from the timestamped capture: the whole run is 34 s. The erase is 139 blocks at ~215 ms each (~30 s of the 34), and the last byte received is Unconditional reboot.

What this rules out

• Not the sysupgrade: flash from a ramfs, so the reboot cannot be eaten by the flash #2251 page-cache hazard. The pivot into RAM succeeded, so reboot was busybox running from tmpfs, not from a rewritten partition. mtd3 was never written.
• Not a bad erase or a corrupt overlay. After the power cycle the camera booted clean on a freshly populated overlay.
• Not board-specific to goke. Reported against gk7205v300 in majestic-webui#154 and reproduced here on hi3516ev300 — though on gk7205v300 the same command did reboot normally, so it is intermittent rather than universal.

Leading hypothesis: the overlay is erased while it is still mounted

do_wipe_overlay() erases rootfs_data and nothing unmounts or remounts it first:

do_wipe_overlay() {
	echo_c 33 "\nOverlayFS"
	echo "Erase overlay partition"
	[ "$flash_type" = "nand" ] || jffs2="-j"
	mark_live_flash_dirty
	set_progress flash_eraseall $jffs2 "$(get_device "rootfs_data")"
}

At that point /dev/mtdblock4 is still a live, read-write jffs2, and pivot_root moved it to /mnt rather than releasing it — so after the erase the in-kernel jffs2 state describes a medium that no longer exists. busybox reboot -f calls reboot(RB_AUTOBOOT) straight into kernel_restart(), which runs device_shutdown() before machine_restart(). A jffs2 GC thread or a pending writeback stuck on erased blocks would block exactly there, which fits the symptom: the announcement is printed, nothing further happens, and only cutting power gets the SoC to reset.

That would also explain the intermittency — whether jffs2 has dirty state or GC in flight at erase time depends on what was written to the overlay just before.

Worth considering alongside it:

• free_resources() stops the watchdog service on boards that have one, so nothing is left armed to rescue a hung reset. If the reboot path can hang at all, disarming the watchdog first removes the one thing that would have papered over it.
• With --web, majestic is deliberately left running across the erase. Anything of its still open on the overlay disappears mid-erase.

What would settle it

A serial console capture across the next reproduction — that is the only place a hang between Unconditional reboot and machine_restart() is visible. Failing that, echo b > /proc/sysrq-trigger as an alternative reboot primitive would tell us whether the block is in device_shutdown() (sysrq-b skips it) or lower down.

If the hypothesis holds, the fix is presumably to get the overlay out of the way before erasing it — mount -o remount,ro or a real unmount of /mnt/overlay post-pivot — rather than to make the reboot more forceful.

Found while working on majestic-webui#154 (OpenIPC/majestic-webui#154).

[widgetii]

Reproduced a second time on the same hi3516ev300, ~40 minutes after the first — 2 for 2 on this board.

This run went through the WebUI's Reset firmware button rather than a shell, driven from a real browser, so it is the user-facing path rather than my SSH invocation. Same outcome: the erase completes, Unconditional reboot is printed, and then no ICMP / :80 / :22. Still down after 5 minutes; recovering it needs a power cycle again.

Browser-side timeline, from the streamed log:

[ 3.8s] Synchronizing time
[ 5.3s] Stopping syslogd: OK
[ 6.8s] Moving the flash phase into RAM
[ 8.3s] Erasing 64 Kibyte @ 50000 -   3% complete.
        ...
[36.9s] Erasing 64 Kibyte @ 870000 -  97% complete.
[38.4s] Unconditional reboot
        <nothing further; camera gone>

So the erase takes ~31 s of a ~38 s run and finishes cleanly, and the hang is after the last line sysupgrade prints. Consistent with the first capture in the report above.

Both runs were on a camera that had been up for a while (22 h the first time, ~10 min the second), i.e. an overlay with real write history behind it in one case and almost none in the other — so "how dirty the jffs2 was" does not obviously separate them, which weakens the GC-state explanation for the intermittency even if the erase-while-mounted hypothesis still stands for the hang itself.

Next reproduction should be on a board with serial attached; without console output there is no way to tell a kernel that never reached machine_restart() from one that reached it and the SoC did not reset.

[widgetii]

Caveat on the two reproductions above — the board they were taken on has since proved unreliable in a way that has nothing to do with sysupgrade, so treat both as unconfirmed until someone reproduces on different hardware.

What happened after the second report:

1. A plain reboot with nothing written to flash (a shell script that prints the same markers and then reboot -d 1, run through the same WebUI streaming path) rebooted normally and the camera was serving again in ~54 s. So this board can reboot.
2. About a minute after that successful boot it became unreachable again — with no reboot command issued and nothing running against it.
3. After a power cycle it came up (up 0 min, HTTP and SSH both answering), and became unreachable again shortly afterwards, again with nothing touching it.

So this camera currently drops off on its own. That is enough to make "erase the overlay, then hang" the wrong conclusion to draw from runs 1 and 2: the same board would have looked identical if it had simply died mid-boot for an unrelated reason.

What still stands on its own evidence:

• The transcripts are real — the erase completes and Unconditional reboot is the last line printed. That much is not in doubt.
• do_wipe_overlay() really does flash_eraseall a mounted, read-write jffs2, and pivot_root only moved it to /mnt. Whether or not it explains this hang, it looks worth fixing on its own terms.

What it needs: a reproduction on a second board, ideally with serial attached. I would not act on the hypothesis in the original report until that exists. Happy to keep this open as the place to collect further sightings — the original reporter hit "reset, then the camera came back but the UI did not follow" on gk7205v300, which is a different symptom (their camera did come back), so that one is not evidence for this either.

[widgetii]

Reproduced on a lab hi3516ev300 with the serial console attached, which is what this needed. The leading hypothesis in the report is wrong: nothing hangs on the way down. The reboot completes, U-Boot runs normally, and it is the next boot that dies — because U-Boot loads a kernel image that has a bit flipped in it.

Rig: lab hi3516ev300, 16 MB NOR, same partition layout as the report, master+2f558d4, 2026-08-21, sysupgrade v1.0.60, U-Boot 2016.11-g6d2ed0c, NOR XT25F128BSSI/HGU. UART on the console, PoE power-cycle for recovery, everything driven in loops so the numbers below are counted rather than remembered.

what was run	boots	left dead, needed a power cycle
sysupgrade -n --web	23	3
reboot -d 1 -f, nothing written to flash	18	0

So it reproduces, at roughly 13%, and it is specific to the run that writes flash.

The reboot is not what hangs

jffs2: Node totlen on flash (0xffffffff) != totlen from node ref (0x0000002c)
reboot: Restarting system            <- kernel_restart() completed
System startup                       <- U-Boot
  SPI Nor ... Name:"XT25F128BSSI/HGU"
  SF: 3145728 bytes @ 0x50000 Read: OK
  ## Booting kernel from Legacy Image at 42000000 ...
     Image Name:   Linux-4.9.37-hi3516ev300
     Loading Kernel Image ... OK
  Starting kernel ...
<nothing, ever, until the power cycle>

device_shutdown() is not where it stops — reboot: Restarting system is printed after it. U-Boot is fine. The kernel never emits its first printk.

That last part is the tell: on a good boot there is ~1.1 s of silence between Starting kernel ... and Booting Linux on physical CPU 0x0 while the payload self-decompresses, and it stays silent even with earlycon. A payload that fails to decompress lands in error() in arch/arm/boot/compressed/misc.c, which is an infinite loop and prints nothing without CONFIG_DEBUG_LL. Dead silence and a mandatory power cycle is exactly its signature.

Why every report so far ended at "nothing further"

The kernel console is muted on these boards. /etc/sysctl.conf sets kernel.printk = 3 3 1 3 and S02sysctl applies it, but the live value is not that:

board	live /proc/sys/kernel/printk
hi3516ev300	0 0 0 0
hi3516cv300	0 0 0 0
gk7205v200	0 0 0 0
ssc30kq	3 3 1 3

console_loglevel=0 suppresses even KERN_EMERG, so an oops, a panic and reboot: Restarting system all produce nothing on the console. (/usr/bin/majestic contains the string /proc/sys/kernel/printk; the three boards that are muted are the hisilicon/goke ones.) Every capture in this issue was taken through a console that could not have shown a kernel fault. The absence of output was never evidence of a hang — and any future diagnosis here needs ignore_loglevel on the cmdline.

What is actually corrupt

bootcmd does one sf read of 3 MB into 0x42000000 and boots it. I made it read the same flash region into two extra addresses and byte-compare them:

sf read 0x40800000 0x50000 0x300000     <- reference 1
sf read 0x41000000 0x50000 0x300000     <- reference 2
sf read 0x42000000 0x50000 0x300000     <- the copy that gets booted
cmp.b ...

The three copies of the same static flash region do not always agree:

byte at 0x4223ab00 (0x69) != byte at 0x40a3ab00 (0x68)
byte at 0x421c0f00 (0x41) != byte at 0x409c0f00 (0x40)
byte at 0x42264200 (0x71) != byte at 0x40a64200 (0x70)
byte at 0x409dca00 (0x59) != byte at 0x411dca00 (0x58)

Always a single bit, always bit 0, always at a 256-byte-aligned offset, at a different offset each time, and the odd copy out is sometimes the booted one and sometimes a reference. Of 11 instrumented sysupgrade boots, 8 had at least one flipped bit somewhere in the 3 MB; of 4 instrumented plain reboots, none did.

Most of those flips land in the ~1 MB of rootfs that the oversized sf read picks up past the end of the uImage, and are harmless. Here is the one that did not — caught in the act, with the compare scoped to the 2 MB payload:

=== ITER 6 ===
reboot: Restarting system
System startup
PAYLOAD-R1R2
Total of 2101248 byte(s) were the same          <- the two references agree
PAYLOAD-VR1
byte at 0x421c0f00 (0x41) != byte at 0x409c0f00 (0x40)   <- the booted copy does not
Starting kernel ...
<dead — power cycled 30 s later>

Every dead boot in this investigation had a corrupt payload; no boot with a clean payload died.

Confirming it, rather than inferring it

At the U-Boot prompt: load the image, flip exactly one byte in the payload, boot it.

=> md.b 0x421c0f00 4
421c0f00: 40 e2 e7 85    @...
=> mw.b 0x421c0f00 0xff 1
=> setenv verify yes
=> bootm 0x42000000
   Verifying Checksum ... Bad Data CRC
ERROR: can't get kernel image!

One byte is enough, and bootm can see it.

Why it is silent and fatal instead of a retry

It is not seen, because the uImage data CRC is never checked:

(Linux)   # fw_printenv verify
          verify=yes
(U-Boot)  => printenv verify
          verify=n

U-Boot overrides the saved environment and boots with verify=n, so a flipped bit is executed rather than caught. Confirmed twice, including after writing verify from U-Boot itself and saveenv-ing it — the value comes back as n and no Verifying Checksum line ever appears on an autoboot. Only a setenv verify yes typed into the live session takes effect, which is why the manual test above works. verify=n is in the shipped environment on hi3516cv300, gk7205v200 and hi3516av300 too.

This is the part that turns an intermittent read error into a brick. With verification on, bootcmd already ends in reset, so a bad read would simply reboot and try again. The check costs ~50 ms on 2 MB — the same-sized cmp.b in these traces takes 48 ms. Worth noting the other three boards' bootcmd has no trailing ; reset, so they would stop at the prompt instead of retrying.

What triggers the bad reads, and what does not

• The NOR erase does. flash_eraseall -j /dev/mtd4 followed by a reboot — no pivot, no service teardown, majestic untouched — reproduces the flips (2 of 6 boots). So this is not about the ramfs pivot, not about --web leaving majestic running, and not about the overlay being mounted.
• Waiting does not help. Erase, sleep 30 s, then reboot: still 2 of 4 boots (so it is not a short thermal transient that settles).
• It is confined to the first seconds after the reset. Sitting at the U-Boot prompt after the same erase-and-reboot, 36 read-and-compare pairs (~108 MB) into the same addresses were error-free — at the default SPI clock and again at 24 MHz.
• The flash contents are fine. From Linux, 12 full reads of mtd2 and 12 of mtd3 give one md5 each. hi3516cv300, gk7205v200 and hi3516av300 are equally stable (10 reads each). This is not a worn-out chip and not a bad block.

So: a NOR erase leaves something unsettled across the SoC reset, U-Boot's first reads after that reset are occasionally wrong by one bit, and nothing checks them.

Where this leaves the original hypothesis

do_wipe_overlay() really does flash_eraseall a mounted, read-write jffs2, and the kernel says so out loud:

jffs2: Node totlen on flash (0xffffffff) != totlen from node ref (0x0000002c)

That is worth fixing on its own terms. It is just not what wedges the camera — the kernel carries that damage all the way to machine_restart() without complaint.

Suggested next steps

1. Make U-Boot verify the uImage before booting it. This is the fix that matters: it converts a silent unbootable board into an automatic retry, for ~50 ms. It has to happen in the U-Boot build, since the environment is overridden. Boards whose bootcmd lacks a trailing ; reset should get one at the same time.
2. Chase the read errors themselves in the vendor hifmc100 U-Boot driver — single-bit, bit 0, 256-byte aligned, only in the seconds after a reset that followed an erase. Reducing the probe speed to 24 MHz did not reproduce them at the prompt, so that experiment needs redoing during autoboot rather than interactively.
3. Unmount or remount-ro the overlay before erasing it, independently of the above.
4. Stop muting the kernel console, or at least leave KERN_EMERG through. Right now a camera that panics looks exactly like a camera that hung, on every hisilicon board we ship.

Caveat

All of this is one board. The verify=n finding is a property of the firmware and applies everywhere; whether the read errors are this unit's NOR interface or general to hi3516ev300 is not established here, and needs a second board with a serial console and the same instrumented bootcmd.

[widgetii]

Split the overlay-erase half of this out into #2300, with the mechanism nailed down (the jffs2 GC thread and flash_eraseall are both writing the medium — the kernel's own jffs2_block_check_erase() catches a cleanmarker in a block it had just erased) and a tested one-function patch: 3/4 runs produce live-kernel jffs2 errors today, 0/5 with the fix.

Worth stating explicitly since it came up: pivot_root is not at fault and does not need changing. It moves sysupgrade's root so the script stops executing from the partition it is about to overwrite (#2251); it cannot move anyone else's. The jffs2 is the upperdir/workdir of the overlayfs that is still root for init(1), majestic, dropbear and getty, so umount is EBUSY by construction and umount -l only hides it. mount -o remount,ro does work on it though, and jffs2_remount_fs() stops the GC thread on the way — which is all that is needed.

This issue stays about the part that actually bricks the camera: the unverified uImage read.