hi3516ev300 + Winbond W25N01GV: silent data-pattern UBIFS rootfs corruption (kernel corrupts its own writes; NAND HW proven good) — hifmc100 ECC bug

[widgetii]

Summary

On hi3516ev300 + Winbond W25N01GV SPI-NAND, the OpenIPC nand-ultimate build boots and mounts the UBIFS rootfs cleanly, but the rootfs silently corrupts over runtime. The failure is data-pattern-specific (always the same logical UBIFS block), silent (ecc_failures=0, corrected_bits=0), and — critically — reproduces even when the running kernel writes the rootfs itself and then re-reads it from flash. The NAND hardware is provably fine. This points to a bug in the kernel hifmc100 SPI-NAND ECC handling for the W25N01GV (classic on-die-ECC vs controller-ECC / OOB-layout problem), not to bad flash, a bad write, or a bootloader/kernel ECC-domain mismatch.

Environment

• SoC: HiSilicon hi3516ev300
• Board: Rostelecom IPC8232SWC-WE (sensor SP2305)
• Flash: Winbond W25N01GV — ID 0xef 0xaa 0x21, 128 MiB SLC, page 2048, OOB 64, controller ECC 4bit/512 ("ECC provided by Flash Memory Controller")
• Firmware: openipc.hi3516ev300-nand-ultimate.tgz, OpenIPC master+6a3cde2, kernel Linux 4.9.37
• U-Boot: 2016.11-g131d3f2 (universal), 2016.11-gbf8b847 (nand)

Symptom

UBIFS error (ubi0:0 pid N): ubifs_check_node: bad CRC: calculated 0x7d5d89cf, read 0x148ef575
UBIFS error (ubi0:0 pid N): ubifs_check_node: bad node at LEB 120:89296
        magic          0x6101831
        node_type      9 (indexing node)
UBIFS error (ubi0:0 pid N): ubifs_read_node: expected node type 9
UBIFS error (ubi0:0 pid N): ubifs_iget: failed to read inode M, error -117
UBIFS error (ubi0:0 pid N): ubifs_lookup: dead directory entry '<name>', error -117
UBIFS warning: ubifs_ro_mode: switched to read-only mode, error -117

• Because the corrupt node is a UBIFS index (TNC) node, a whole subtree of inodes becomes unreachable — the reported inode/filename varies, but the underlying bad node is always LEB 120 (offsets ~89248–101472 across runs).
• The rootfs volume is read-only, yet it corrupts — so this is not a stray RW write.
• cat /sys/class/mtd/mtd3/ecc_failures = 0, corrected_bits = 0 — the controller reports no ECC activity at all (this hisi driver does not appear to update MTD ECC stats).

Reproduction

1. Flash openipc.hi3516ev300-nand-ultimate onto a W25N01GV-equipped hi3516ev300 board.
2. Boot. Rootfs mounts R/O, majestic starts, RTSP/HTTP come up — appears fine.
3. Force the kernel to re-read the rootfs from flash (defeat the page cache and UBIFS TNC cache):

   for i in 1 2 3 4 5; do
       sync
       echo 3 > /proc/sys/vm/drop_caches
       find / -xdev -type f -exec cat {} + >/dev/null 2>&1
   done
   dmesg | grep -c 'UBIFS error'      # -> non-zero, e.g. 78
   dmesg | grep -c 'bad CRC'          # -> e.g. 15, all at "LEB 120"

   The bad-CRC UBIFS errors appear and the filesystem drops to read-only. On a fresh boot the first full read is often clean; the errors surface once cached znodes are evicted and the node is re-read from flash.

Decisive diagnostics (what this is NOT)

1. Not the NAND hardware. From U-Boot on the exact partition:

• Read the same physical block 15×: identical CRC every time (no read instability / read-disturb).
• 10× {erase → write pattern → read → verify} cycles: all pass.
• nand bad: no bad blocks.

2. Not the write method / not a bootloader-vs-kernel ECC domain mismatch. The rootfs was rewritten three different ways — all corrupt identically at LEB 120:

• nand write from the universal U-Boot (verified byte-perfect on U-Boot read-back).
• nand write from the nand U-Boot (verified byte-perfect on U-Boot read-back).
• From the running kernel itself: NFS-booted the kernel (init=/bin/sh, mtdparts=...), then flash_eraseall /dev/mtd3 + nandwrite -p /dev/mtd3 rootfs.ubi. Writer and reader are then the same code / same ECC — and it still corrupts on re-read (drop_caches test above).

So the kernel corrupts data it wrote itself, on re-read — this eliminates any u-boot-vs-kernel ECC/OOB-layout difference as the cause.

3. The two U-Boots do read the same flash differently (evidence of ECC-scheme sensitivity): reading the identical mtd3 contents, crc32 = 0529bf41 (universal) vs 8d080558 (nand). This is a side note; the primary result (2) is that the kernel breaks its own writes.

Suspected root cause

Data-pattern-dependent, silent, "same error at the same logical offset regardless of physical block" is the documented failure mode of the W25N01GV internal (on-die) ECC interacting badly with the controller ECC / OOB layout:

• linux-mtd: the W25N01GV internal ECC "seems to work properly only for 512B aligned writes"; with one specific data pattern, "read data differed from what was written… turning off ECC allowed the original data to be read correctly." (http://lists.infradead.org/pipermail/linux-mtd/2024-October/106485.html , https://igor-misic.blogspot.com/2020/12/winbond-nand-flash-w25n01gv-problem.html )
• General hisi hifmc guidance: having both on-die ECC and controller ECC active (or mismatched OOB layouts) causes exactly this; one ECC scheme must be chosen consistently.

The kernel driver intends to disable the chip's on-die ECC and use the FMC 4bit/512 ECC:

drivers/mtd/nand/hifmc100/hifmc_spi_nand_ids.c (W25N01GV entry id = {0xef,0xaa,0x21}, and):

/* Disable chip internal ECC */
reg = spi_nand_feature_op(spi, GET_OP, FEATURE_ADDR, 0);
if (reg & FEATURE_ECC_ENABLE) {
        reg &= ~FEATURE_ECC_ENABLE;
        spi_nand_feature_op(spi, SET_OP, FEATURE_ADDR, reg);
        ...
        if (reg & FEATURE_ECC_ENABLE)
                DB_MSG("Error: Chip internal ECC disable failed!\n");
}

Directions worth investigating (any one may be the fix):

• Confirm the on-die ECC is actually disabled at runtime on this W25N01GV revision (the disable is best-effort and the "disable failed" path is only a DB_MSG). If it silently stays enabled, on-die + FMC dual-ECC would produce exactly this corruption.
• OOB layout mismatch between what nandwrite/UBIFS stores and what the FMC ECC expects for this chip.
• Read speed: the W25N01GV table entry uses READ_QUAD(1, INFINITE, 104) (104 MHz quad read). If that mode is marginal on this board, sporadic silent read errors would result; a slower/standard read mode would confirm/deny.
• Consider using the chip's on-die ECC (mainline registers W25N01GV as NAND_ECCREQ(1, 512)) instead of the FMC ECC, i.e. the opposite ECC choice.

Impact

The nand-ultimate build is effectively unusable on W25N01GV hi3516ev300 cameras: it looks healthy after flashing and then silently rots the rootfs into read-only within normal runtime. No user-space reflash workaround exists (proven above — u-boot and kernel writers both fail). A kernel-side driver fix is required.

I'm happy to run targeted tests on the affected hardware (dump feature registers, try a patched read mode / ECC config, etc.) to help pin the fix.