Support for scripting in flash using littlefs #28724

andyp1per · 2024-11-24T20:23:52Z

This completely replaces the flash based support, including logging with a littlefs backend. This results in much more reliable FS access as well as support for things like scripting.

With thanks to @ntamas for the original code

tpwrules · 2024-11-25T04:46:15Z

Oh excellent!! Have you tried with both NOR and NAND?

andyp1per · 2024-11-25T10:16:19Z

Oh excellent!! Have you tried with both NOR and NAND?

No, still a lot of work to do as there is lots of duplication currently. The code is from @ntamas but needs a bunch of rework to get into master. I have a plan, but the main thing is that it now works for me locally and I know how to proceed from here.

ntamas · 2024-11-25T13:32:41Z

Let me know if I can help with reworking this PR for inclusion. This is something that we at @skybrush-io would also be interested in.

jpkh · 2024-11-26T11:18:05Z

Oh yes, there are a lot small H743 based FPVDrone FCs that only have dataflash. This will be important update to all those (and others). Thanks Andy.

robustini · 2024-11-26T13:13:54Z

Thanks Andy, this will be most useful!

andyp1per · 2024-11-27T22:02:45Z

@ntamas actually it's not so bad - I think what you have done is correct. I really want the block logger to die, but I don't think that's going to be possible on boards with very small flash chips

tridge · 2024-11-29T21:33:23Z

it will be very interesting to see how this goes! and some stats on storage overhead (and flash cost) would be useful

tridge · 2024-11-29T21:34:55Z

@andyp1per assuming this works out and testing is good, we should fork littlefs so we point to our own github org, we've tended to regret it when we don't do that

andyp1per · 2024-11-29T21:50:20Z

Working quite nicely for me now. Not yet flown it.

andyp1per · 2024-12-01T12:37:06Z

Unfortunately I have yet to be convinced that the architecture of littlefs is suitable for logging, there are a lot of unaddressed issues which particularly bit hard.

ntamas · 2024-12-02T10:07:28Z

Are you experiencing performance issues with LittleFS? If so, how about splitting the flash memory into two partitions, one for logging with the old block logger, and one for storing files (scripts etc) that are assumed to be mostly static?

andyp1per · 2024-12-02T16:46:34Z

Are you experiencing performance issues with LittleFS? If so, how about splitting the flash memory into two partitions, one for logging with the old block logger, and one for storing files (scripts etc) that are assumed to be mostly static?

That's obviously the fallback, but at an even higher flash cost and would require modifying either lfs or the block logger to cope with some kind of partitioning scheme. I'm going to see what is achievable with LFS first before going that route.

andyp1per · 2024-12-07T00:00:56Z

We don't have a block logger based build to check the size against

andyp1per · 2024-12-07T14:46:42Z

Actually this is looking pretty promising

tridge

it would be good to know what the flash cost is - is it comparable to FATFS/sdcard? (likely it is)
worth thinking about supporting this in SITL with a command line option to enable instead of normal file system support, like we do for the logger block support in SITL

libraries/AP_Logger/AP_Logger_File.cpp

optimize free space when using littlefs

use correct read status for nor flash implement format on littlefs optimize device calls in littlefs flash usage check for fileops allowed in littlefs littlefs optimization and support for mtime

…using littlefs

andyp1per · 2024-12-12T21:01:00Z

I've added SITL support, but can't get it to work. It seems there are some issues with the filebd provider and littlefs provides no working examples. There is an open issue asking how to make it work. A project for someone else in the future maybe :)

tpwrules · 2024-12-13T05:44:06Z

.gitmodules

+[submodule "modules/littlefs"]
+	path = modules/littlefs
+	url = https://github.com/ArduPilot/littlefs.git


Would much prefer to just vendor the two relevant files. Git submodules are always a pain and adding another means constantly making sure it's updated or removed when switching branches...

tpwrules

Really excited to see this; sorry for not properly testing yet. Glad my vague memories of littlefs were helpful! Some miscellaneous notes.

Is this meant to be mutually exclusive with FATFS support? Could a board have both NAND/NOR and SD?

Also a new littlefs version 2.10 just came out.

tpwrules · 2024-12-13T05:44:41Z

libraries/AP_HAL_ChibiOS/hwdef/scripts/defaults_normal.h

 #ifndef AP_TERRAIN_AVAILABLE
 // enable terrain only if there's an SD card available:
-#define AP_TERRAIN_AVAILABLE HAL_OS_FATFS_IO
+#define AP_TERRAIN_AVAILABLE (HAL_OS_FATFS_IO || (HAL_OS_LITTLEFS_IO && (BOARD_FLASH_SIZE>1024)))


Is LittleFS big enough for a meaningful amount? Is it okay with the terrain access patterns? Is there a need to consume the flash before we are sure it is useful?

tpwrules · 2024-12-13T05:45:51Z

libraries/AP_Logger/AP_Logger.cpp

@@ -56,6 +56,14 @@ extern const AP_HAL::HAL& hal;
 #define HAL_LOGGER_ARM_PERSIST 15
 #endif

+#ifndef HAL_LOGGER_MIN_MB_FREE
+#if AP_FILESYSTEM_LITTLEFS_ENABLED
+#define HAL_LOGGER_MIN_MB_FREE 1


Probably needs to be bigger so that a meaningful amount can be logged. Also to match the parameter range annotation.

tpwrules · 2024-12-13T05:47:01Z

libraries/AP_Logger/AP_Logger_File.cpp

@@ -980,6 +985,11 @@ void AP_Logger_File::io_timer(void)
        write_fd_semaphore.give();
        return;
    }
+
+#if AP_FILESYSTEM_LITTLEFS_ENABLED && CONFIG_HAL_BOARD != HAL_BOARD_SITL
+    bool sync_block = AP::littlefs().sync_block(_write_fd, _write_offset, nbytes);


Could be done later, but it would be nicer to have a bytes_until_sync method on every filesystem, coupled with some logic to sync after a minimum number of bytes logged.

I would prefer not to add another singleton.

tpwrules · 2024-12-13T05:47:20Z

libraries/AP_Logger/AP_Logger_File.cpp

-          the best strategy for minimizing corruption on microSD cards
-          seems to be to write in 4k chunks and fsync the file on each
-          chunk, ensuring the directory entry is updated after each
-          write.
+          fsync on littlefs is extremely expensive (20% CPU on an H7) and not
+          required since the whole point of the filesystem is to avoid corruption


Needs updating/combining.

tpwrules · 2024-12-13T05:47:45Z

libraries/AP_Logger/AP_Logger_File.h

@@ -125,7 +129,11 @@ class AP_Logger_File : public AP_Logger_Backend
    // data and failures-to-boot.
    uint32_t _free_space_last_check_time; // milliseconds
    const uint32_t _free_space_check_interval = 1000UL; // milliseconds
+#if AP_FILESYSTEM_LITTLEFS_ENABLED
+    const uint32_t _free_space_min_avail = 4096; // bytes


Probably need more on NAND.

andyp1per added the WIP label Nov 24, 2024

andyp1per marked this pull request as ready for review November 29, 2024 21:49

andyp1per removed the WIP label Nov 29, 2024

andyp1per added the NeedsTesting label Nov 29, 2024

andyp1per force-pushed the pr-littlefs-flash-v3 branch 3 times, most recently from 01c281d to 424f673 Compare November 30, 2024 10:25

andyp1per added the WIP label Dec 1, 2024

andyp1per force-pushed the pr-littlefs-flash-v3 branch from bc0fd66 to 92f9c58 Compare December 7, 2024 14:45

andyp1per added Enhancement and removed WIP labels Dec 7, 2024

andyp1per force-pushed the pr-littlefs-flash-v3 branch from 6c40ab4 to ed2df88 Compare December 7, 2024 20:59

andyp1per added DevCallEU and removed NeedsTesting labels Dec 8, 2024

tridge reviewed Dec 11, 2024

View reviewed changes

libraries/AP_Logger/AP_Logger_File.cpp Outdated Show resolved Hide resolved

andyp1per added 22 commits December 12, 2024 09:53

modules: add littlefs v2.9.3

9bbbfff

waf: build in littlefs support

c43a646

wscript: build in littlefs

9b936f9

Replay: account for littlefs when building

9942d3f

AP_Bootloader: correct use usage

c17c9c8

AP_Periph: correct use usage

2c8eccb

AP_Logger: sync littlefs writes on block boundaries

7f9ac1f

optimize free space when using littlefs

AP_HAL_ChibiOS: littlefs for MambaH743v4

2def2fc

AP_HAL_ChibiOS: littlefs for MatekH7A3

36d02aa

AP_HAL_ChibiOS: use littlefs on KakuteH7Mini-Nand

9cf53f3

AP_Scripting: allow scripting on 2Mb boards with littlefs

f0947d7

AP_Filesystem: add littlefs based filesystem support

5182f73

use correct read status for nor flash implement format on littlefs optimize device calls in littlefs flash usage check for fileops allowed in littlefs littlefs optimization and support for mtime

AP_Filesystem: reduce metadata_max on W25N flash

7299c8a

AP_Logger: add flash speed test

4016b7b

AP_Logger: add performance debug to W25NXX logger

a16d574

AP_Filesystem: add littlefs singleton for sync block use

c4a9d54

AP_Logger: user sync_block from littlefs to decide when to sync when …

0f5842f

…using littlefs

AP_HAL_ChibiOS: terrain turned on on 2Mb boards with littlefs

706006f

AP_HAL_ChibiOS: turn off storage backup on anything other than FATFS

1c5101e

AP_HAL_ChibiOS: littlefs support for Aocoda-RC-H743Dual

cbb574a

AP_HAL_ChibiOS: littlefs support for JHEMCU-H743HD

c35762b

AP_HAL_ChibiOS: littlefs support for KakuteH7v2

ea198c4

andyp1per force-pushed the pr-littlefs-flash-v3 branch from 99fcb68 to ea198c4 Compare December 12, 2024 09:53

andyp1per added 4 commits December 12, 2024 20:55

modules: update littlefs build to v2.9.3-ardupilot

29dca7c

AP_Filesystem: provide SITL implementation for littlefs

9401c64

AP_Logger: support SITL implementation of littlefs

2e32c65

waf: build littlefs filebd

8735826

tpwrules reviewed Dec 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for scripting in flash using littlefs #28724

Support for scripting in flash using littlefs #28724

andyp1per commented Nov 24, 2024 •

edited

Loading

tpwrules commented Nov 25, 2024

andyp1per commented Nov 25, 2024

ntamas commented Nov 25, 2024

jpkh commented Nov 26, 2024

robustini commented Nov 26, 2024

andyp1per commented Nov 27, 2024

tridge commented Nov 29, 2024

tridge commented Nov 29, 2024

andyp1per commented Nov 29, 2024

andyp1per commented Dec 1, 2024

ntamas commented Dec 2, 2024

andyp1per commented Dec 2, 2024

andyp1per commented Dec 7, 2024

andyp1per commented Dec 7, 2024

tridge left a comment

andyp1per commented Dec 12, 2024

tpwrules Dec 13, 2024 •

edited

Loading

tpwrules left a comment •

edited

Loading

tpwrules Dec 13, 2024 •

edited

Loading

tpwrules Dec 13, 2024

tpwrules Dec 13, 2024

tpwrules Dec 13, 2024

tpwrules Dec 13, 2024

Support for scripting in flash using littlefs #28724

Are you sure you want to change the base?

Support for scripting in flash using littlefs #28724

Conversation

andyp1per commented Nov 24, 2024 • edited Loading

tpwrules commented Nov 25, 2024

andyp1per commented Nov 25, 2024

ntamas commented Nov 25, 2024

jpkh commented Nov 26, 2024

robustini commented Nov 26, 2024

andyp1per commented Nov 27, 2024

tridge commented Nov 29, 2024

tridge commented Nov 29, 2024

andyp1per commented Nov 29, 2024

andyp1per commented Dec 1, 2024

ntamas commented Dec 2, 2024

andyp1per commented Dec 2, 2024

andyp1per commented Dec 7, 2024

andyp1per commented Dec 7, 2024

tridge left a comment

Choose a reason for hiding this comment

andyp1per commented Dec 12, 2024

tpwrules Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

tpwrules left a comment • edited Loading

Choose a reason for hiding this comment

tpwrules Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

tpwrules Dec 13, 2024

Choose a reason for hiding this comment

tpwrules Dec 13, 2024

Choose a reason for hiding this comment

tpwrules Dec 13, 2024

Choose a reason for hiding this comment

tpwrules Dec 13, 2024

Choose a reason for hiding this comment

andyp1per commented Nov 24, 2024 •

edited

Loading

tpwrules Dec 13, 2024 •

edited

Loading

tpwrules left a comment •

edited

Loading

tpwrules Dec 13, 2024 •

edited

Loading