Watchdog Reset Caused by Slow SD Card or Removal (Possibly Intermittent connection) During Flight

[aero11223344]

## Bug report
When the SD card becomes slow or stops logging during flight, or on ground, removed and reinserted while the vehicle is armed, the system reports:

AP_Logger: Stuck Thread (Write)
Internal Error: 0x800

Following these errors, the CPU load increases sharply (~70% above) triggering a watchdog reset. This can be seen in the log as:

WDG: T6 SL0 FL0 FT0 FA0 FTP0 FLR0 FICSR0 MM0 MC0 I

This causes the flight controller to reboot, which results in a loss of control and a crash if the aircraft is airborne.

The attached log is a sample of what has been seen. Here the bug is simulated by frequently inserting and removing the card and the error msg with watchdog reset activity appears.

Version
4.0.5, 4.4.4, 4.5.5, 4.6.2

Platform
[ ] Copter

Airframe type
hexarotor

Hardware type
Pixhawk4

Logs
https://drive.google.com/file/d/1rkvh2qtm10GrCSpO920fzp4OTniBobGF/view?usp=drive_link

[yuri-rage]

@aero11223344 please remove the access restriction on your Google Drive link.

[aero11223344]

I have lifted the access restriction, so the log file is now accessible to everyone.

[tpwrules]

Original discuss thread: https://discuss.ardupilot.org/t/pixhawk4-cpu-overload/138782/23

[tpwrules]

I have a Pixhawk 4 Mini and a Pixhawk 5X now and I am unable to replicate this issue on either.

[tpwrules]

I take that back, I can replicate this on the PH4-Mini using its OEM SD card. I had to use a UC-867 SD card extender so I could wiggle the cable but it does happen pretty reliably even on 4.6.3 and master. I couldn't replicate it just popping the card in and out repeatedly.

I also used the LOG_* settings from your log file (in case they mattered), disabled the crash check and arming checks and RC failsafe and auto disarms so I could test armed on the bench.

[tpwrules]

I can replicate this easily on Cube Black but was unable to on Cube Orange. I suspect without a lot of proof that this does not affect H7 autopilots (a lot of the driver code and hardware is different vs. F4/F7), making it merely a really bad problem instead of a mega critical one simply due to the distribution of users.

I do note I was able to make the Cube Orange crash by shorting out the SD power lines (I suspect), but aside from this I was only ever able to trigger thread hangs and the vehicle continued operating normally. I was never able to trigger a true watchdog reset. I did test that the mavproxy debug function worked to trigger a watchdog reset so I don't think they are going undetected.

Thanks for your patience in the time it took to replicate. It will take some further time to attach a debugger to my PH4-Mini and investigate the root cause.

[tpwrules]

Tracked this down to a bug in the ChibiOS SD card driver for F4/F7. Confirmed that bug can't affect H7.

Working with ChibiOS on a solution: https://forum.chibios.org/viewtopic.php?f=35&t=6616

[tridge]

fixed in #31620