SparkFun RTK Arduino Lib Builder

Based on Espressif's excellent Arduino Lib Builder, this repo compiles the customised ESP32 Arduino libraries needed by the SparkFun RTK Everywhere Firmware.

Background

SparkFun's RTK Everywhere Firmware runs on ESP32 and makes extensive use of Espressif's Arduino Libraries, including Bluetooth and WiFi. The standard pre-compiled libraries are excellent but the RTK Firmware needs customised versions with a few subtle changes. The changes are:

libmbedtls (libmbedcrypto)

libmbedtls, specifically libmbedcrypto, implements secure cryptographic communication on (e.g.) TCP/IP. It underpins secure communication with HTTPS.

libmbedcrypto uses a lot of memory, approximately 36KB. With the standard, pre-compiled Arduino library, this is allocated in RAM. That's a lot of RAM to lose. For the RTK Firmware, we need that to be allocated in PSRAM instead. The Espressif IDF makes this possible through CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC. But with the standard, pre-compiled Arduino library, this is hard coded to CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC. We need to re-compile libmbedcrypto using a customised version of esp_mem.c.

libbt

libbt contains the Bluetooth stack for ESP32. It is fantastic and works very well. To support Apple Accessory on Bluetooth Classic, we need L2CAP (Logical Link Control and Adaptation Protocol) and SDP (Service Discovery Protocol) to be enabled. The Espressif IDF makes this possible through CONFIG_BT_L2CAP_ENABLED and CONFIG_BT_SDP_COMMON_ENABLED. But with the standard, pre-compiled Arduino library, both are disabled. We need to re-compile libbt with L2CAP and SDP enabled, using a customised version of defconfig.esp32.

While we were adding support for SDP, we noticed that the advertised 128-bit iAP2 UUID byte order was reversed. The best fix we could find was to replace ARRAY_TO_BE_STREAM with ARRAY_TO_BE_STREAM_REVERSE in add_raw_sdp in btc_sdp.c. We include that change when we re-compile.

IDF Version

The RTK Firmware currently uses version v3.0.7 of the arduino-esp32 core. This is based on ESP-IDF v5.1 (specifically v5.1.4+, commit 632e0c2a). The later versions of the core, from v3.1.0 onwards, consume more RAM and we run into problems when we have Bluetooth Classic+BLE and WiFi enabled simultaneously. So, for now, we are sticking with v3.0.7 and IDF v5.1. The customised libraries compiled here are based on IDF v5.1.

Building

Espressif have fully documented the library build process. Everything here is based on that documentation and the ESP32 Arduino Lib Builder source code - for release/v5.1. We use Docker to compile the libraries on a virtual ubuntu machine. Espressif provide ready-made docker images which you can download and use directly to compile the libraries: docker image docs, docker images on docker hub. But here we prefer to use a custom version of the original Dockerfile so we can apply the patches described above and then build the modified libraries.

Building locally with Docker

Here is a checklist for how to compile the libraries locally. We tend to use Windows and here we provide a Windows batch (.bat) file to compile the libraries. If you are on Mac or Linux, you will need to change the batch file to (e.g.) a shell script. Or you can just run each line of the batch file manually, one at a time.

• Install Docker Desktop. There are versions for Windows, Mac and Linux
  • You don't need to create a docker account and you don't need to be signed in, but you may find it useful
• Ensure Docker Desktop is running
  • You don't need to be signed in, but you may find it useful
• On Windows, you may see an error saying "WSL needs updating Your version of Windows Subsystem for Linux (WSL) is too old". If you do:
  • Open a command prompt
  • Type wsl --update to update WSL. At the time of writing, this installs Windows Subsystem for Linux 2.6.1
  • Restart the Docker Desktop
• Download a copy of this repo and unzip it in (e.g.) your Documents folder
• Open a Command Prompt (cmd) and cd into the SparkFun_RTK_Arduino_Lib_Builder folder
• Run compile_with_docker.bat
  • The build is performed by the Dockerfile
• Go make a cup of tea. It takes a while...
• When the build is complete, you will find libbt.a and four libmbed.a files in your folder
• If you are compiling the RTK Everywhere Firmware locally, copy these five files into SparkFun_RTK_Everywhere_Firmware\Firmware\RTK_Everywhere\Patch
• If you want to take a look at the Lib Builder source files, you can do this by opening a bash shell in a container for the arduino_lib_builder image:
  • Ensure Docker Desktop is running
  • docker run --rm -it -e TERM=xterm-256color arduino_lib_builder:latest /bin/bash
  • (This replaces the CMD echo $(ls /*.*) at the end of the Dockerfile with CMD /bin/bash and opens a bash shell)
  • The source files are in opt/esp/lib-builder
  • exit to leave the bash shell

Building with GitHub Actions

This repo includes two Workflows which will run the same Dockerfile under a GitHub Action

non-release-build.yml will build the libraries and attach them to the Action as a Zip file Asset

build-and-release.yml will build the libraries and push them to the SparkFun_RTK_Everywhere_Firmware repo, in the Patch folder. This is for SparkFun use only. If you have cloned or forked this repo, this Action won't work for you - you won't have the correct permissions to push the libraries.

TODO

The build currently fails with the error:

/opt/esp/lib-builder/components/arduino/cores/esp32/esp32-hal-cpu.c:170: undefined reference to esp_timer_impl_update_apb_freq

We need to understand why that is happening and correct it. Our nasty workaround is to comment the call to esp_timer_impl_update_apb_freq. If you can tell us the correct way to fix this, please open an issue. Thanks!

• Your friends at SparkFun