Reorganizing Erlang libraries

[pguyot]

Libraries shipped with AtomVM are currently:

• alisp
• eavmlib
• esp32boot
• esp32devmode
• estdlib
• etest
• exavmlib
• gleam_avm

main on ESP32 and RP2 loads atomvm lib and then the user application. So does atomvm script on Generic Unix. On STM32, there is a single AVM while emscripten can load any number of avm. atomvmlib is the combination of several of the libraries above.

Proposal is to:

1. Have all VMs on MCUs load a single avm package
2. Therefore have rebar3 and mix pack required libraries with user code, either as an Erlang release (relx) or with some specific plugin arguments (atomvm rebar3 plugin can do it with -p)
3. Have each library published on hex, so they can be fetched with mix / rebar3
4. Have each library become an Erlang application. init module would be the first called by VMs, and it would load a boot script (if present) or try to gues the entry point. The search for entry point would move from C to Erlang.
5. Applications could be specialized for a given platform, so that this code isn't loaded on other platforms
6. We could also publish on hex a subset of an OTP application (stdlib) to add missing modules

For example:

ekernel (equivalent of OTP kernel + erts)

• application.erl
• code.erl
• dist_util.erl
• erl_epmd.erl
• erlang.erl
• erpc.erl
• erts_debug.erl
• gen_tcp_inet.erl
• gen_tcp_socket.erl
• gen_tcp.erl
• gen_udp_inet.erl
• gen_udp_socket.erl
• gen_udp.erl
• inet-priv.hrl
• inet.erl
• init.erl
• kernel.erl
• logger_std_h.erl
• logger.erl
• net_kernel_sup.erl
• net_kernel.erl
• net.erl
• socket_dist_controller.erl
• socket_dist.erl
• socket.erl

estdlib (equivalent of OTP stdlib + crypto + ssl)

• base64.erl
• binary.erl
• calendar.erl
• crypto.erl
• ets.erl
• gen_event.erl
• gen_server.erl
• gen_statem.erl
• io_lib.erl
• io.erl
• lists.erl
• maps.erl
• math.erl
• proplists.erl
• queue.erl
• sets.erl
• ssl.erl
• string.erl
• supervisor.erl
• timer.erl
• unicode.erl

eavmlib platform independent utilities

• ahttp_client.erl
• atomvm.erl
• avm_pubsub.erl
• console.erl
• epmd.erl
• http_server.erl
• json_encoder.erl
• logger_manager.erl
• mdns.erl
• network.erl
• network_fsm.erl
• port.erl
• timer_manager.erl
• timestamp_util.erl

eavmlib_websocket

• websocket.erl (socket version)

eavmlib_esp32 esp32 hardware interface

• esp.erl
• esp_adc.erl
• gpio.erl
• i2c.erl
• ledc.erl
• spi.erl
• uart.erl

• dependency on eavmlib_websocket

eavmlib_rp2 rp2 hardware interface

• gpio.erl
• pico.erl

• dependency on eavmlib_websocket

eavmlib_stm32 stm32

• gpio.erl

eavmlib_emscripten emscripten hardware interface

• emscripten.erl
• websocket (emscripten version)

etest (equivalent of OTP eunit)

• etest
• eunit

alisp (unchanged)

These libraries would have Elixir and Gleam equivalents, i.e. exavmlib would be split.

[bettio]

I completely agree on the idea of reorganizing libraries, it is something very welcome.
So I think the discussion is about how.

• etest, alisp as separated libraries, I agree 100%.
• we must split target specific modules, so we don't end shipping useless modules to the wrong target (e.g. emscripten on the ESP32), so I agree as well, but I have some additional comments:
  -- an idea might be calling them avm_target_(h)al or just avm_target
• About ekernel:
  -- I'm not sure why we separate it from estdlib in our specific case, I didn't understand the reasoning about this choice
  -- I was thinking about networking modules such as socket: shouldn't we keep them separated (for the sake of saving flash on some targets such as some STM32, e.g. the blackpil)?

[pguyot]

Regarding names, do you suggest that what I called eavmlib_esp32 above be avm_esp32 or avm_esp32_hal instead?

Regarding ekernel, my initial thought was that matching OTP applications would make it more easy for users to use OTP applications instead, and figure out where is what, especially for Erlangers (I realize this is a very Erlanger way of seeing things). But estdlib refers to modules with native code, so users cannot use OTP stdlib without our modules with native code.

I agree that networking modules could be split, if we drop the 1-1 with OTP applications, we should do that.

[UncleGrumpy]

As far as STM32 goes the libs are not on the device, so when they are packed into the application any unused modules are pruned, (unless prune is set to false) so these modules will never end up on the device.

The only downside I see is for ESP32 is that we would want to implement some of the esp32boot functionality into main.c, like checking the app partition in NVS (to allow OTA app updates).

[pguyot]

Let me push this idea again, especially considering that with JIT and escriptize some of the ideas have progressed.

We definitely should make prune the only way for packbeam to work and provide for a declarative way to circumvent it if it removes too much code. I was thinking of developing a more agressive pruning algorithm that could remove functions (for example we never call module_info functions yet they are in every lib), and maybe literals.

We could consider .avm format to be a temporary storage between desktop and device, and each .avm be very specific to a given application and device. This is where we could decide to natively compile code if needed as well. We wouldn't ship the .avm anymore but rather the pristine .beams and rebar3/mix/gleam would do the work of packing, pruning, natively compiling if needed and flashing.