Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

punyforth integration into tasmota? #43

Open
wolfgangr opened this issue Jan 18, 2019 · 18 comments
Open

punyforth integration into tasmota? #43

wolfgangr opened this issue Jan 18, 2019 · 18 comments

Comments

@wolfgangr
Copy link

Dear Attila,

I am looking for a scripting language for expanding the configurability of the tasmota framework
https://github.com/arendst/Sonoff-Tasmota
As punyforth, tasmota runs on ESP8266 platform and started on sonoff devices.
It implemets a web surface, MQTT client, serial and http command, remote syslog feature...

Forth is a natural choice, and punyforth runs on the target platform - great :-)

However, tasmota is not based on RTOS, but on arduino-esp8266 and implements a simple arduino type architcture

init() { 
     do some stuff; 
}

loop() {
    repeat stuff all the time;
}

Do you think it is possible to get punyforth built and running?
A first glance on https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/user_main.c made me optimistic.

This reads like it can be called from arduino framework as well.
It's just the xTaskCreate that worries me a little bit.
Does this belong to RTOS?
How deeply are punyforth and RTOS woven together ?

I think I willl need

  • hooks to create tasks
  • hooks to initiate the whole thing
  • hooks for the arduino yield() which resembles the scheduler
  • or may be it suffices just to call it once in every loop turn?
  • hooks to redirect tasmota command processing stream to forth
  • information on the memory and flash management to keep it coexistent
  • some hints on the details of the multitasking implementatin
  • maybe some extra hardware function wrappers

I'll try to understand the code myself, but of course, any pointer to assist me in my endeavour will be appreciated :-)
@zeroflag
Copy link
Owner

zeroflag commented Jan 20, 2019 •
edited
Loading

hi @wolfgangr,

The core of punyforth is independent of RTOS, but there are lots of esp specific functionalities (like WiFi controlling, GPIO, i2c, spi, netcon) which are coming from RTOS.

Most of these are defined in ext.S. The CCALL macro is used to call C functions from Punyforth.

Here is an example:

defprimitive "load",4,load,REGULAR
    DPOP a2			// block number
    CCALL forth_load
    NEXT

This will call the forth_load() C function defined in forth_io.c which will call some RTOS code.

If you want to have all these functionality (GPIO, netcon, etc) you would need to change these C functions to call the arduino equivalents of the RTOS code.

There are also some words in ext.S which are only for optimization these don't need to be changed.

It's just the xTaskCreate that worries me a little bit.
Does this belong to RTOS?

Yes. The Punyforth outer interpreter is executed as an RTOS task. Regarding the init() and loop(), I'm not sure what's the right way to use them. Is code in loop() supposed to return? What happens if you put an infinite loop inside the loop? If this is not allowed the outer interpreter should be modified to evaluate one word at a time I guess, because currently that's an infinite loop.

If the init() and loop() can be avoided somehow by using the lower level arduino machinery then that would be an other option.

@wolfgangr
Copy link
Author

@zeroflag
Attila, thank you for response.
Glad to see that I'm not walking alone :-)

I work on a fork of tasmota and keep documenting a journal of my considerations at the issue tracker there: WelterRocks/WESPOTA#15

I've selected three candidates for my implementation:
punyfort, cforth
and cxxforth

@wolfgangr wolfgangr mentioned this issue Jan 21, 2019
Open
@wolfgangr
Copy link
Author

punyforth is my current favorite, above all because of its existing multitasking implementation.
I hope this will help me to weave it ito the task management of arduino & tasmota.

However, at he moment,
WelterRocks/WESPOTA#15 (comment)
I see a large problem since punyforth eats up most of RAM.

  • How much of that has to be attributed to punyforth itself, how much to the RTOS and ESP core?
  • Can we shift more of runtime FORTH to FLASH?
  • Can we set the compiler to store the dictionary in FLASH, instead of RAM?
  • Can we move/recompile dictionaries which are already compiled to FLASH?

@wolfgangr
Copy link
Author

If you want to have all these functionality (GPIO, netcon, etc) you would need to change these C functions to call the arduino equivalents of the RTOS code.

Yes, I see.
Do I find them all in arch/esp8266/rtos/user ?

I've started dissection and a preliminary appraisal of neccesary work, about here:
WelterRocks/WESPOTA#15 (comment)

I think there is some work to do, but I couldn't identify any major obstacle along the way, yet.

@wolfgangr
Copy link
Author

Is code in loop() supposed to return?

Yes, of course. That's what cooperative multitasking is all about.

What happens if you put an infinite loop inside the loop?

Blocking code in loop() freezes tasmota.
The loop() resembles the task handler of any arduino code.

If this is not allowed the outer interpreter should be modified to evaluate one word at a time I guess, because currently that's an infinite loop.

Yes, that might be one approach.
And maybe a good one for the start and later on as a fallback for debug purposes.

So, the next question were, how can I hook into the outer interpreter?

@wolfgangr
Copy link
Author

However, my considerations go beyond that:

What about the inner interpreter?
The goal at the end is to implement drivers in forth and hook them into tasmota.

I think your task manager task.forth is a good point to start with.
So I will have different forth tasks to be called in sequence in one single arduino loop() trip.
The suspend/resume to be implemented then had to be linked some way into the pause mechanism, I thought.

I could not yet figure out the role of the xpause symbol.
grep finds it both in task.forth and in ext.S
In the last, it is written just below the readchar definition.
I think it makes sense to assume (and resembles your proposal) that a task switch is called when forth is waiting for the next char on the serial line, but I haven't yet figured out the syntax of those .S files in detail.

How and when are those .S files processed?

  • Is it generic assembler code? If so, could you point me to some reference guide?
  • or is it one of the pyhton scripts running during build?
  • or even by Forth itself?
  • or whatever?

@wolfgangr
Copy link
Author

So, the hook question from above generalizes to:
How can I suspend from and resume to FORTH without screwing the C environment?
I think the compiled C-code assumes some CPU-registers, C return and parameter stack und maybe some other "stuff like that" intact.

My big question for the moment (beyond the RAM issue):
Can I implement a suspend from forth as a simple C-return
And as resume a call to sth like forth_load such that forth resumes just where it suspended before?
Ideally, forth should manage its state between such suspend - resume.
(So that forth beaves as a state machine, in the context of a multitasking framework vocabulary)

If done from within a pause, should it not be sufficient just to save the task pointer, referring to his Taskstruct of the next task to run? I think, saving state of a task is what this Taskpointer is all about, right?

So the cycle goes:

  • ardunino_setup() -> .... forth_load
  • forth realizes it's virgin state and performs its setup work,
    among other, it initiates a task framework with just the REPL in and returns by a call to pause
  • pause saves the next task to load and forth_load returns control to C
    if necessary, any C state (stack, CPU register, ... what else?) is restored before
  • arduino_setup() continues after forth returns from forth_load
    ===== arduino starts looping =====
  • arduino somewhere in its loop() calls forth_load again.
  • forth now realizes that it has already been initialized
    -- it restores its previous state by loading the Task struct
    -- performs one task, or one scheduler round, or whatever is configured from priority considerations
    -- up to some pause condition that decides to suspend again
  • forth returns control to C again the same way as described for the init
  • arduino recycles its loop(),
  • going on infinitely, thus switching between forth and tasmota, until some nasty blocking code triggers the watchdog

Alternatively, the suspend / resume sequence might be coded into a separate FORTH task within the task.forth framework. There might be even more than one copy of it in the task list, allowing finetuned scheduling of the task switching to tasmota&arduino core.

Can you follow me?
Can you help me with my questions?

  • RAM optimisation
  • C-ish subroutine return from forth pause
  • Forth task resume with a load_forth call

@zeroflag
Copy link
Owner

zeroflag commented Jan 21, 2019 •
edited
Loading

  • Can we shift more of runtime FORTH to FLASH?
  • Can we set the compiler to store the dictionary in FLASH, instead of RAM?
  • Can we move/recompile dictionaries which are already compiled to FLASH?

Currently no. User defined words are stored in RAM. As far as I remember @holinepu did some work on compiling words to FLASH.

Do I find them all in arch/esp8266/rtos/user ?

Yes.

So, the next question were, how can I hook into the outer interpreter?

The outer interpreter is defined in: outerinterpreter.S. These .S files are assembly files and they contain either native asm code or compiled threaded forth code in a textual format (using labels and symbols instead of binary). These files contain the minimal set of words that are required to parse and compile textual forth code. These were written by hand.

For example the below code is equivalent to the "word 2dup" forth code.

    .int xt_word                           // ( a len )
    .int xt_dup2

The outer interpreter is in a loop, there are 3 places where it jumps back to its beginning.

    .int xt_branch
    lbl outer_interpreter

The outer interpreter is invoked from punyforth.S which is invoked by the xTaskCreate in user_main.c

So if you remove the loop from the outer interpreter and call that from the arduino loop() it might work. But you also need to save the registers which are used by the outer and inner interpreter.

There might be an other problem, namely that readchar (which gets the next character of the source code or the next key entered by the user) blocks by default if there is no input available. But it has an other version which is used in multi tasking mode that doesn't block.

@holinepu
Copy link

Can we shift more of runtime FORTH to FLASH?
Can we set the compiler to store the dictionary in FLASH, instead of RAM?
Can we move/recompile dictionaries which are already compiled to FLASH?
Currently no. User defined words are stored in RAM. As far as I remember @holinepu did some work on compiling words to FLASH. yes, I've done some work on punyForth 2 months ago. I can compile the program in RAM and then move it to flash momery. Gradually the program can be bigger and bigger. I've sent the program to Mattias and Peter Forth but no response. So, now I switch to RISC-v RV32I ，RV32M . I've completed the assemb|er，disassembler，decompiler and cross-compi|er for mecrispForth and tinyForth. The same above functions have been finished in my punyForth development system. I'd like to share it with Forthers it they'd like.

@holinepu
Copy link

Of course the newer added program can be compiled under cross-compiler, that way we don't have to worry about the problem of compiling in the limited RAM. Wouldn't that be great!?

@wolfgangr
Copy link
Author

Hi @holinepu
great to have a response from you.

Some findings from other forth'es:

Cforth uses a elaborated host based forth compilation machine
https://github.com/MitchBradley/cforth/blob/esp32-v1.0/src/cforth/embed/BuildProcess.txt
Thus it has a bunch of libraries already compiled in, which reside in flash.
I thnik it should be possible, one an application i developped to maturity level, to drop this code into the build directory and flash it to the ES8266 this way, as well.
Sound not really forth'ish, but will do the job.

mecrisp (on STM32 blue pill) can interactively switch between compiletoram and compiletoflash.
It supports a cornerstone word to erase flash down to some marker set before.
So you once flash a mecrisp core and the rest is interactively - forth'ish, I'd say.
Unfortunately, mecrisp is written in assembler and not yet ported to the ESP8266 / Tensilla core.

@wolfgangr
Copy link
Author

yes, I've done some work on punyForth 2 months ago.
I can compile the program in RAM and then move it to flash momery.
Gradually the program can be bigger and bigger.

Great - looks close to what I have seen in mecrisp.
I assume you know mecrisp and the way it works, there?
Well, you cannot move code over, you keep a source file on your workstation.
Modify and compile to ram, until it settles.
Switch over to flash, compile again and it is persistent.

switch to RISC-v RV32I ，RV32M

does this relate to the ESPXXX family? Does not ring a bell in my ears.
Ask Google - Ah, I see, that's the VHDL path.
I'm afraid that's beyond my capabilities.

Well, I think the ESP8266 is a FPGA-based SOC anyway.
So yes, having a native forth engine running on it would be great.
But I'm afraid the information required to implement that is not available in public.

I've completed the assemb|er，disassembler，decompiler and cross-compi|er for mecrispForth and tinyForth.

For the ESP-family as well?
I'd prefer mecrisp because I found loads of tutorials and examples.
Maybe it's not that mature. but I think that is what forth is about.

The same above functions have been finished in my punyForth development system.

I this available in public?

I'd like to share it with Forthers it they'd like.

Sure, I would like it. At least give it a thorough test.

Of course the newer added program can be compiled under cross-compiler,
that way we don't have to worry about the problem of compiling in the limited RAM.
Wouldn't that be great!?

To be honest, I considered that as a crutch only.
It mayb be fine for large scale deployments.
But for me, forth is the platform for lot sizes close to 1.
This is hands on.
Test on RAM - successively move to flash as components mature.
Keep source files on a host, so you always can start off again.
Just a bare minimum forth in flash to bootstrap.
like in this tutorial:
https://github.com/jeelabs/embello/tree/master/explore/1608-forth/rnw#installation

Keeping the progression steps manageable, only small dictionaries have to stay in ram.
The cornerstone funcionality is essential, so that you can selectively remove some top part of the flash based dictionay as well. But this is just 4 lines of forth code.

If it works on a 64KB flash / 20 kB RAM STM32 BluePill, I think it will work on a > 500 k free flash ( 3.5 MB on a EUR 3,- NodeMCU) 128 k RAM ESP, I hope.

@holinepu
Copy link

holinepu commented Jan 27, 2019 via email

@holinepu
Copy link

holinepu commented Jan 27, 2019 via email

@holinepu
Copy link

holinepu commented Jan 27, 2019 via email

@holinepu
Copy link

hi, Wolfgangr
I tried to response your request of the files relating to how to expand the program in RAM to flash rom by email, but it failed to the upper 3 comments above. I've attached the files on above, but fail to see it there. How can I attach the file to you by email. Your emil address?

@holinepu
Copy link

holinepu commented Mar 7, 2019 via email

@holinepu
Copy link

holinepu commented Jun 30, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zeroflag @wolfgangr @holinepu