New smart meter driver written in Berry

[kobuki]

Since the utility company mounted a new smart meter with P1 port in our fuse box, I've been struggling with reading the P1 port it's equipped with. The native smart meter interface is pretty good, but since I receive bad telegrams quite frequently and that driver can't really deal with random outliers or outright bad values (even with the median filter active), and doesn't support checking the embedded CRC, I decided to make my own driver that tries to solve this, this time using Berry. It's only for OBIS ASCII telegrams ("DSMR" data).

The driver and some usage docs are here: https://github.com/kobuki/tasmota-smr-berry

It aims to be mostly compatible with the metric descriptors of the original Tasmota driver, so those can be used with my driver, too. A few things are not supported, like summary for the previous month and some detail settings like number of digits after the dot for numeric values, posting frequency (instantly vs. teleperiod - it's always instantaneous), the main page and teleperiod sensors got a separate config, etc. I hope it can be useful for others. If you try it out, I'd be glad to receive some feedback aobut your setup and how it works for you.

PS: my meter has a surprisingly flaky P1 output. Sometimes I only get a 20% good packets, but sometimes it's almost perfect, around 90% as validated by CRC. At first glance seems to correlate with the temperature of the device. It's the best around 44-45°C, and in the colder nights nowadays it's routinely 20% or sometimes worse. Weird.

[sfromis]

Great. Took a quick peek at the code (not having such a smart meter), I noticed a thing about your handling of timing and the multitasking/multithreading Tasmota does not have. This means that all your code is blocking, and busy-waiting is bad practice, like this:

        var t1 = tasmota.millis() + 150
        while !tasmota.time_reached(t1)

The preferred way is use driver callbacks like every_50ms to do such polling for input. This allows normal processing of Tasmota to proceed without being "dead" to the world.

And your happyTasmota / tasmota.yield() does not "solve" the issue of your code being blocking, it will basically only keep the watchdog from biting. Whether this is a "real problem" depends on how slow your processing is. Berry code tends to run at a decent speed, but those calls of happyTasmota will significantly slow your blocking code down.

[kobuki]

First I tried as you said, by implementing every_xxx() functions (I still use every_250ms() for messaging tasks). That caused a lot of missing data, even with every_10ms(). So I detect serial data using a "fast loop" callback, but then if I don't get a stranglehold on the serial port in a busy-wait burst, I lose data, every time. Believe me, I tried but at the end I concluded this is the only way. There's no interrupt-based callback for this. 150 ms is calculated from the theoretical readout of a whole telegram (around 100 ms) plus some leeway for the meter. This will need some optimization, though, as other meters have varying amount of data and sometimes a different baud rate, mainly on older meters.

I'm fully aware there's no parallel multitasking, even though the ESP32 model I use has 2 cores and in theory, WiFi is handled on the other core. But actually, and according to the docs, tasmota.yield() is to let Tasmota do its regular tasks like resetting the WD, tending to WiFi tasks, etc. Without it, it could be more serious. Load is pretty light. I tested it for a few days, I see no problems.

Oh no, berry is fairly slow. But anything using a native function is fast. First I tried to avoid regexes, but then I saw that its performance impact is negligible. The CRC16 function is written in Berry, because the one in the module uses a different algo and poly. For 1132 bytes, the CRC is around 550 ms without tasmota.yield() calls. I tried using a LUT version, that's around a 4x speedup at ~130 ms. I think the native one is only a few.

[sfromis]

No, yield is not to let Tasmota tend to lower level functions, only to let Arduino do such. Tasmota can do no processing during yield.

And when it comes to speed, basic optimization is always a good step to take in any language, like when I saw a speedup of almost 5x with this variant of crc16 (less speedup on the faster ESP32-S3, and more speedup on ESP32-C3), even without replacing the inner loop with a lookup table.

def crc16(data, crc, len)
    var di = 0
    while di < len
        crc ^= data[di]
        var i = 0
        while i < 8
            if crc & 0x0001
                crc = (crc >> 1) ^ 0xa001
            else
                crc >>= 1
            end
            i += 1
        end
        di += 1
    end
    return crc
end

That range object created with .. every time (especially for the inner loop) costs more than the calculations and looping.

To keep up with fast serial data, my experience is that cumulative concatenation like your self.telegram .. buf slowed things down, instead collecting data in a list for processing after the whole packet was received. That would also lend itself to the optimization of breaking code execution into chunks, each launching the next with tasmota.set_timer(0, ....). That gives Tasmota "air" to do other processing.

[kobuki]

Well, about yield(), it's correct, but the point is not to hog the device too long with any task or script to keep it responsive. This is achieved so I see no issues. If someone wants to optimize the Berry code further, PRs are welcome. I consider this a singular solution for a specific task, I don't currently see it a priority to please other tasks that might run on it. I still might do it, though, should I have more time to work on it.

Thanks, I've quickly tried your version of the CRC16 code, and it's indeed a lot faster this way, about 116 ms (vs. ~550 ms) for the same 1132 bytes. Maybe with a LUT it could be sped up more. I'll definitely change the algo in my code to this one but I see further optimizations pointless in this case.

Using the list approach you mentioned for serial might speed things up a bit, but I doubt I'd gain a lot. bytes .. bytes is just shuffling around data in native code, after all. But it would definitely help in a more dynamic approach to accommodate different telegram sizes. so I might change that part.

[sfromis]

As I said, my experience is that getting processing out of the serial receive phase helped a lot, as it is not just shuffling of bytes but also objects allocations and disposals, like with those range objects which slowed down CRC. Only after this, I could keep up with full speed 115200 without losing data, while doing the polling via a tasmota.add_fast_loop callback.

[johanvdk1]

I just wrote one for Arduino, my first ever Arduino project. Little programming experience (35 years ago). But it reads every parameter from the P1 port, virtually no CRC errors (about 99% ok) but http and mqtt timeouts -poor WiFi- are blocking. So it fails to get the message correctly and drops it. Reports all parameters through MQTT and most to Influx.

[kobuki]

it reads every parameter from the P1 port

The P1 port presents actual metered values, not parameters. I don't exactly get what you mean. My code allows to narrow what's reported, not unlike the original Tasmota smart meter interface.

virtually no CRC errors

It turned out my frequent errors were caused by a flaky USB charger I used as power supply (I can't use the power from the port becaues I only had a 4-pin cable at installation and it's in a sealed plastic box). I use a basic, but good quality PSU now (from Mean Well) and transfer is 100% with some blips within a short period every 3-5 days. I suspect that the meter is busy with reporting or some housekeeping in its FW. Messages are collected in Influx and Home Assistant.

[johanvdk1]

Good you found the cause of your problem!
A metered value is a parameter, but so is for example the serial number, the day/night tariff. So parameter is in my case the combination of measured values and just reported data. So it's probably what you mean by measured values.
Mine is written in C++ on an ESP32 with Arduino IDE so it's probably a totally different animal :-). When I started I did not know about the existence of Tasmota.

[kobuki]

When I started I did not know about the existence of Tasmota.

Yeah, I suggest also having a look at the Smart Meter Interface. And on ESP32, Berry support makes writing stuff like this really easy and faster than native code.