Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TLS for MQTT #34

Open
jardleex opened this issue Feb 6, 2024 · 14 comments
Open

TLS for MQTT #34

jardleex opened this issue Feb 6, 2024 · 14 comments
Labels
enhancement New feature or request

Comments

@jardleex
Copy link

jardleex commented Feb 6, 2024

Hello,

would it be possible to support TLS on the MQTT sink?
I wanted to lock down my home automation a bit more and as I got quite some stuff going on via MQTT it'll be great to secure that.
For some setup's it may also be required to skip certificate validation. I'm using Lets Encrypt here which should work with enabled verification.

Thank you for your work.
Do you have any way of donating to you?
@AnotherDaniel
Copy link
Owner

Hi @jardleex, I was afraid that someday, someone would ask for this...

Seriously though: I'm on it. Will result in a bunch of new config vars for this plugin, but that's what certificate handling gets you... ;-)

@AnotherDaniel
Copy link
Owner

Ok, just release v1.5.5 with TLS support for the MQTT sink plugin - give it a whirl, hope this works for you!

@jardleex
Copy link
Author

jardleex commented Feb 7, 2024 •
edited
Loading

Hello @AnotherDaniel

also this change works pretty good.
Not the 1h time shift between smahub and iobroker comes as smahub runs on UTC time while iobroker uses TZ: Europe/Berlin. smahub seems not to use this environment variable when set. Just a minor thing.

The only thin I noticed worth mentioning is the 2024-02-07 07:45:33 ERROR Missing CA certificate error. I'm using traefik with LetsEncrypt with a valid/public domain and thus can offer valid certificates. So in this case here it's no actual error for the setup.

# docker-compose service:
  smahub:
    image: ghcr.io/anotherdaniel/smahub:v1.5.5
    restart: unless-stopped
    depends_on:
      iobroker:
        condition: service_healthy
    environment:
      SMAHUB_VERBOSE: true
      SMAHUB_VERBOSER: true
      SHM2_ENABLED: false
      TRIPOWERX_ENABLED: true
      TRIPOWERX_ADDRESS: redacted
      TRIPOWERX_USER: smahub
      TRIPOWERX_PASSWORD: 'redacted'
      TRIPOWERX_PROTOCOL: https
      TRIPOWERX_VERIFYTLS: false
      TRIPOWERX_UPDATEFREQ: 20
      MQTT_ENABLED: true
      MQTT_ADDRESS: redacted
      MQTT_PORT: 2884
      MQTT_TLS: 2
      MQTT_USER: redacted
      MQTT_PASSWORD: 'redacted'
      MQTT_UPDATEFREQ: 20

# smahub logs
2024-02-07 07:45:33 INFO     SHM2 plugin disabled
2024-02-07 07:45:33 INFO     Starting Tripower X source
2024-02-07 07:45:33 INFO     EV Charger plugin disabled
2024-02-07 07:45:33 INFO     demo source plugin disabled
2024-02-07 07:45:33 INFO     Starting MQTT sink
2024-02-07 07:45:33 INFO     HA-MQTT sink plugin disabled
2024-02-07 07:45:33 INFO     gen_ha_sensors sink plugin disabled
2024-02-07 07:45:33 ERROR    Missing CA certificate
2024-02-07 07:45:33 INFO     demo sink plugin disabled
2024-02-07 07:45:33 DEBUG    Starting new HTTPS connection (1): redacted:443
2024-02-07 07:45:33 INFO     TLS enabled
2024-02-07 07:45:33 DEBUG    MQTT sink connected to redacted:2884
2024-02-07 07:45:33 INFO     Connected to MQTT Broker

# iobroker log
mqtt.0	2024-02-07 08:45:33.400	info	Client [smahub] connected with secret 1707291933386_9697

@AnotherDaniel
Copy link
Owner

Have another go - timezone setting support should be there now, and that log message was a mistake.

@jardleex
Copy link
Author

jardleex commented Feb 9, 2024

Confirmed for 1.5.6 that TZ environment variables works and missing CA certificate error is gone.

Thank you!

@jardleex jardleex closed this as completed Feb 9, 2024
@jardleex
Copy link
Author

Hi,
today I've updated the MQTT instance in iobroker which caused a restart of it.
smahub was not able to reconnect though, so I restarted it in the end which fixed the error message below:

2024-02-14 11:42:21 INFO     Got 401 - trying reauth
2024-02-14 12:42:28 INFO     Got 401 - trying reauth
2024-02-14 13:42:36 INFO     Got 401 - trying reauth
2024-02-14 14:42:43 INFO     Got 401 - trying reauth
2024-02-14 14:43:37 WARNING  Disconnected from MQTT Broker
Exception in thread Thread-1 (_thread_main):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 3591, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
         ^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1181, in _loop
    rc = self.loop_write()
         ^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1577, in loop_write
    rc = self._packet_write()
         ^^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 2464, in _packet_write
    write_length = self._sock_send(
                   ^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 649, in _sock_send
    return self._sock.send(buf)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1243, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2427)
2024-02-14 14:47:26 INFO     SHM2 plugin disabled
2024-02-14 14:47:26 INFO     Starting Tripower X source
2024-02-14 14:47:26 INFO     EV Charger plugin disabled
2024-02-14 14:47:26 INFO     demo source plugin disabled
2024-02-14 14:47:26 INFO     Starting MQTT sink
2024-02-14 14:47:26 INFO     HA-MQTT sink plugin disabled
2024-02-14 14:47:26 INFO     gen_ha_sensors sink plugin disabled
2024-02-14 14:47:26 INFO     demo sink plugin disabled
2024-02-14 14:47:26 INFO     TLS enabled
2024-02-14 14:47:26 INFO     Connected to MQTT Broker

Does that say something to you?
Thank you!

@jardleex jardleex reopened this Feb 14, 2024
@AnotherDaniel
Copy link
Owner

AnotherDaniel commented Feb 14, 2024 •
edited
Loading

Does it work again after restarting all involved components, do I see that correctly?
The googling I did so far is inconclusive, other than hinting at the dumpster-fire robustness that is the Python ecosystem 🫤

@jardleex
Copy link
Author

Yes, everything worked again after a restart cycle.
I just wanted to raise this as it might contain valuable information for fixing something.
Generally the MQTT reconnect works flawless as I did multiple restarts of my broker since the new smahub feature.

I'd be okay for me to close this one here in this state.

@jardleex
Copy link
Author

Nah, we can't close this one.
The error comes up every time I restart the broker. The mentioned restarts must have happened when I ain't used TLS yet but after #27.
I'll do some more testing and will report what I found.

@AnotherDaniel
Copy link
Owner

Ok, I'll also try to reproduce this.

@jardleex
Copy link
Author

I did some tests again and only got the issue with MQTTS. With plain MQTT the reconnect worked just fine.
I triggered the error message above by simply restarting the mqtt broker.

@AnotherDaniel
Copy link
Owner

@jardleex just to let you know I haven't forgotten about this issue - not that much free mind-time at the moment, and when I tried to pick this up my TLS config (that was fine the first time around) refused to work. As I said, not forgotten...

@AnotherDaniel
Copy link
Owner

Add-on: I just figured out what my recent issue was, and have no idea why this ever worked? Anyways - broker-restart-reconnect works for me, but that might not be related to your scenario. But please have a go with v1.6.2 and let me know how that looks for you!

@jardleex
Copy link
Author

jardleex commented Mar 8, 2024

Hi,
no worries, that's hobby for all of us. We as users can be happy to have able people like you to help us out of the misery the manufactures left us in.

Sadly 1.6.2 ain't changed the end result if I restart my broker:

2024-03-08 12:26:56 WARNING  Disconnected from MQTT Broker
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.name: Sunny Tripower X 25
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.configuration_url: https://REDACTED
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.identifiers: 3016101878
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.model: SMA-Sunny Tripower X 25
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.manufacturer: SMA
2024-03-08 12:26:58 DEBUG    TriPowerX.3016101878.device_info.sw_version: 03.06.15.R
2024-03-08 12:26:58 DEBUG    https://REDACTED:443 "POST /api/v1/measurements/live HTTP/1.1" 200 None
Exception in thread Thread-1 (_thread_main):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
2024-03-08 12:26:58 DEBUG    value of Operation.Evt.EvtNo is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.FstStop is currently not availably (nighttime?)
    self._target(*self._args, **self._kwargs)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 3591, in _thread_main
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.HzLimRed is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.PFCtlComCfg.PF is currently not availably (nighttime?
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.PFCtlComCfg.PFExt is currently not availably (nighttime?)
    self.loop_forever(retry_first_connection=True)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.PFCtlComCfg.PFExtIn is currently not availably (nighttime?)
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.PFCtlComCfg.PFIn is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.VArCtlComCfg.VArNom is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.VArModCfg.VArCtlVolCfg.VolRef.VolRefPu is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.W is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WMaxIn is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WMaxInNomPrc is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WNom is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WSptMax is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WSptMaxNom is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WSptMin is currently not availably (nighttime?)
2024-03-08 12:26:58 DEBUG    value of Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WSptMinNom is currently not availably (nighttime?)
    rc = self._loop(timeout)
         ^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1181, in _loop
    rc = self.loop_write()
         ^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 1577, in loop_write
    rc = self._packet_write()
         ^^^^^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 2464, in _packet_write
    write_length = self._sock_send(
                   ^^^^^^^^^^^^^^^^
  File "/opt/smahub/.venv/lib/python3.11/site-packages/paho/mqtt/client.py", line 649, in _sock_send
    return self._sock.send(buf)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1242, in send
    return self._sslobj.write(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2427)

Please let me know if I can provide any additional info to support you.

@jardleex jardleex mentioned this issue Mar 8, 2024
Open
@AnotherDaniel AnotherDaniel added the enhancement New feature or request label May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jardleex @AnotherDaniel