TinyTuya Server

[jasonacox]

I updated the server.py UI to make it easier to see the devices and control them. It is basic but useful for testing.

Future Enhancements?

I would like to improve the UI to make it more intuitive to control the devices. A nicer look would be good too.

• Add MQTT support to send device state updates as well as control as requested in Add MQTT to tinytuya server #255
• More intuitive / clean UI for controlling devices (difficult since devices all vary but at least some general on/off controls, numeric sliders, color selection)
• Possibly embed in module (must not be too heavy) to allow simple python -m tinytuya server type execution.

Would love to hear from the community:

• Is this valuable and worth investing time to improve?
• What other features / look would you like to see?
• Are you a skilled UX designer / javascript guru and want to help? 😉

[uzlonewolf]

A while back I ended up writing my own server-like program as I went in a bit of a different direction than the current server. It reads in a list of devices, opens persistent connections to all of them, and then monitors for updates. When an update is received it both forwards it on to connected clients and checks it against a list of device/actions and triggers other actions if the requirements are met. The client connections are currently a plain TCP socket it sends JSON over as the program it's talking to handles the WebSockets, but I am considering having it handle WebSockets natively as well (I use https://github.com/dpallot/simple-websocket-server to add WebSockets to things). I plan on releasing it once it's ready, but there are a few things which need to be finished and/or cleaned up first.

[jasonacox]

Love it! I had to make some minor edits to get it running in current state (sent PR) and just added a generic localhost.pem for testing.

Also note for others wanting to test, you can use your devices.json to connect to your devices and be sure to install the SimpleWebSocketServer library:

pip3 install SimpleWebSocketServer

[uzlonewolf]

I figured out how to work in basic HTTP(S) requests and am in the process of adding that.

As for the certificate, I think I'm going to add command line arguments for that, the bind host/port, and the various config files.

[jasonacox]

Awesome! If this is going to be a server that someone would run all the time, it would fit well in a docker/podman container. We may want to add the ability to set those things via environmentals (for docker run).

[uzlonewolf]

We may want to add the ability to set those things via environmentals (for docker run).

Does it really need to be an environmental for docker? I don't really use docker, but a quick search says you can just call it with arguments like normal with docker run --rm <yourImageName> <args>.

Anyway, argparse has been added with options for most settings. If devicelist.json isn't provided then it automatically loads devices.json instead. HTTP(S) has also been added, but right now it does nothing but serve up a demo HTML page that connects to the websocket and print the traffic. (Direct HTTP device control is not yet implemented.) Websocket and TCP commands have been added to reload the configuration files without needing to restart the server; to support applications with no persistent storage I want to eventually have it so you can start the server without any configuration (besides the bind port/IP) and then load it in via the websocket. Lots of refactoring and breaking things out into modules. The SSL/TLS certificate is first loaded from /etc/letsencrypt/live/this_site/fullchain.pem and, if that fails, ./localhost.pem is then tried. If neither of those load then it continues on without SSL/TLS support; you can tell it to try again once it's running by sending a reload_cert command.

Todo:
Add HTTP commands to make it compatible with the existing server API.
Make it (optionally) completely configurable via websocket without needing any JSON files at all.
Add MQTT.
Make it discover devices on the fly instead of just at startup.
Documentation.

[jasonacox]

Great progress!

[mschlenstedt]

Hey,

I love the idea of this server. I recently created a MQTT<->TinyTuya Bridge for a specific device (pool heat pump) using my own server script.

This can be easily modified to communicate with server.py directly via HTTP instead of using tinytuya functions. This would be a seperate server process for the MQTT stuff, so developement can be seperated between server.py and the MQTT bridge.

If there's interest, I can provide the modified script as a proof of concept. Let me know what you are thinking about that.

What I would like to have in the server.py, too:

• Device files which translates DPS IDs into names for easier and more readable status, see here for an example: https://github.com/make-all/tuya-local/blob/main/custom_components/tuya_local/devices I think identifying via productID should be possible,
• Using names instead of DeviceIDs in the commands. E. g. /set/{DeviceID}/{Key}/{Value} could also be /set/{Name}/{Key}/{Value}
• Using names instead of DPS IDs in the commands (see first bullet). E. g. /set/{DeviceID}/{Key}/{Value} could also be /set/{Name}/{KeyName}/{Value}

For the last two bullets we have to think about handling of unique names and maybe forbidden letters (space, special signs etc.).

[uzlonewolf]

The "multiserver" rewrite I'm working on actually does the "device names instead of ID" thing already. Special characters aren't a problem as you can just URL-encode them (i.e. space becomes %20, / becomes %2f, etc).

I really, really, really want to add the "function name instead of DP ID" thing, and the lack of it is why progress on my multiserver has stalled. Adding it in a way which will not absolutely hammer the cloud server is nontrivial though and is going to require a number of changes I just have not yet had time to implement.

[mschlenstedt]

Love it. Does your multiserver do the MQTT stuff or should I make a pull request for a seperate mqtt.py server which communicates with server.py via the API?

[uzlonewolf]

I have hooks to implement MQTT natively but have not finished it yet as I wanted to get the DPS name stuff done first.

[mschlenstedt]

Ok, than I will push my small MQTT script. Maybe it's useful until your solution is ready.

I implemented the name stuff in one of my own scripts already. It is quite simple and I think it will not hammer the main server.

I created one json file per product ID which contains the translation ID to Name. When parsing the devices.json. I then look for an existing translation-json when parsing the devices.json and include it in the main tuyadevices dict. https://github.com/mschlenstedt/tinytuya/blob/3c7cb35ea38aa45a84bab1e2017421d09c69f3f3/server/server.py#L231

This way you just have to loop through parts of the dict each time you need a translation, which is fast and do not need much resources - even with tons of devices.

[mschlenstedt]

As a first step I added ability to use {DeviceName} instead of {DeviceID} in API calls, see PR #352

Next step will be to have DPS ID to Code (Name) mapping like e.g. IO Broker and FHEM uses with tinytuya.

[spinza]

This seems like a good place to discuss. I was about to start building a tool that takes devices and publishes them on MQTT. This server idea is good in that it exposes everything as a default via API (and potentially MQTT). I'm new to tinytuya so may be missing key concepts

Some thoughts:

• Is there a single "database" of DP IDs for various devices and how they are exposed and settable that we can use? Is this what the cloud access provides?
• Can the API expose these values using the right labels/enumerated types/etc. instead of codes automatically? As well as treat the typing of fields etc correctly? In spite of those devices not being coded up in the underlying tinytuya python classes.

This leads me to what I wanted to do:

I wanted to write an MQTT service that publishes the devices using the Homie convention to MQTT. This allows any device to being published to that topic to be autodiscovered by OpenHAB or anything else that uses that convention. Home Assistant also appears to have something similar called MQTT Discovery. It would have been good if they supported the standard though (though there may be some work and dsicussion in that regard).

Below is an example of how Homie looks. You can see that the DP ID mappings etc. would be very useful in populating this quickly. One should then be able to quickly scan a network and publish all devices where the mappings are available easily as well all in one go in which case these devices will just pop up in any tool supporting the Homie convention and then be seamlessly added to those tools.

You can see configured types as well as reporting whether a topic is "settable" that allows updates to be sent to the topic. One could also publish the raw DP IDs as well as a minimum in case mappings aren't available.

  homie / device123 / $homie → 3.0
  homie / device123 / $name → My device
  homie / device123 / $state → ready
  homie / device123 / $nodes → mythermostat

  homie / device123 / mythermostat / $name → My thermostat
  homie / device123 / mythermostat / $properties → temperature

  homie / device123 / mythermostat / temperature → 22 
  homie / device123 / mythermostat / temperature / $name → Temperature
  homie / device123 / mythermostat / temperature / $unit → °C
  homie / device123 / mythermostat / temperature / $datatype → integer
  homie / device123 / mythermostat / temperature / $settable → true

The Home Assistant version does similar things but uses json encoded config topics.

Does one fetch these mappings for any device from the Tuya cloud or do we need to build up a database/dataset of these mappings per the README?

[uzlonewolf]

There is no single database that I know of, however for devices you own there is a way to download them from the cloud. I have a PR implementing almost all of what's in your example over at #370 but have been stupid-busy recently and haven't had time to finish it.

[mschlenstedt]

And until @uzlonewolf Server implementation is finished, you can use the MQTT Gateway here: https://github.com/jasonacox/tinytuya/tree/master/server/mqtt

[spinza]

I think we need to reorganise how the device mappings are fetched from the cloud. There are two lists returned from the cloud with mappings. status and functions. The former I guess represents readable data from the device and the latter represents ones we can set (or write to).
Currently these are combined into a single list here.

We need to know which are writeable and/or readable to enable better automation of any API, or MQTT server that uses these mappings automatically.

[uzlonewolf]

Added a 'settable' flag to the mappings, but it's mostly untested. Seems only ~10% of DPs are not settable with my device list.

Settable DPs: 1055
Non-settable DPs: 121

[uzlonewolf]

79f6733