Are there any difference between mqttsrc/sink and tensor_query_* API?

[sungbin-lee]

Hi dev.
I am trying to implement edge pipeline Tutorial(T3, T4).
T3: https://nnstreamer.github.io/tutorial3_pubsub_mqtt.html#
T4: https://nnstreamer.github.io/tutorial4_query.html

Example in this tutorial is written only same device(localhost) and I did it. However I couldn't do this in other device(192.168.0.x).

I mean I did
localhost/other device by using mqttsrc/sink API. - MQTT protocol
only localhost by using tensor_query_* API. - TCP/MQTT protocol

are there any other examples between different devices using tensor_query_* API?
Also, I found https://github.com/nnstreamer/nnstreamer-example/tree/main/bash_script/example_tensor_query_mqtt and In client example, dest-host is written tcp://localhost, it needs protocol prefix like that?
Of course, I tried this but failed.

I tried MQTT protocol using below pipelines.

Server pipeline. (ubuntu 20.04) with MQTT broker
$ gst-launch-1.0 tensor_query_serversrc connect-type=MQTT port= dest-port=1883 topic=example ! other/tensors,num_tensors=1,dimensions=3:640:480:1,types=uint8,framerate=30/1 ! tensor_query_serversink connet-type=MQTT

Client pipeline. (ubuntu 20.04)
$ gst-launch-1.0 v4l2src ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=RGB,framerate=30/1 !
tensor_converter ! tensor_query_client connect-type=MQTT dest-host=<server==broker_ip> dest-port=1883 topic=example ! tensor_decoder mode=direct_video ! videoconvert ! ximagesink

I tried TCP, simlilarly
The error I got "Failed to connect host ip:port in client.

[gichan-jang]

Hello, @Sungbin710 .
I revised some things that confuse users after checking your question. Thanks.

are there any other examples between different devices using tensor_query_* API?

I updated the video that connection between different devices. Please check it.
You just change dest-host, 'dest-port' when you use tensor_query::hybrid.
nnstreamer/nnstreamer-example#294

Server pipeline. (ubuntu 20.04) with MQTT broker
$ gst-launch-1.0 tensor_query_serversrc connect-type=MQTT port= dest-port=1883 topic=example ! other/tensors,num_tensors=1,dimensions=3:640:480:1,types=uint8,framerate=30/1 ! tensor_query_serversink connet-type=MQTT

Please update connect-type from MQTT to HYBRID. tensor_query don't send data using MQTT. I removed this enum.(#3992)

dest-host is written tcp://localhost, it needs protocol prefix like that?

You can use 127.0.0.1 instead of tcp://localhost for local host.

[sungbin-lee]

Thank you for your reply.
I'm still trying to do this. And I have some questions.

1. HYBRID, How it works?
   gst-inspect-1.0 tensor_query_client
   

I understand what HYBRID can do, however I wonder we don't have to use mqtt protocol when we use mqtt broker.
HYBRID has no relevance with mqtt protocol? HYBRID is just implented with mqtt broker by using TCP? I have a little confused.

2. TCP connect

Server pipeline. (ubuntu 20.04)
$ gst-launch-1.0 tensor_query_serversrc connect-type=TCP dest-port=<client_port> port=<server_port> ! other/tensors,num_tensors=1,dimensions=3:640:480:1,types=uint8,framerate=30/1 ! tensor_query_serversink connet-type=TCP

Client pipeline. (ubuntu 20.04)
$ gst-launch-1.0 v4l2src ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=RGB,framerate=30/1 !
tensor_converter ! tensor_query_client connect-type=TCP dest-host= dest-port=<server_port> port=<client_port> ! tensor_decoder mode=direct_video ! videoconvert ! ximagesink

I got same errors.

[gichan-jang]

HYBRID, How it works?

1. Servers connect to MQTT broker.
2. Servers publish the information required for TCP connection (IP and port) with topic (For examaple, object_deteciton).
3. Client connects to MQTT broker and get the connection information of the server from the MQTT broker.
4. Client connect to the available server and send the data.
   In summary, through MQTT, the information necessary for the connection between the server and the client is exchanged, and the data is transmitted directly.

Server pipeline. (ubuntu 20.04)
$ gst-launch-1.0 tensor_query_serversrc connect-type=TCP dest-port=<client_port> port=<server_port> ! other/tensors,num_tensors=1,dimensions=3:640:480:1,types=uint8,framerate=30/1 ! tensor_query_serversink connet-type=TCP
Client pipeline. (ubuntu 20.04)
$ gst-launch-1.0 v4l2src ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=RGB,framerate=30/1 !
tensor_converter ! tensor_query_client connect-type=TCP dest-host= dest-port=<server_port> port=<client_port> ! tensor_decoder mode=direct_video ! videoconvert ! ximagesink
I got same errors.

You use wrong properties in your query elements.
$ gst-launch-1.0 tensor_query_serversrc connect-type=TCP host=<server_address> port=<server_port> ! other/tensors,num_tensors=1,dimensions=3:640:480:1,types=uint8,framerate=30/1 ! tensor_query_serversink connet-type=TCP
Client pipeline. (ubuntu 20.04)
$ gst-launch-1.0 v4l2src ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=RGB,framerate=30/1 !
tensor_converter ! tensor_query_client connect-type=TCP dest-host=<server_address> dest-port=<server_port> host=<client_address> port=<client_port> ! tensor_decoder mode=direct_video ! videoconvert ! ximagesink

[sungbin-lee]

I'm leaving a message to check if it's correct.
Thanks for your help, I did Among-device AI with connect-type=TCP by using tensor_query_*

Now I am trying to implement connect-type=HYBRID.
Although tensor_query_* API supports TCP/UDP (not support MQTT), it uses mqtt broker.

I did it after changing Enable MQTT OFF to ON in CMakeListst.txt

Is it correct that connect-type=HYBRID needs paho-mqtt library because it uses mqtt broker?

And, as you said, mqtt protocol is not suppored in tensor_query_* API yet.
Is there any plan to support it?

[gichan-jang]

Thanks for your help, I did Among-device AI with connect-type=TCP by using tensor_query_*

Nice!

Is it correct that connect-type=HYBRID needs paho-mqtt library because it uses mqtt broker?

Right. As replied in the previous comment, MQTT and TCP are used in connect-type=HYBRID. So MQTT connection is needed.
In the case of TCP connection, MQTT is not required.

And, as you said, mqtt protocol is not suppored in tensor_query_* API yet.
Is there any plan to support it?

We are not considering transmitting data from tensor_query through MQTT. Because it's inefficient for sending the large data.
If you want to pub/sub with MQTT, you can use mqttsrc and mqttsink. (https://github.com/nnstreamer/nnstreamer/tree/main/gst/mqtt)