v2.1.3 Gremsy Vio camera payload fails to download camera definition due to "unsupported protocol" error #192

rayw-dronesense · 2024-11-26T17:41:38Z

2024-11-26 12:38:58.851 20122-20563 Mavsdk                  com.dronesense.pilot.blue            I  Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-11-26 12:38:58.852 20122-20563 Mavsdk                  com.dronesense.pilot.blue            I  Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-11-26 12:38:58.853 20122-20563 Mavsdk                  com.dronesense.pilot.blue            E  Error while downloading text, curl error code: Unsupported protocol
2024-11-26 12:38:58.853 20122-20563 Mavsdk                  com.dronesense.pilot.blue            E  Failed to download camera definition.
2024-11-26 12:38:58.854 20122-20563 Mavsdk                  com.dronesense.pilot.blue            I  Downloaded file, result Error
2024-11-26 12:38:58.854 20122-20563 Mavsdk                  com.dronesense.pilot.blue            D  Failed to fetch camera definition!

The text was updated successfully, but these errors were encountered:

JonasVautherin · 2024-11-26T22:58:34Z

We build curl without ssl, I think it won't support https: https://github.com/mavlink/MAVSDK/blob/main/third_party/curl/CMakeLists.txt#L12

Would you have a way to use http instead?

A weird thing in your case is that the camera definition is downloaded from the internet. Shouldn't it be served by the drone? Or is it somehow a simulator setup?

julianoes · 2024-11-27T00:16:37Z

I tried to add openssl with https support in mavlink/MAVSDK#2386 and I gave up trying to get it building for all platforms. If you want to waste a few days staring at CI playing with dependencies, feel free. I find it utterly frustrating.

rayw-dronesense · 2024-11-27T15:10:38Z

@JonasVautherin

We build curl without ssl, I think it won't support https: https://github.com/mavlink/MAVSDK/blob/main/third_party/curl/CMakeLists.txt#L12

Would you have a way to use http instead?

A weird thing in your case is that the camera definition is downloaded from the internet. Shouldn't it be served by the drone? Or is it somehow a simulator setup?

It's a real drone and payload. Nope, they for some reason decided it was a good idea to have the file fetched from Github every time; here's their documentation: https://docs.gremsy.com/payloads/vio/camera-setting-menu/camera-definition-file-download - pretty sure I have no way of changing this behavior...

For generate the camera setting menu, the QGroundControl app need to download the camera definition file from Gremsy github server. The download path will be sent by the Vio automatically.

@julianoes

I tried to add openssl with https support in mavlink/MAVSDK#2386 and I gave up trying to get it building for all platforms. If you want to waste a few days staring at CI playing with dependencies, feel free. I find it utterly frustrating.

Oh boy...sorry to hear about that. By any chance, were we able to get it to work on Android at least?

julianoes · 2024-11-28T00:04:02Z

Pretty sure Android was one of the broken ones:
https://github.com/mavlink/MAVSDK/actions/runs/11588668231/job/32262783180#step:6:1416

Is the file available with http though? In that case, you can just intercept (but not in Java) the message containing the URL and change https to http.

Or what we could also try is to just parse https and use http to download. If it works good, otherwise no hard done, I'd think...

JonasVautherin · 2024-11-28T20:53:39Z

Whoops it got resolved automatically because I mentioned it in the PR. Let me reopen it.

JonasVautherin · 2024-11-28T21:01:27Z

Trying to backport it to v2.12 so that we can maybe release it before v3: mavlink/MAVSDK#2454

rayw-dronesense · 2024-12-03T14:54:34Z

@julianoes

Pretty sure Android was one of the broken ones:
https://github.com/mavlink/MAVSDK/actions/runs/11588668231/job/32262783180#step:6:1416

That's unfortunate. Thanks for the info!

Is the file available with http though? In that case, you can just intercept (but not in Java) the message containing the URL and change https to http.

Or what we could also try is to just parse https and use http to download. If it works good, otherwise no hard done, I'd think...

Hmm... I don't think Github lets you download using HTTP anymore; here's what happens when I take that URL and do a wget with "HTTP" instead of "HTTPS":

$ wget http://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
--2024-12-03 09:50:55--  http://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
Resolving github.com (github.com)... 140.82.112.4
Connecting to github.com (github.com)|140.82.112.4|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml [following]
--2024-12-03 09:50:56--  https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
Connecting to github.com (github.com)|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/697574913/7809a88c-be05-4ff3-8f7e-5a72ae3e69ee?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241203T145056Z&X-Amz-Expires=300&X-Amz-Signature=953f4511aeadde975534734960e4d0aeedd6555d6f07228f74fd7e0b997a79d1&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dvio_camera_f1_def.xml&response-content-type=application%2Foctet-stream [following]
--2024-12-03 09:50:56--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/697574913/7809a88c-be05-4ff3-8f7e-5a72ae3e69ee?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241203T145056Z&X-Amz-Expires=300&X-Amz-Signature=953f4511aeadde975534734960e4d0aeedd6555d6f07228f74fd7e0b997a79d1&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dvio_camera_f1_def.xml&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16600 (16K) [application/octet-stream]
Saving to: ‘vio_camera_f1_def.xml’

@JonasVautherin Thanks!

JonasVautherin · 2024-12-03T15:30:25Z

Can you try with mavsdk_server:2.1.4? Just pushed it to MavenCentral. It should support HTTPS

rayw-dronesense · 2024-12-03T15:57:07Z

@JonasVautherin - Thanks for the updated release! With 2.1.4 it does get a bit further, but runs into this issue,

2024-12-03 10:55:50.389  2182-4091  Mavsdk                  com.dronesense.pilot.blue            I  Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-12-03 10:55:50.392  2182-4091  Mavsdk                  com.dronesense.pilot.blue            I  Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-12-03 10:55:50.452  2182-4091  Mavsdk                  com.dronesense.pilot.blue            E  Error while downloading text, curl error code: Problem with the SSL CA cert (path? access rights?)
2024-12-03 10:55:50.452  2182-4091  Mavsdk                  com.dronesense.pilot.blue            E  Failed to download camera definition.
2024-12-03 10:55:50.456  2182-4091  Mavsdk                  com.dronesense.pilot.blue            I  Downloaded file, result Error
2024-12-03 10:55:50.456  2182-4091  Mavsdk                  com.dronesense.pilot.blue            D  Failed to fetch camera definition!

JonasVautherin · 2024-12-03T16:28:51Z

Oh yeah, that's a good question: in order to properly leverage HTTPS, curl (openssl?) needs to have access to certificates. Usually they are found on the system, but I honestly don't know how we should do that with our static library 🤔.

An easy way would be to disable the verification (see e.g. here). We could justify it by saying that downloading a wrong camera_definition.xml is not a security issue 😅. Ideally we would have a way to feed certificates to mavsdk_server though?

Would you mind looking into libcurl to see if there is a way to give it certificates? Apparently the executable has some options for that (see here), but in our case we use libcurl (not the executable). The next question would be whether we can read system certificates on Android...

I had not realized before, but it isn't as easy as just enabling https in curl 🙈.

EDIT: here I see stuff like this:

    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 1);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1);
    curl_easy_setopt(curl, CURLOPT_CAINFO, "./ca.crt");

So if we had read access to the system certificates on Android, we could maybe just use that? I checked on my device and I have certificates in /system/etc/security/cacerts, for instance.

JonasVautherin · 2024-12-03T16:41:17Z

Oh maybe we could just set CURLOPT_CAPATH to /system/etc/security/cacerts on Android (with an ifdef in mavsdk)?

And it seems like this answer does it at build time!

I got this to work on Android by recompiling libcurl and configuring the default search path for certificates. This can be done by passing the option:

--with-ca-path=/system/etc/security/cacerts to ./configure

or

-DCURL_CA_PATH=/system/etc/security/cacerts to cmake

JonasVautherin · 2024-12-03T16:51:35Z

@rayw-dronesense: would you be able to try this PR? mavlink/MAVSDK#2459

rayw-dronesense · 2024-12-03T16:54:47Z

@JonasVautherin - Trying now...

rayw-dronesense · 2024-12-03T16:56:25Z

@JonasVautherin - I guess it makes it further now... but,

2024-12-03 11:56:06.591  6622-7070  Mavsdk                  com.dronesense.pilot.blue            I  Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-12-03 11:56:06.591  6622-7070  Mavsdk                  com.dronesense.pilot.blue            I  Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-12-03 11:56:06.671  6622-7070  Mavsdk                  com.dronesense.pilot.blue            E  Error while downloading text, curl error code: SSL peer certificate or SSH remote key was not OK
2024-12-03 11:56:06.673  6622-7070  Mavsdk                  com.dronesense.pilot.blue            E  Failed to download camera definition.
2024-12-03 11:56:06.679  6622-7070  Mavsdk                  com.dronesense.pilot.blue            I  Downloaded file, result Error

rayw-dronesense · 2024-12-03T17:06:48Z

Gonna try doing a clean and rebuild for good measure.

JonasVautherin · 2024-12-03T17:16:24Z

Does your device have /system/etc/security/cacerts?

rayw-dronesense · 2024-12-03T17:18:00Z

Does your device have /system/etc/security/cacerts?

Yes it does,

~ % adb shell ls /system/etc/security/cacerts
00673b5b.0
02756ea4.0
02b73561.0
03f2b8cf.0
04f60c28.0
052e396b.0
08aef7bb.0
0d5a4e1c.0
0d69c7e1.0
10531352.0
111e6273.0
119afc2e.0
124bbd54.0
12d55845.0
1676090a.0
17b51fe6.0
1dac3003.0
1dcd6f4c.0
1df5a75f.0
1e1eab7c.0
1e8e7201.0
1eb37bdf.0
1f58a078.0
21855f49.0
219d9499.0
23f4c490.0
262ba90f.0
27af790d.0
2add47b6.0
2d9dafe4.0
2fa87019.0
33815e15.0
33815e15.1
343eb6cb.0
35105088.0
3929ec9f.0
399e7759.0
3a3b02ce.0
3ad48a91.0
3c58f906.0
3c6676aa.0
3c860d51.0
3c9a4d3b.0
3d441de8.0
3e7271e8.0
40dc992e.0
418595b9.0
450c6e38.0
455f1b52.0
48a195d8.0
4be590e0.0
4e18c148.0
5046c355.0
524d9b43.0
52b525c7.0
559f7c71.0
57692373.0
58a44af1.0
5a250ea7.0
5a3f0ff8.0
5cf9d536.0
5e4e69e7.0
5f47b495.0
60afe812.0
6187b673.0
63a2c897.0
6645de82.0
67495436.0
69105f4f.0
6e8bf996.0
6fcc125d.0
72f369af.0
75680d2e.0
76579174.0
7672ac4b.0
7999be0d.0
7a819ef2.0
7d453d8f.0
81b9768f.0
82223c44.0
8470719d.0
85cde254.0
86212b19.0
87753b0d.0
882de061.0
89c02a45.0
8d6437c3.0
91739615.0
9282e51c.0
9339512a.0
9479c8c3.0
9576d26b.0
95aff9e3.0
961f5451.0
9685a493.0
9772ca32.0
9ab62355.0
9c3323d4.0
9d6523ce.0
9dbefe7b.0
9f533518.0
a0bc6fbb.0
a2c66da8.0
a2df7ad7.0
a3896b44.0
a7605362.0
a7d2cf64.0
a81e292b.0
ab5346f4.0
aeb67534.0
b0ed035a.0
b0f3e76e.0
b3fb433b.0
b7db1890.0
b872f2b4.0
bc3f2570.0
bdacca6f.0
bf64f35b.0
c491639e.0
c51c224c.0
c7e2a638.0
c90bc37d.0
cb156124.0
cb1c3204.0
ccc52f49.0
cf701eeb.0
d06393bb.0
d16a5865.0
d18e9066.0
d4c339cb.0
d5727d6a.0
d59297b8.0
d66b55d9.0
d6e6eab9.0
d7746a63.0
d8317ada.0
dbc54cab.0
dc99f41e.0
dfc0fe80.0
e268a4c5.0
e442e424.0
e48193cf.0
e60bf0c0.0
e775ed2d.0
e8651083.0
ea169617.0
ed39abd0.0
ee7cd6fb.0
ee90b008.0
f61bff45.0
f80cc7f6.0
fac084d7.0
facacbc6.0
fb126c6d.0
fde84897.0
ff783690.0

rayw-dronesense · 2024-12-03T17:30:02Z

Update: clean and rebuild didn't change anything. Still got Error while downloading text, curl error code: SSL peer certificate or SSH remote key was not OK

rayw-dronesense · 2024-12-03T17:42:45Z

Just for fun I tried disabling SSL verification per the above post...and that got it past the download definition no problem. But getPossibleSettings... errors out,

2024-12-03 12:41:55.354 21290-21576 Mavsdk                  com.dronesense.pilot.blue            I  Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-12-03 12:41:55.357 21290-21576 Mavsdk                  com.dronesense.pilot.blue            I  Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-12-03 12:41:55.514 21290-21576 Mavsdk                  com.dronesense.pilot.blue            I  Downloaded file, result Success
2024-12-03 12:41:55.514 21290-21576 Mavsdk                  com.dronesense.pilot.blue            D  Successfully loaded camera definition
2024-12-03 12:41:55.514 21290-21576 Mavsdk                  com.dronesense.pilot.blue            E  tinyxml2::Parse failed: Error=XML_ERROR_EMPTY_DOCUMENT ErrorID=13 (0xd) Line number=0
2024-12-03 12:41:55.601 21290-21408 Mavsdk                  com.dronesense.pilot.blue            E  Unknown setting to get: CAM_MODE
2024-12-03 12:41:55.601 21290-21408 Mavsdk                  com.dronesense.pilot.blue            E  Unknown setting to set: CAM_MODE

rayw-dronesense · 2024-12-03T17:50:35Z

Yeah okay, looks like the error is correct - it's getting an empty string for some reason.

        std::thread([this, camera_information]() {
            std::string content{};
            const auto result = fetch_camera_definition(camera_information, content);

            if (result == Camera::Result::Success) {
                LogDebug() << "Successfully loaded camera definition";
                LogDebug() << "RAWR: THE CONTENT IS " << content;

The output is empty,

2024-12-03 12:49:40.988  8277-8568  Mavsdk                  com.dronesense.pilot.blue            D  RAWR: THE CONTENT IS 
2024-12-03 12:49:40.988  8277-8568  Mavsdk                  com.dronesense.pilot.blue            E  tinyxml2::Parse failed: Error=XML_ERROR_EMPTY_DOCUMENT ErrorID=13 (0xd) Line number=0

rayw-dronesense · 2024-12-03T18:31:22Z

Got it. Also had to enable "Follow Location" to allow it to handle redirects,

        curl_easy_setopt(curl.get(), CURLOPT_SSL_VERIFYPEER, 0L);
        curl_easy_setopt(curl.get(), CURLOPT_FOLLOWLOCATION, 1L);

Then it started to work,

2024-12-03 13:29:47.293  2230-4159  Mavsdk                  com.dronesense.pilot.blue            I  Downloaded file, result Success
2024-12-03 13:29:47.293  2230-4159  Mavsdk                  com.dronesense.pilot.blue            D  Successfully loaded camera definition
2024-12-03 13:29:47.293  2230-4159  Mavsdk                  com.dronesense.pilot.blue            D  RAWR: THE CONTENT IS <?xml version="1.0" encoding="UTF-8" ?>
                                                                                                    <mavlinkcamera>
                                                                                                        <definition version="5">
                                                                                                            <model>Vio Payload</model>
                                                                                                            <vendor>Gremsy</vendor>
                                                                                                        </definition>
                                                                                                        <parameters>
                                                                                                            <!-- control = 0 tells us this should not create an automatic UI control -->
                                                                                                            <parameter name="CAM_MODE" type="uint32" default="1" control="0">
                                                                                                                <description>Camera Mode</description>

So now it's just a matter of having the certs validate the proper way I guess

JonasVautherin · 2024-12-03T19:26:58Z

Did you try with CURLOPT_SSL_VERIFYPEER=1 and CURLOPT_FOLLOWLOCATION=1? Maybe the error "SSL peer certificate or SSH remote key was not OK" happens on the empty string because the redirection was not followed?

rayw-dronesense · 2024-12-03T19:30:57Z

Did you try with CURLOPT_SSL_VERIFYPEER=1 and CURLOPT_FOLLOWLOCATION=1? Maybe the error "SSL peer certificate or SSH remote key was not OK" happens on the empty string because the redirection was not followed?

Just tried it. Same Error while downloading text, curl error code: SSL peer certificate or SSH remote key was not OK error

JonasVautherin mentioned this issue Nov 28, 2024

Enable ssl in curl mavlink/MAVSDK#2453

Merged

JonasVautherin closed this as completed in mavlink/MAVSDK#2453 Nov 28, 2024

JonasVautherin reopened this Nov 28, 2024

JonasVautherin mentioned this issue Dec 3, 2024

curl: point to system CA at build time on Android mavlink/MAVSDK#2459

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2.1.3 Gremsy Vio camera payload fails to download camera definition due to "unsupported protocol" error #192

v2.1.3 Gremsy Vio camera payload fails to download camera definition due to "unsupported protocol" error #192

rayw-dronesense commented Nov 26, 2024

JonasVautherin commented Nov 26, 2024

julianoes commented Nov 27, 2024

rayw-dronesense commented Nov 27, 2024 •

edited

Loading

julianoes commented Nov 28, 2024

JonasVautherin commented Nov 28, 2024

JonasVautherin commented Nov 28, 2024 •

edited

Loading

rayw-dronesense commented Dec 3, 2024 •

edited

Loading

JonasVautherin commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

JonasVautherin commented Dec 3, 2024 •

edited

Loading

JonasVautherin commented Dec 3, 2024 •

edited

Loading

JonasVautherin commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

JonasVautherin commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024 •

edited

Loading

JonasVautherin commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

v2.1.3 Gremsy Vio camera payload fails to download camera definition due to "unsupported protocol" error #192

v2.1.3 Gremsy Vio camera payload fails to download camera definition due to "unsupported protocol" error #192

Comments

rayw-dronesense commented Nov 26, 2024

JonasVautherin commented Nov 26, 2024

julianoes commented Nov 27, 2024

rayw-dronesense commented Nov 27, 2024 • edited Loading

julianoes commented Nov 28, 2024

JonasVautherin commented Nov 28, 2024

JonasVautherin commented Nov 28, 2024 • edited Loading

rayw-dronesense commented Dec 3, 2024 • edited Loading

JonasVautherin commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

JonasVautherin commented Dec 3, 2024 • edited Loading

JonasVautherin commented Dec 3, 2024 • edited Loading

JonasVautherin commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

JonasVautherin commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024 • edited Loading

JonasVautherin commented Dec 3, 2024

rayw-dronesense commented Dec 3, 2024

rayw-dronesense commented Nov 27, 2024 •

edited

Loading

JonasVautherin commented Nov 28, 2024 •

edited

Loading

rayw-dronesense commented Dec 3, 2024 •

edited

Loading

JonasVautherin commented Dec 3, 2024 •

edited

Loading

JonasVautherin commented Dec 3, 2024 •

edited

Loading

rayw-dronesense commented Dec 3, 2024 •

edited

Loading