GDK 0.68.0 Windows build crashes in `GA_connect`

[pavel-kokolemin]

GDK 0.68.0 Windows build crashes in GA_connect when used with multi-sig accounts (single-sig accounts work fine).

I have reproduced the crash in a minimal program (main.cpp).
To build:

x86_64-w64-mingw32-c++ main.cpp -o gdk_crash -I/root/gdk/include -I/root/gdk/build-windows-mingw-w64/src -I/prebuild/mingw-w64/nlohmann_json/include /root/gdk/build-windows-mingw-w64/libgreenaddress.dll.a

Copy libgreenaddress.dll and required mingw libs:

/root/gdk/build-windows-mingw-w64/libgreenaddress.dll
/usr/lib/gcc/x86_64-w64-mingw32/10-posix/libssp-0.dll
/usr/lib/gcc/x86_64-w64-mingw32/10-posix/libstdc++-6.dll
/usr/lib/gcc/x86_64-w64-mingw32/10-posix/libgcc_s_seh-1.dll
/usr/x86_64-w64-mingw32/lib/libwinpthread-1.dll

Then run the binary (gdk_crash.exe):

GA_init...
send: {"datadir":"./","log_level":"debug"}
34.413 DEBUG - populated the cache with 0 files
34.413 DEBUG - loading 0 cache files
34.414 INFO - `init` took 4.43793ms
GA_create_session...
GA_connect...
send: {"name":"liquid"}
[2023-10-05 09:27:34.409299] [0x00001688] [info]    change_state_to: requesting state connected
[2023-10-05 09:27:34.409299] [0x00002204] [info]    net: thread started for gdk version 
[2023-10-05 09:27:34.426265] [0x00002204] [info]    net: desired connected actual disconnected
[2023-10-05 09:27:34.426265] [0x00002204] [info]    net: connect to wss://green-liquid-mainnet.blockstream.com/v2/ws
[2023-10-05 09:27:39.444271] [0x000028c4] [info]    Error getting remote endpoint: system:10009 (The file handle supplied is not valid)
[2023-10-05 09:27:39.459714] [0x000028c4] [info]    asio async_shutdown error: asio.ssl:336462100 (asio.ssl error)
[2023-10-05 09:27:39.459714] [0x00002204] [info]    transport connect ignoring exception:Dynamic exception type: boost::wrapexcept<autobahn::network_error>
std::exception::what: failed to connect
[2023-10-05 09:27:39.475348] [0x00002204] [info]    ignoring exception:Dynamic exception type: boost::wrapexcept<std::exception>
std::exception::what: std::exception
[2023-10-05 09:27:39.475348] [0x00002204] [info]    ignoring exception:Dynamic exception type: boost::wrapexcept<autobahn::protocol_error>
std::exception::what: session already stopped
[2023-10-05 09:27:39.490892] [0x00002204] [info]    ignoring exception:network transport already disconnected
[2023-10-05 09:27:39.490892] [0x00002204] [debug]   waiting for connection to die
[2023-10-05 09:27:39.506534] [0x00002204] [debug]   connection is dead
[2023-10-05 09:27:39.506534] [0x00002204] [info]    net: backing off for 2s
Segmentation fault

I tried building on Debian 12 and Debian 11, same crash happens.
It works fine on Linux:

GA_init...
send: {"datadir":"./","log_level":"debug"}
11.086 DEBUG - populated the cache with 0 files
11.086 DEBUG - loading 0 cache files
11.086 INFO - `init` took 279.464µs
GA_create_session...
GA_connect...
send: {"name":"liquid"}
[2023-10-05 10:40:11.088242] [0x00007fb3000bc640] [debug]   endpoint constructor
[2023-10-05 10:40:11.088272] [0x00007fb3000bc640] [debug]   client constructor
[2023-10-05 10:40:11.088307] [0x00007fb3000bc640] [debug]   set_pong_timeout_handler
[2023-10-05 10:40:11.088319] [0x00007fb2f87216c0] [info]    net: thread started for gdk version 
[2023-10-05 10:40:11.088335] [0x00007fb3000bc640] [debug]   asio::init_asio
[2023-10-05 10:40:11.088582] [0x00007fb3000bc640] [info]    change_state_to: requesting state connected
[2023-10-05 10:40:11.088610] [0x00007fb2f87216c0] [info]    net: desired connected actual disconnected
[2023-10-05 10:40:11.088617] [0x00007fb2f87216c0] [info]    net: connect to wss://green-liquid-mainnet.blockstream.com/v2/ws
[2023-10-05 10:40:11.088711] [0x00007fb2f87216c0] [debug]   set_open_handler
[2023-10-05 10:40:11.088721] [0x00007fb2f87216c0] [debug]   set_close_handler
[2023-10-05 10:40:11.088725] [0x00007fb2f87216c0] [debug]   set_fail_handler
[2023-10-05 10:40:11.088735] [0x00007fb2f87216c0] [debug]   set_message_handler
[2023-10-05 10:40:11.088846] [0x00007fb2f87216c0] [debug]   create_connection
[2023-10-05 10:40:11.088922] [0x00007fb2f87216c0] [debug]   asio con transport constructor
[2023-10-05 10:40:11.089015] [0x00007fb2f87216c0] [debug]   connection constructor
[2023-10-05 10:40:11.089078] [0x00007fb2f87216c0] [debug]   transport::asio::init
[2023-10-05 10:40:11.090339] [0x00007fb2f87216c0] [debug]   starting async DNS resolve for green-liquid-mainnet.blockstream.com:443
[2023-10-05 10:40:11.262926] [0x00007fb2f8f226c0] [debug]   Async DNS resolve successful. Results: 34.110.237.101:443 
[2023-10-05 10:40:11.263000] [0x00007fb2f8f226c0] [debug]   Starting async connect
[2023-10-05 10:40:11.263777] [0x00007fb2f8f226c0] [debug]   asio handle_resolve_timeout timer cancelled
[2023-10-05 10:40:11.393245] [0x00007fb2f8f226c0] [debug]   Async connect to 34.110.237.101:443 successful.
[2023-10-05 10:40:11.393367] [0x00007fb2f8f226c0] [debug]   connection start
[2023-10-05 10:40:11.393451] [0x00007fb2f8f226c0] [debug]   asio connection init
[2023-10-05 10:40:11.393601] [0x00007fb2f8f226c0] [debug]   asio connection handle pre_init
[2023-10-05 10:40:11.393617] [0x00007fb2f8f226c0] [debug]   asio connection post_init
[2023-10-05 10:40:11.395008] [0x00007fb2f8f226c0] [debug]   asio handle_connect_timeout timer cancelled
[2023-10-05 10:40:11.526730] [0x00007fb2f8f226c0] [debug]   Checking for pinned certificate
[2023-10-05 10:40:11.526924] [0x00007fb2f8f226c0] [debug]   Found pinned certificate 64e286b76063602a372efd60cde8db2656a49ee15e84254b3d6eb5fe38f4288b
[2023-10-05 10:40:11.527622] [0x00007fb2f8f226c0] [debug]   asio connection handle_post_init
[2023-10-05 10:40:11.527676] [0x00007fb2f8f226c0] [debug]   connection handle_transport_init
[2023-10-05 10:40:11.527824] [0x00007fb2f8f226c0] [debug]   connection send_http_request
[2023-10-05 10:40:11.528297] [0x00007fb2f8f226c0] [debug]   Raw Handshake request:
GET /v2/ws HTTP/1.1
Connection: Upgrade
Host: green-liquid-mainnet.blockstream.com
Sec-WebSocket-Key: KLziMDuB262iqt5fuv6PPQ==
Sec-WebSocket-Protocol: wamp.2.msgpack
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: WebSocket++/0.8.1


[2023-10-05 10:40:11.529058] [0x00007fb2f8f226c0] [debug]   asio post init timer cancelled
[2023-10-05 10:40:11.529366] [0x00007fb2f8f226c0] [debug]   handle_send_http_request
[2023-10-05 10:40:11.529410] [0x00007fb2f8f226c0] [debug]   asio async_read_at_least: 1
[2023-10-05 10:40:11.829420] [0x00007fb2f8f226c0] [debug]   asio con handle_async_read
[2023-10-05 10:40:11.829552] [0x00007fb2f8f226c0] [debug]   handle_read_http_response
[2023-10-05 10:40:11.829968] [0x00007fb2f8f226c0] [debug]   Raw response: HTTP/1.1 101 Switching Protocols
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Connection: Upgrade
Date: Thu, 05 Oct 2023 06:40:11 GMT
Sec-WebSocket-Accept: obqBne+fO1l0JKwY1nVxYdO6oQo=
Sec-WebSocket-Protocol: wamp.2.msgpack
Server: openresty/1.21.4.1
Upgrade: websocket
Via: 1.1 google


[2023-10-05 10:40:11.830353] [0x00007fb2f8f226c0] [debug]   p = 0 bytes transferred = 0
[2023-10-05 10:40:11.830379] [0x00007fb2f8f226c0] [debug]   asio async_read_at_least: 1
[2023-10-05 10:40:11.830524] [0x00007fb2f8f226c0] [debug]   open handshake timer cancelled
[2023-10-05 10:40:11.831771] [0x00007fb2f8f226c0] [debug]   connection send
[2023-10-05 10:40:11.831972] [0x00007fb2f8f226c0] [debug]   write_push: message count: 1 buffer size: 125
[2023-10-05 10:40:11.832703] [0x00007fb2f8f226c0] [debug]   write_pop: message count: 0 buffer size: 0
[2023-10-05 10:40:11.833118] [0x00007fb2f8f226c0] [debug]   connection handle_write_frame
[2023-10-05 10:40:12.000661] [0x00007fb2f8f226c0] [debug]   asio con handle_async_read
[2023-10-05 10:40:12.000807] [0x00007fb2f8f226c0] [debug]   p = 0 bytes transferred = 918
[2023-10-05 10:40:12.000827] [0x00007fb2f8f226c0] [debug]   calling consume with 918 bytes
[2023-10-05 10:40:12.001189] [0x00007fb2f8f226c0] [debug]   Processing Bytes: 82 7E 03 92 93 02 CF 00 01 0D 68 83 2D 39 C3 8B A9 78 5F 63 62 5F 6E 6F 64 65 D9 2D 6C 69 71 75 69 64 2D 6D 61 69 6E 6E 65 74 2D 62 61 63 6B 65 6E 64 2D 65 75 72 6F 70 65 2D 77 65 73 74 32 2D 76 37 37 63 2D 32 34 38 32 AB 78 5F 63 62 5F 77 6F 72 6B 65 72 A9 77 6F 72 6B 65 72 30 30 31 A9 78 5F 63 62 5F 70 65 65 72 B4 74 63 70 34 3A 31 32 37 2E 30 2E 30 2E 31 3A 33 32 38 31 30 A8 78 5F 63 62 5F 70 69 64 CD 09 E1 A5 72 65 61 6C 6D A6 72 65 61 6C 6D 31 A6 61 75 74 68 69 64 BD 34 4D 53 50 2D 33 50 54 57 2D 59 54 46 4D 2D 47 48 46 48 2D 4C 57 35 37 2D 45 4E 43 59 A8 61 75 74 68 72 6F 6C 65 A9 61 6E 6F 6E 79 6D 6F 75 73 AA 61 75 74 68 6D 65 74 68 6F 64 A9 61 6E 6F 6E 79 6D 6F 75 73 AC 61 75 74 68 70 72 6F 76 69 64 65 72 A7 64 79 6E 61 6D 69 63 A9 61 75 74 68 65 78 74 72 61 84 A9 78 5F 63 62 5F 6E 6F 64 65 D9 2D 6C 69 71 75 69 64 2D 6D 61 69 6E 6E 65 74 2D 62 61 63 6B 65 6E 64 2D 65 75 72 6F 70 65 2D 77 65 73 74 32 2D 76 37 37 63 2D 32 34 38 32 AB 78 5F 63 62 5F 77 6F 72 6B 65 72 A9 77 6F 72 6B 65 72 30 30 31 A9 78 5F 63 62 5F 70 65 65 72 B4 74 63 70 34 3A 31 32 37 2E 30 2E 30 2E 31 3A 33 32 38 31 30 A8 78 5F 63 62 5F 70 69 64 CD 09 E1 A5 72 6F 6C 65 73 82 A6 62 72 6F 6B 65 72 81 A8 66 65 61 74 75 72 65 73 8A B8 70 75 62 6C 69 73 68 65 72 5F 69 64 65 6E 74 69 66 69 63 61 74 69 6F 6E C3 BA 70 61 74 74 65 72 6E 5F 62 61 73 65 64 5F 73 75 62 73 63 72 69 70 74 69 6F 6E C3 B0 73 65 73 73 69 6F 6E 5F 6D 65 74 61 5F 61 70 69 C3 B5 73 75 62 73 63 72 69 70 74 69 6F 6E 5F 6D 65 74 61 5F 61 70 69 C3 BD 73 75 62 73 63 72 69 62 65 72 5F 62 6C 61 63 6B 77 68 69 74 65 5F 6C 69 73 74 69 6E 67 C3 B3 70 75 62 6C 69 73 68 65 72 5F 65 78 63 6C 75 73 69 6F 6E C3 B7 73 75 62 73 63 72 69 70 74 69 6F 6E 5F 72 65 76 6F 63 61 74 69 6F 6E C3 B4 70 61 79 6C 6F 61 64 5F 74 72 61 6E 73 70 61 72 65 6E 63 79 C3 BC 70 61 79 6C 6F 61 64 5F 65 6E 63 72 79 70 74 69 6F 6E 5F 63 72 79 70 74 6F 62 6F 78 C3 AF 65 76 65 6E 74 5F 72 65 74 65 6E 74 69 6F 6E C3 A6 64 65 61 6C 65 72 81 A8 66 65 61 74 75 72 65 73 8B B5 63 61 6C 6C 65 72 5F 69 64 65 6E 74 69 66 69 63 61 74 69 6F 6E C3 BA 70 61 74 74 65 72 6E 5F 62 61 73 65 64 5F 72 65 67 69 73 74 72 61 74 69 6F 6E C3 B0 73 65 73 73 69 6F 6E 5F 6D 65 74 61 5F 61 70 69 C3 B5 72 65 67 69 73 74 72 61 74 69 6F 6E 5F 6D 65 74 61 5F 61 70 69 C3 B3 73 68 61 72 65 64 5F 72 65 67 69 73 74 72 61 74 69 6F 6E C3 AE 63 61 6C 6C 5F 63 61 6E 63 65 6C 69 6E 67 C3 B8 70 72 6F 67 72 65 73 73 69 76 65 5F 63 61 6C 6C 5F 72 65 73 75 6C 74 73 C3 B7 72 65 67 69 73 74 72 61 74 69 6F 6E 5F 72 65 76 6F 63 61 74 69 6F 6E C3 B4 70 61 79 6C 6F 61 64 5F 74 72 61 6E 73 70 61 72 65 6E 63 79 C3 B2 74 65 73 74 61 6D 65 6E 74 5F 6D 65 74 61 5F 61 70 69 C3 BC 70 61 79 6C 6F 61 64 5F 65 6E 63 72 79 70 74 69 6F 6E 5F 63 72 79 70 74 6F 62 6F 78 C3 
[2023-10-05 10:40:12.001295] [0x00007fb2f8f226c0] [debug]   bytes left after consume: 0
[2023-10-05 10:40:12.001308] [0x00007fb2f8f226c0] [debug]   Complete message received. Dispatching
[2023-10-05 10:40:12.004569] [0x00007fb2f8f226c0] [debug]   asio async_read_at_least: 1
[2023-10-05 10:40:12.004737] [0x00007fb2f87216c0] [info]    net: connection successful
[2023-10-05 10:40:12.041028] [0x00007fb3000bc640] [info]    change_state_to: changed to connected
Success

main.cpp.txt

[jgriffiths]

Hi @pavel-kokolemin thanks for the report.

Can you confirm if this works for you with an earlier gdk version, and if so what is the last version that works?

[pavel-kokolemin]

Can you confirm if this works for you with an earlier gdk version, and if so what is the last version that works?

I have tried release_0.67.1 and some older versions and they do not crash:

GA_init...
send: {"datadir":"./","log_level":"debug"}
06.852 DEBUG - populated the cache with 0 files
06.852 DEBUG - loading 0 cache files
06.852 INFO - `init` took 973.43µs
GA_create_session...
GA_connect...
send: {"name":"liquid"}
[2023-10-05 12:02:06.851755] [0x00000cf0] [info]    net: thread started for gdk version
[2023-10-05 12:02:06.851755] [0x00000ba8] [info]    change_state_to: requesting state connected
[2023-10-05 12:02:06.867991] [0x00000cf0] [info]    net: desired connected actual disconnected
[2023-10-05 12:02:06.867991] [0x00000cf0] [info]    net: connect to wss://green-liquid-mainnet.blockstream.com/v2/ws
[2023-10-05 12:02:06.978625] [0x00000570] [debug]   Checking for pinned certificate
[2023-10-05 12:02:06.994263] [0x00000570] [debug]   Found pinned certificate 64e286b76063602a372efd60cde8db2656a49ee15e84254b3d6eb5fe38f4288b
TX message (125 octets) ...
TX message: hello ["realm1", {"authid":"","authmethods":[],"roles":{"subscriber":{},"publisher":{},"callee":{"features":{"call_timeout":true}},"caller":{"features":{"call_timeout":true}}}}]      
RX message received.
RX message: welcome [8796734805482752, {"x_cb_node":"liquid-mainnet-backend-europe-west2-v77c-2482","x_cb_worker":"worker001","x_cb_peer":"tcp4:127.0.0.1:40206","x_cb_pid":2529,"realm":"realm1","authid":"WXFV-EPA6-VH4U-XRA9-GC5V-FG3K","authrole":"anonymous","authmethod":"anonymous","authprovider":"dynamic","authextra":{"x_cb_node":"liquid-mainnet-backend-europe-west2-v77c-2482","x_cb_worker":"worker001","x_cb_peer":"tcp4:127.0.0.1:40206","x_cb_pid":2529},"roles":{"broker":{"features":{"publisher_identification":true,"pattern_based_subscription":true,"session_meta_api":true,"subscription_meta_api":true,"subscriber_blackwhite_listing":true,"publisher_exclusion":true,"subscription_revocation":true,"payload_transparency":true,"payload_encryption_cryptobox":true,"event_retention":true}},"dealer":{"features":{"caller_identification":true,"pattern_based_registration":true,"session_meta_api":true,"registration_meta_api":true,"shared_registration":true,"call_canceling":true,"progressive_call_results":true,"registration_revocation":true,"payload_transparency":true,"testament_meta_api":true,"payload_encryption_cryptobox":true}}}}]
[2023-10-05 12:02:07.295945] [0x00000cf0] [info]    net: connection successful
[2023-10-05 12:02:07.376252] [0x00000ba8] [info]    change_state_to: changed to connected
Success

So it looks like a regression in release_0.68.0.

[borgbyte]

@pavel-kokolemin can you try branch release_0.68_dev?

[pavel-kokolemin]

@pavel-kokolemin can you try branch release_0.68_dev?

@borgbyte work fine, thanks.

[borgbyte]

@pavel-kokolemin fyi 0.68.1 is in pre-release

[lightyear15]

Hello @pavel-kokolemin

even though we managed to fix your windows issue by reverting some changes, we are still investigating the root cause of the crash.

We are having hard times replicating it though. May I ask some questions?

• what windows version were you using?
• looking at the logs produced by the crashing process, it looks like greenaddress library was having some network issues

Error getting remote endpoint: system:10009 (The file handle supplied is not valid)

and following
do you remember your machine having some connectivity problems?

As last request,
can you please retest version 0.68.0 with your main.cpp file changed like this?

--- main.cpp.txt	2023-10-25 10:10:51.859571074 +0200
+++ main.cpp2.txt	2023-10-25 10:15:11.635713340 +0200
@@ -31,7 +31,8 @@
 
 int main() {
     std::cout << "GA_init..." << std::endl;
-    check(GA_init(json({{"datadir", "./"}, {"log_level", "debug"}})));
+    auto *jason = json({{"datadir", "./"}, {"log_level", "debug"}});
+    check(GA_init(jason));
 
     std::cout << "GA_create_session..." << std::endl;
     GA_session *session;
@@ -43,5 +44,8 @@
     std::cout << "GA_connect..." << std::endl;
     check(GA_connect(session, json({{"name", "liquid"}})));
 
+    check(GA_destroy_json(jason));
+    check(GA_destroy_session(session));
+
     std::cout << "Success" << std::endl;
 }

thanks

[pavel-kokolemin]

Hi @lightyear15

what windows version were you using?

I'm using Windows 11. The first time the crash was reported by our test user, so I doubt it's related to my Windows installation. And why do other builds work?

do you remember your machine having some connectivity problems?

No. The crash always happens and I have a good network connection.

can you please retest version 0.68.0 with your main.cpp file changed like this?

I applied the patch and tried 0.68.0 and the latest master branch (commit 5021276) and it still crashes. 0.68.2 works fine as I already reported.

We are having hard times replicating it though.

Can you send your libgreenaddress.dll build?

Thanks.

[lightyear15]

It seems on Windows10 the bug does not pop up.

The zip file attached contains the libgreenaddress.dll and the executable result of the build of my version of your main.cpp (the one patched according to comment above + some few more logs)
We tested this binary with windows11 as well and no crash has been reported
gdkwincrash.zip

[pavel-kokolemin]

I can't reproduce the crash either.
But it crashes:

• When running gdkcrash.exe from the bash shell (I use Git for Windows).
• In our wallet application.

While the GDK build without 52f2b2e commit works normally in all situations.
I will do some more testing and get back to you.
Thanks for your help.

[lightyear15]

I can't reproduce the crash either. But it crashes:

* When running gdkcrash.exe from the bash shell (I use Git for Windows).

* In our wallet application.

While the GDK build without 52f2b2e commit works normally in all situations. I will do some more testing and get back to you. Thanks for your help.

Thanks @pavel-kokolemin for your help:

I am not sure I understand the situation:

• does the crash happen with the binary I sent you last week?
• if it doesn't but you still see the crash when running your wallet or the test main.cpp applications built by yourself, there might be something wrong in your build process then

can you give me some insight on what your build instructions are?

Thanks

[pavel-kokolemin]

Hi @lightyear15
Sorry for the delay.
Please try to reproduce the crash with this:
main.cpp.txt
It crashes reliably for me.

I built it as before:
GDK:

./tools/build.sh --mingw-w64

And gdk_crash.exe:

x86_64-w64-mingw32-c++ main.cpp -o gdk_crash -I/path/to/include/gdk -I/path/to/gdk/build-windows-mingw-w64/external_deps/nlohmann_json/include /path/to/gdk/build-windows-mingw-w64/libgreenaddress.dll.a

Copied gdk_crash.exe, libgreenaddress.dll and these mingw libs (from Debian 12):

/usr/lib/gcc/x86_64-w64-mingw32/12-posix/libstdc++-6.dll
/usr/lib/gcc/x86_64-w64-mingw32/12-posix/libgcc_s_seh-1.dll
/usr/x86_64-w64-mingw32/lib/libwinpthread-1.dll

I have tried:
latest master (with commit 52f2b2e): crash
release_0.68.0 (with commit 52f2b2e): crash
release_0.68.4 (without commit 52f2b2e): no crash

The same crash happened when I tried mingw from Debian 11.

[lightyear15]

Hi @pavel-kokolemin

can you please do one more round of tests using gdk from this branch?
https://github.com/Blockstream/gdk/tree/giulio_mingw_fix, commit 357ceb3

Thanks a lot for your invaluable help in testing and investigating this issue.
I hope this will finally put an end to this issue 🤞

[pavel-kokolemin]

Hi @lightyear15, thank you!

I tried this commit and it works a bit better (no crash after the first iteration), but it still crashes after about 10..20 iterations. And I checked release_0.68.4 again and there are no crashes after all 100 iterations.