How to use re-use properly?

[denravonska]

(sorry for the issue spam)

Do you have an example of how to do socket reuse properly? When I try it the first request seems to work fine:

info:1094 Program: Embeddable HTTP 1.0/1.1 client
info:1095 Version: 0.0.0
info:1096 Repo:    https://github.com/howerj/httpc
info:1097 Author:  Richard James Howe
info:1098 Email:   [email protected]
info:1099 Options: stk=128 tst=1 grw=1 log=1 cons=3 redirs=3 hmax=8192 sz=592
info:1103 License: The Unlicense (public domain)
info:521 domain:    212.183.159.230
info:522 port:      80
info:523 SSL:       false
info:526 path       /5MB.zip
debug:1312 state -- initial   -> open     
debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=0-1023' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
info:906 HEADER: HTTP/1.1 206 Partial Content/28
info:795 unknown field: Date: Fri, 22 Mar 2024 08:55:05 GMT
info:795 unknown field: Server: Apache
info:795 unknown field: Last-Modified: Mon, 02 Jun 2008 15:30:42 GMT
info:795 unknown field: ETag: "603fc-500000-44eb0adaf4c80"
info:739 Accept-Ranges: bytes
info:773 Content Length: 1024
info:795 unknown field: Cache-Control: no-tranform
info:795 unknown field: Content-Range: bytes 0-1023/5242880
info:795 unknown field: Keep-Alive: timeout=15, max=100
info:766 connection may be kept alive
info:795 unknown field: Content-Type: application/zip
info:922 header done
debug:1312 state -- recv-head -> recv-body
debug:1312 state -- recv-body -> done     

Subsequent requests on the same client and socket seem to be thrown out of sync:

debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=1024-2047' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
info:906 HEADER: HTTP/1.1 206 Partial Content/28
info:795 unknown field: Date: Fri, 22 Mar 2024 08:55:05 GMT
info:795 unknown field: Server: Apache
info:795 unknown field: Last-Modified: Mon, 02 Jun 2008 15:30:42 GMT
info:795 unknown field: ETag: "603fc-500000-44eb0adaf4c80"
info:739 Accept-Ranges: bytes
info:773 Content Length: 1024
info:795 unknown field: Cache-Control: no-tranform
info:795 unknown field: Content-Range: bytes 1024-2047/5242880
info:795 unknown field: Keep-Alive: timeout=15, max=99
info:766 connection may be kept alive
info:795 unknown field: Content-Type: application/zip
info:922 header done
debug:1312 state -- recv-head -> recv-body
debug:1312 state -- recv-body -> done     
debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=2048-3071' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
debug:218 malloc 0x55c7dc393570/256
debug:225 realloc 0x55c7dc393570/512
error:818 Got '\r' with no '\n'
error:904 protocol error (could not read first line)
debug:1312 state -- recv-head -> back-off 
debug:1312 state -- back-off  -> sleeps   
info:663 backing off for 1000 ms, retried 1
debug:1312 state -- sleeps    -> open     
debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=2048-3071' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
info:906 HEADER: HTTP/1.1 206 Partial Content/28
info:795 unknown field: Date: Fri, 22 Mar 2024 08:55:06 GMT
info:795 unknown field: Server: Apache
info:795 unknown field: Last-Modified: Mon, 02 Jun 2008 15:30:42 GMT
info:795 unknown field: ETag: "603fc-500000-44eb0adaf4c80"
info:739 Accept-Ranges: bytes
info:773 Content Length: 1024
info:795 unknown field: Cache-Control: no-tranform
info:795 unknown field: Content-Range: bytes 2048-3071/5242880
info:795 unknown field: Keep-Alive: timeout=15, max=100
info:766 connection may be kept alive
info:795 unknown field: Content-Type: application/zip
info:922 header done
debug:1312 state -- recv-head -> recv-body
debug:1312 state -- recv-body -> done     
debug:1312 state -- open      -> send-head
debug:634 custom header 'Range: bytes=3072-4095' added
info:642 GET  request complete
debug:1312 state -- send-head -> send-body
info:1035 no callback - nothing to do
debug:1312 state -- send-body -> recv-head
info:906 HEADER: ��Ļ!��x�p�i�륯�-[(9�n���^��$$:$�����pz�p����O���8��J�{n�r��O�������e���J"<�Z����-]��+)v���bռ���N�P���?�9�b#8�'�u��V��:�S��6���R��)��}�Y�ƫ�x&�bMq/182
error:856 unknown HTTP protocol/version: ��Ļ!��x�p�i�륯�-[(9�n���^��$$:$�����pz�p����O���8��J�{n�r��O�������e���J"<�Z����-]��+)v���bռ���N�P���?�9�b#8�'�u��V��:�S��6���R��)��}�Y�ƫ�x&�bMq
error:909 start line parse failed
debug:1312 state -- recv-head -> back-off 
debug:1312 state -- back-off  -> sleeps   
info:663 backing off for 1000 ms, retried 1
^C

See provided sample file built with gcc -o http_test -g http_test.c unix.c httpc.c -lssl. I had to rename it to .txt due to github.
http_test.txt

[denravonska]

Got it working in a test by clearing the position- and length variables of httpc_t before an operation.

diff --git a/httpc.c b/httpc.c
index bd0fcf9..c3b5506 100644
--- a/httpc.c
+++ b/httpc.c
@@ -1355,6 +1355,11 @@ static int httpc_op_heap(httpc_options_t *a, const char *url, int op, httpc_call
                h->rcv_param = rcv_param;
                h->snd_param = snd_param;
        }
+
+       h->position = 0;
+       h->length = 0;
+       h->max = 0;
+       h->length_set = 0;
        const int r = httpc_state_machine(h, url, op);
        if (r != HTTPC_YIELD && r != HTTPC_REUSE)
                a->state = NULL; /* make sure this is not reused */

Not sure if it's the correct approach though.

Edit: I think the following will need the same re-assignment treatment, or a HEAD followed by a GET will fail since h will be reused with empty callbacks:

		h->rcv       = rcv;
		h->snd       = snd;
		h->rcv_param = rcv_param;
		h->snd_param = snd_param;

[howerj]

I'll try to have a look at this once (if) I have time over the weekend, I should probably add some more unit tests to cover these new additions as well.

[denravonska]

I've found some more fields that would have to be moved.

diff --git a/httpc.c b/httpc.c
index c7e6b79..cb103c9 100644
--- a/httpc.c
+++ b/httpc.c
@@ -1357,12 +1357,18 @@ static int httpc_op_heap(httpc_options_t *a, const char *url, int op, httpc_call
                        return HTTPC_ERROR;
                memset(h, 0, sizeof *h);
                a->state     = h;
-               h->os        = a;
-               h->rcv       = rcv;
-               h->snd       = snd;
-               h->rcv_param = rcv_param;
-               h->snd_param = snd_param;
        }
+
+       h->os        = a;
+       h->rcv       = rcv;
+       h->snd       = snd;
+       h->rcv_param = rcv_param;
+       h->snd_param = snd_param;
+       h->position = 0;
+       h->length = 0;
+       h->max = 0;
+       h->length_set = 0;
+       h->state = SM_INIT;
        const int r = httpc_state_machine(h, url, op);
        if (r != HTTPC_YIELD && r != HTTPC_REUSE)
                a->state = NULL; /* make sure this is not reused */

Two ways to reproduce when using persistent connections:

• HEAD followed by GET -> since HEAD doesn't have callbacks (but has allocated h), no data callback will be used for the GETrequest
• Consecutive GETs -> length wasn't reset so httpc requested invalid data (IIRC)
• Consecutive GETs without state reset -> standard headers weren't added

Note that this is assuming httpc_op_heap is only going to be called once per request. I don't know if that's always the case (async + yield?).

[howerj]

Yeah, that isn't going to work when the HTTPC_YIELD option is set.