Buffer issues while attempting to use cr_journals()

[damianooldoni]

I would like to use function cr_journals() on a vector containing a lot of issn codes (~1500) to retrieve journal titles.
First, I tried to do it by calling cr_journals() on the entire vector:

journal_titles <- cr_journals(issn = issns)$data$title

It failed with the following error message:

Error in curl::curl_fetch_memory(x$url$url, handle = x$url$handle) :
schannel: next InitializeSecurityContext failed: SEC_E_MESSAGE_ALTERED (0x8009030F) - Het bericht of de handtekening die zijn opgegeven voor controle zijn gewijzigd.

translating from Dutch: "The message or the signature given for control are changed."

I tried to do it one by one:

journal_names <- NA
for (i in 1:length(issns)) {
  journal_name <- cr_journals(issn = issns[i])$data$title
  journal_names[i] <- ifelse(is.null(journal_name), NA, journal_name)
}

but I get at a certain point the same error message.

After few attempts I get this other error:

Error in curl::curl_fetch_memory(x$url$url, handle = x$url$handle) :
schannel: next InitializeSecurityContext failed: SEC_E_BUFFER_TOO_SMALL (0x80090321) - De buffers die zijn geleverd aan een functie zijn te klein.

translating from Dutch: "The buffers given to a function are too small."

Any hint/suggestion/idea is very appreciated. Thanks!

Session Info

Session info here.

[sckott]

thanks for the issue @damianooldoni

Are you using your email address to get in to the fast lane? https://github.com/ropensci/rcrossref/blob/master/R/rcrossref-package.R#L70-L88 https://github.com/CrossRef/rest-api-doc#etiquette

[sckott]

I tried with

library(rcrossref)
z <- cr_journals(query="american", limit=1000)
issns <- Filter(Negate(is.na), z$data$issn)
length(issns)
cr_journals(issn = issns)

didn't raise any problems. its possible using an email address if you're not will help, but the error seems more like a curl problem, and not with crossref rate limiting you.

does it work with a small number of issn's?

can you pass verbose = TRUE in the call to see if there's some useful info in the response headers

[damianooldoni]

Thanks @sckott,
I am now using e-mail address, sorry to not doing it since the begining. However, the problem is not solved.
By running your example, I get still the error:
SEC_E_MESSAGE_ALTERED (0x8009030F): The message or signature supplied for verification has been altered
or, alternatively:
SEC_E_BUFFER_TOO_SMALL (0x80090321): The buffers supplied to a function was too small.

Sometimes I can query 150 or even 200 issn at once, sometimes the error comes up at the very beginning, sometimes much later. I didn't find a logic behind. For example, last time I tried your example, with verbose = TRUE I get the buffer error at the issns[195] and the previous time it was at ìssn[33]`. I am investigating a little more on internet and on GItHub issues page of curl package. It can be a Windows problem (firewall, antivirus, ...) I will move my question to curl if I don't find a solution.
Thanks for your help!

[sckott]

@damianooldoni weird. i would try to do your requests just with curl on the command line or use any R http client so we can narrow down the issue

[sckott]

@damianooldoni any thoughts on this?

[damianooldoni]

No, @sckott, I 'm sorry. Actually the library didn't need this kind of information. So, I had to change my plans very soon. And no free time available for this. I am sorry.

[sckott]

thanks @damianooldoni

[sckott]

@jeroen have you seen these #155 (comment) types of errors ever (SEC_E_MESSAGE_ALTERED and SEC_E_BUFFER_TOO_SMALL)? i'm not sure what they mean

[jeroen]

No i've not seen this before. It looks like a bug in the libcurl ssl backend on Windows. @damianooldoni are you using the latest version of the curl package? Which version of Windows do you have?

[sckott]

any thoughts @damianooldoni on jeroen's comment above ?

[nicholasmfraser]

I am receiving exactly the same error messages as @damianooldoni (SEC_E_MESSAGE_ALTERED and SEC_E_BUFFER_TOO_SMALL) when using function cr_works() to loop through a vector of ~1000 DOIs. As with @damianooldoni it is inconsistent, it sometimes fails on the first few requests, sometimes after 100+ requests.

I am on Windows 7, using latest curl package for R (3.2), and am using my email address to get in the 'fast lane'. I've tried adding firewall exceptions, adding pauses between requests (to ensure its not related to server rate limiting), using the httr::set_config() function following this similar issue, upgraded and reinstalled curl/libcurl/openssl on windows, all to no avail. I have not been able to replicate the issue so far using curl on the command line. Any thoughts?

Edit: I added below the response from a successful call and an unsuccessful call when using verbose=TRUE.

Successful:

> GET /works/10.1101/001511 HTTP/1.1
Host: api.crossref.org
Accept-Encoding: gzip, deflate
Accept: application/json, text/xml, application/xml, */*
User-Agent: r-curl/3.2 crul/0.6.0 rOpenSci(rcrossref/0.8.4) (mailto:myemail)
X-USER-AGENT: r-curl/3.2 crul/0.6.0 rOpenSci(rcrossref/0.8.4) (mailto:myemail)

* schannel: client wants to read 16384 bytes
* schannel: encdata_buffer resized 17408
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: encrypted data got 4003
* schannel: encrypted data buffer: offset 4003 length 17408
* schannel: decrypted data length: 3943
* schannel: decrypted data added: 3943
* schannel: decrypted data cached: offset 3943 length 16384
* schannel: encrypted data length: 31
* schannel: encrypted data cached: offset 31 length 17408
* schannel: server closed the connection
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 3943
* schannel: decrypted data buffer: offset 0 length 16384
< HTTP/1.1 200 OK
< Content-Type: application/json;charset=UTF-8
< Vary: Accept
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: X-Requested-With
< Content-Length: 3637
< Server: http-kit
< Date: Tue, 28 Aug 2018 11:01:50 GMT
< X-Rate-Limit-Limit: 50
< X-Rate-Limit-Interval: 1s
< Connection: close
< 
* Closing connection 111
* schannel: shutting down SSL/TLS connection with api.crossref.org port 443
* schannel: clear security context handle
* Hostname api.crossref.org was found in DNS cache
*   Trying 208.254.38.72...
* TCP_NODELAY set
* Connected to api.crossref.org (208.254.38.72) port 443 (#112)
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: sending initial handshake data: sending 180 bytes...
* schannel: sent initial handshake data: sent 180 bytes
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: encrypted data got 2760
* schannel: encrypted data buffer: offset 2760 length 4096
* schannel: encrypted data length: 2670
* schannel: encrypted data buffer: offset 2670 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: encrypted data got 1426
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 365
* schannel: encrypted data buffer: offset 365 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: encrypted data got 176
* schannel: encrypted data buffer: offset 541 length 4096
* schannel: sending next handshake data: sending 190 bytes...
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: encrypted data got 51
* schannel: encrypted data buffer: offset 51 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 3/3)
* schannel: stored credential handle in session cache

Unsuccessful:

> GET /works/10.1101/001537 HTTP/1.1
Host: api.crossref.org
Accept-Encoding: gzip, deflate
Accept: application/json, text/xml, application/xml, */*
User-Agent: r-curl/3.2 crul/0.6.0 rOpenSci(rcrossref/0.8.4) (mailto:myemail)
X-USER-AGENT: r-curl/3.2 crul/0.6.0 rOpenSci(rcrossref/0.8.4) (mailto:myemail)

* schannel: client wants to read 16384 bytes
* schannel: encdata_buffer resized 17408
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: encrypted data got 2671
* schannel: encrypted data buffer: offset 2671 length 17408
* schannel: decrypted data length: 2611
* schannel: decrypted data added: 2611
* schannel: decrypted data cached: offset 2611 length 16384
* schannel: encrypted data length: 31
* schannel: encrypted data cached: offset 31 length 17408
* schannel: server closed the connection
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 2611
* schannel: decrypted data buffer: offset 0 length 16384
< HTTP/1.1 200 OK
< Content-Type: application/json;charset=UTF-8
< Vary: Accept
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: X-Requested-With
< Content-Length: 2305
< Server: http-kit
< Date: Tue, 28 Aug 2018 11:01:51 GMT
< X-Rate-Limit-Limit: 50
< X-Rate-Limit-Interval: 1s
< Connection: close
< 
* Closing connection 112
* schannel: shutting down SSL/TLS connection with api.crossref.org port 443
* schannel: clear security context handle
* Hostname api.crossref.org was found in DNS cache
*   Trying 208.254.38.72...
* TCP_NODELAY set
* Connected to api.crossref.org (208.254.38.72) port 443 (#113)
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: sending initial handshake data: sending 180 bytes...
* schannel: sent initial handshake data: sent 180 bytes
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: encrypted data got 2760
* schannel: encrypted data buffer: offset 2760 length 4096
* schannel: encrypted data length: 2670
* schannel: encrypted data buffer: offset 2670 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: encrypted data got 1426
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 365
* schannel: encrypted data buffer: offset 365 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: encrypted data got 176
* schannel: encrypted data buffer: offset 541 length 4096
* schannel: sending next handshake data: sending 190 bytes...
* Error in curl::curl_fetch_memory(x$url$url, handle = x$url$handle) : 
  schannel: next InitializeSecurityContext failed: SEC_E_MESSAGE_ALTERED (0x8009030F) - Die zu überprüfende Nachricht bzw. Signatur wurde geändert.
schannel: SSL/TLS connection with api.crossref.org port 443 (step 2/3)
* schannel: encrypted data got 7
* schannel: encrypted data buffer: offset 7 length 4096
* schannel: next InitializeSecurityContext failed: SEC_E_MESSAGE_ALTERED (0x8009030F) - Die zu überprüfende Nachricht bzw. Signatur wurde geändert.
* Closing connection 113
* schannel: shutting down SSL/TLS connection with api.crossref.org port 443
* schannel: clear security context handle

[jeroen]

This must be a bug in libcurl. Does this only happen for this specific host? @jay @bagder does this error sound familiar?

[bagder]

Never seen it before.

SEC_E_MESSAGE_ALTERED is wonderfully undocumented and hard to find described by Microsoft. I suppose a reason to try using another TLS backend to see if the problem persists then too.

The very name of this error does however not sound like its a libcurl error to me, its sounds like a problem with the server. But that's just a wild guess..

[nicholasmfraser]

I just looped through the same dataset several times using the GET method of the httr package to query the crossref API directly, and did not encounter the same issue. I haven't had this issue querying any other APIs before either. I'll try to recreate it on another machine later to see if this is specific to my current machine.

[jeroen]

It's definitely a Windows issue because it happens in the schannel back-end.

[sckott]

both crul and httr R packages use jeroen's curl pkg under the hood, so I am guessing it shouldn't matter which of crul and httr we use. at least I hope.

[jay]

I only know of one curl bug report of SEC_E_MESSAGE_ALTERED. I think it's interesting that both reporters have a failure with exactly 7 bytes received. Can you wireshark it?

Also steffengy/schannel-rs#8 may be related.

Edit: Receiving 7 "encrypted" bytes (raw bytes to be decrypted) is usually a sign of an unencrypted TLS alert sent before encryption has been established.

[nicholasmfraser]

I came back to this task recently and encountered the same issue again. I took a look at the issue of steffengy/schannel-rs#8 referenced above and removed TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 from the schannel SSL cipher suites order. The bug appears to have now disappeared. I do not know why this caused an issue, but just in case anyone else is experiencing the same thing.

[sckott]

thanks @nicholasmfraser for the follow up

removed TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 from the schannel SSL cipher suites order

this is on your machine?

[nicholasmfraser]

Yes, on my local machine (Windows 7). It must be something specific to my machine, I couldn't reproduce it on another machine that also had Windows 7.

[sckott]

Okay, thanks. Where did you find instructions on how to remove those?

[nicholasmfraser]

I used the information available here, following the same process to add cipher suites at the bottom of the page (I had to restart my computer before the changes took effect): https://docs.microsoft.com/en-us/windows/desktop/secauthn/tls-cipher-suites-in-windows-7

[sckott]

ah, thanks very much @nicholasmfraser