-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SSLV3_ALERT_HANDSHAKE FAILURE #351
Comments
Anna, The proxy in the DMZ was updated today to a newer version. I just confirmed that the change was made and completed, and that I have access to the services via the proxy. Can you please double check to confirm if you are continuing to have the issue now? If not, then those reported errors and your check via Windows may have happened when the proxy was temporarily down. That would point to a mistake on my part failing to communicate the short outage before hand. If the problem is still occurring, then it points to a more general problem impacting RFCs that doesn't impact OWP. Thanks, Hank |
I’ve left the office already but I will check on it tomorrow. Thanks! I
figured it was something like that but wanted to be sure.
…On Monday, October 28, 2024, HankHerr-NOAA ***@***.***> wrote:
Anna,
The proxy in the DMZ was updated today to a newer version. I just
confirmed that the change was made and completed, and that I have access to
the services via the proxy. Can you please double check to confirm if you
are continuing to have the issue now? If not, then those reported errors
and your check via Windows may have happened when the proxy was temporarily
down. That would point to a mistake on my part failing to communicate the
short outage before hand. If the problem is still occurring, then it points
to a more general problem impacting RFCs that doesn't impact OWP.
Thanks,
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RUUVAM7YFYGUYY76ZLZ52NEHAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBSGU3DANJUGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Hey Hank,
I am still getting the handshake failure.
On Mon, Oct 28, 2024 at 2:46 PM Anna Schneider - NOAA Federal <
***@***.***> wrote:
… I’ve left the office already but I will check on it tomorrow. Thanks! I
figured it was something like that but wanted to be sure.
On Monday, October 28, 2024, HankHerr-NOAA ***@***.***>
wrote:
> Anna,
>
> The proxy in the DMZ was updated today to a newer version. I just
> confirmed that the change was made and completed, and that I have access to
> the services via the proxy. Can you please double check to confirm if you
> are continuing to have the issue now? If not, then those reported errors
> and your check via Windows may have happened when the proxy was temporarily
> down. That would point to a mistake on my part failing to communicate the
> short outage before hand. If the problem is still occurring, then it points
> to a more general problem impacting RFCs that doesn't impact OWP.
>
> Thanks,
>
> Hank
>
> —
> Reply to this email directly, view it on GitHub
> <#351 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BLNZ2RUUVAM7YFYGUYY76ZLZ52NEHAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBSGU3DANJUGY>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Sorry for the delayed response, I was on lunch, Hank took off for the day. I was just able to visit the proxy, could you check again on your side |
No problem. Just ran it again and it is still giving me the error. I
checked with our local ITO and nothing changed on our end from before the
error started.
…On Tue, Oct 29, 2024 at 9:03 AM EvanPagryzinski-NOAA < ***@***.***> wrote:
Sorry for the delayed response, I was on lunch, Hank took off for the day.
I was just able to visit the proxy, could you check again on your side
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RXUGZSVXXYVDOYPDLTZ56WVJAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBUG4YTGMZRHA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
You are able to access things via the website correct? Can you attach the full error statement, Ill try out the script to see if I can reproduce |
Yeah I can still access the website. Everything was working until about
12:30 yesterday then it started giving me that error. I am sure it's not
the script as I only changed what folder the output was going to and one
other minor thing in the Evaluation txt file. I have attached the full
error message here.
…On Tue, Oct 29, 2024 at 9:10 AM EvanPagryzinski-NOAA < ***@***.***> wrote:
You are able to access things via the website correct? Can you attach the
full error statement, Ill try out the script to see if I can reproduce
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RRKMJTNAWA7KHVOFRLZ56XQVAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBUG4ZTEOJSGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
We are using the wres.nwc.nws.noaa.gov environment in this example.
Found the WRES CA file at /data/ldad/localapps/WRES/cacerts/cowres_dod_ca_chain.pem
Traceback (most recent call last):
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/urllib3/connection.py", line 400, in connect
self.sock = ssl_wrap_socket(
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/rh/rh-python38/root/usr/lib64/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/opt/rh/rh-python38/root/usr/lib64/python3.8/ssl.py", line 1073, in _create
self.do_handshake()
File "/opt/rh/rh-python38/root/usr/lib64/python3.8/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1131)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='wres.nwc.nws.noaa.gov', port=443): Max retries exceeded with url: /job (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1131)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "WRES_COWRES_Full.py", line 75, in <module>
post_result = requests.post( url="https://"+host+"/job",
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/requests/api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/requests/sessions.py", line 535, in request
resp = self.send(prep, **send_kwargs)
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/requests/sessions.py", line 648, in send
r = adapter.send(request, **kwargs)
File "/opt/rh/rh-python38/root/usr/lib/python3.8/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='wres.nwc.nws.noaa.gov', port=443): Max retries exceeded with url: /job (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1131)')))
|
Hmmmm, maybe when they were messing with the DMZ stuff they changed cert information. Let me try using the cert you all are using and see if that is the issue. It seems like an issue with the cert though based on what you are saying though |
ok thanks. If it is an issue with that is there a different one I should be
using?
…On Tue, Oct 29, 2024 at 9:39 AM EvanPagryzinski-NOAA < ***@***.***> wrote:
Hmmmm, maybe when they were messing with the DMZ stuff they changed cert
information. Let me try using the cert you all are using and see if that is
the issue. It seems like an issue with the cert though based on what you
are saying though
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RT6ISBOS5YKWBADY43Z5623BAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBUHAYDEOJUGI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
An interim solution you can try is to change this line in the python script to ignore SSL verification:
Ill look into the cert now and see if it works for me. Nothing should have changed there, but thats my best guess at the issue |
Ok thanks, I'm doing other stuff today so I don't need to be running it
right now. Let me know what you find out.
…On Tue, Oct 29, 2024 at 9:42 AM EvanPagryzinski-NOAA < ***@***.***> wrote:
An interim solution you can try is to change this line in the python
script to ignore SSL verification:
verify = wres_ca_file, -> verify = False,
Ill look into the cert now and see if it works for me. Nothing should have
changed there, but thats my best guess at the issue
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RRRX26GVIUMXYQZYADZ563IXAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBUHAYTCNRYGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Hm, I tried using the cert that you had posted about using in a previous ticket and I had no issue making a call to the proxy to post a job but it could be that im on the internal networks so I don't need to use the DMZ Let me know if my workaround works or not, otherwise this may be an IT issue on either your or our side. |
It did not work. I already checked with our IT person and the issue is not
on our side as nothing was changed when the error began.
…On Tue, Oct 29, 2024 at 10:34 AM EvanPagryzinski-NOAA < ***@***.***> wrote:
Hm, I tried using the cert that you had posted about using in a previous
ticket and I had no issue making a call to the proxy to post a job but it
could be that im on the internal networks so I don't need to use the DMZ
Let me know if my workaround works or not, otherwise this may be an IT
issue on either your or our side.
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RTFRQRDW3VNE7O5LRDZ57BIRAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBUHEZTAOBQGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Huh, I checked the database and it looks like people are still submitting evaluations so at least this doesn't appear to be an issue with our backend service. If you can access the GUI as well then there shouldn't be some special issue with using a script. If you need to use it before the issue is resolved you can run evaluations via the front end but I will work with Hank tomorrow to debug the proxy to see if we can see your requests coming through. Sorry for the issue, seems like the updates to the proxy that our IT did may have broken something |
The evaluations I'm doing are with HEFS so I have to load my own files
which I cannot do with the online GUI. I don't need to do this today so I
can wait till you guys look further into it tomorrow.
…On Tue, Oct 29, 2024 at 11:18 AM EvanPagryzinski-NOAA < ***@***.***> wrote:
Huh, I checked the database and it looks like people are still submitting
evaluations so at least this doesn't appear to be an issue with our backend
service. If you can access the GUI as well then there shouldn't be some
special issue with using a script. If you need to use it before the issue
is resolved you can run evaluations via the front end but I will work with
Hank tomorrow to debug the proxy to see if we can see your requests coming
through.
Sorry for the issue, seems like the updates to the proxy that our IT did
may have broken something
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RRB7TGFKIDU62WC7OTZ57GPHAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBVGAZDANBWG4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Anna, Taking a brief look at this after work hours... I think you may be making HTTP requests using an SSL/TLS version that is too old; specifically SSL v3. Details are below, but, to be clear, I may be barking up the wrong tree, so please let me know if I am. This is a bit outside of my area of expertise. My first clue was in the message:
While I've heard of "ssl", obviously, I wasn't familiar with the specific prefix, "SSLV3". So I looked it up and discovered that SSL v3 is an older protocol, with following quote from a Google search:
I also found this:
The reason I was not familiar with the "SSLV3" prefix is because we use TLSv1.2 (I think; Evan or James may be able to confirm more readily than I can). Noting that the latest changes to the proxy in the DMZ involve an upgrade to a newer version of HAProxy, I then wondered if they turned off use of SSLv3 in HAProxy much like the web search results are recommending. Sure enough, Google's AI summary says this:
We upgraded the HAProxy to 2.4; the previous version was 1.something, but I can't recall the something. So, this makes me think your code needs to be modified to use TLS v1.2 instead of SSLv3. Unfortunately, my knowledge of Python is limited, so I can't be exactly sure what code recommendation to make to ensure you are using a newer TLS version. Our script is provided and described in this wiki: https://github.com/NOAA-OWP/wres/wiki/WRES-Scripts-Usage-Guide Perhaps Evan can provide more details. If you have any reason to disagree with what I said above, please let me know. Again, this is just my best guess based on the message you provided and what I'm finding in a web search. If I am "barking up the wrong tree", I can revisit this tomorrow with Evan and we can figure it out from there. Thanks, Hank |
Interesting, I'm not sure why though this would suddenly turn into a
problem right after you all did an update. It was working fine all morning
and then suddenly gave me that error.
The code you linked to still uses requests.post which is what I am using.
There is no point in my program where I am specifically calling out SSLV3.
Link: data_post_response = requests.post( url=job_location + "/input/" +
side,
verify = wres_ca_file,
files = {"data":
file_content} )
Mine: post_result = requests.post( url="https://"+host+"/job",
verify = wres_ca_file,
data = { 'projectConfig': evaluation,
'postInput': 'true',
'keepInput': 'true' } )
…On Tue, Oct 29, 2024 at 2:29 PM HankHerr-NOAA ***@***.***> wrote:
Anna,
Taking a brief look at this after work hours... I think you may be making
HTTP requests using an SSL/TLS version that is too old; specifically SSL
v3. Details are below, but, to be clear, I may be barking up the wrong
tree, so please let me know if I am. This is a bit outside of my area of
expertise.
My first clue was in the message:
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake
failure (_ssl.c:1131)
While I've heard of "ssl", obviously, I wasn't familiar with the specific
prefix, "SSLV3". So I looked it up and discovered that SSL v3 is an older
protocol, with following quote from a Google search:
SSLv3 is no longer considered secure and should be turned off in favor of
more modern protocols. SSLv3 has several vulnerabilities, including:...
I also found this:
SSLv3 was renamed to TLS after SSL was upgraded to TLSv1.0. The latest
standard version of SSL is TLSv1.2.
The reason I was not familiar with the "SSLV3" prefix is because we use
TLSv1.2 (I think; Evan or James may be able to confirm more readily than I
can). Noting that the latest changes to the proxy in the DMZ involve an
upgrade to a newer version of HAProxy, I then wondered if they turned off
use of SSLv3 in HAProxy much like the web search results are recommending.
Sure enough, Google's AI summary says this:
HAProxy 1.5 and later versions disallow SSLv3 (Secure Socket Layer):
We upgraded the HAProxy to 2.4; the previous version was 1.something, but
I can't recall the something. So, this makes me think your code needs to be
modified to use TLS v1.2 instead of SSLv3.
Unfortunately, my knowledge of Python is limited, so I can't be exactly
sure what code recommendation to make to ensure you are using a newer TLS
version. Our script is provided and described in this wiki:
https://github.com/NOAA-OWP/wres/wiki/WRES-Scripts-Usage-Guide
Perhaps Evan can provide more details.
If you have any reason to disagree with what I said above, please let me
know. Again, this is just my best guess based on the message you provided
and what I'm finding in a web search. If I am "barking up the wrong tree",
I can revisit this tomorrow with Evan and we can figure it out from there.
Thanks,
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RQAZCPBQLORDGNQ32LZ5742LAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBVGM3DIMZTGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Based on the timing you described, it appears that the error didn't start happening until after the upgrade to HAProxy 2.4. So it makes sense that 2.4 exposes the error, while before that upgrade, HAProxy didn't complain. I was worried your Python snippets would show nothing obvious. Again, Evan may be able to help tomorrow. Also, if I'm right about the HAProxy rejecting your attempt to connect because of the SSL version, then I should see something in the HAProxy traffic log stating as much in a message (I hope; log messaging can be finicky). Can you please let me know the time of your last attempt to connect? The HAProxy traffic log is likely to be very big and difficult to parse through, so the timing may help. I'm also hoping I can trigger off of your More tomorrow. Have a great evening! Hank |
One more quick test you can perform from the same machine where you are running Python...
If it works, then, at the very end, you see just "Up". The verbose output should also include a 200 response. Let me know what happens when you run the above command. Again, more tomorrow, Hank |
Yeah just tried again. The last attempt would be at 21:51z or 02:51 pm
pacific time. I did try and do an upgrade to the requests library and it
wouldn't allow it. the pip version on our awips is 19.3.1 if that is of any
help.
…On Tue, Oct 29, 2024 at 2:49 PM HankHerr-NOAA ***@***.***> wrote:
Interesting, I'm not sure why though this would suddenly turn into a
problem right after you all did an update. It was working fine all morning
and then suddenly gave me that error.
Based on the timing you described, it appears that the error didn't start
happening until after the upgrade to HAProxy 2.4. So it makes sense that
2.4 exposes the error, while before that upgrade, HAProxy didn't complain.
I was worried your Python snippets would show nothing obvious. Again, Evan
may be able to help tomorrow. Also, if I'm right about the HAProxy
rejecting your attempt to connect because of the SSL version, then I should
see something in the HAProxy traffic log stating as much in a message (I
hope; log messaging can be finicky).
*Can you please let me know the time of your last attempt to connect?*
The HAProxy traffic log is likely to be very big and difficult to parse
through, so the timing may help. I'm also hoping I can trigger off of your
postInput and keepInput settings, both of which are somewhat rare,
particularly keepInput.
More tomorrow. Have a great evening!
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RRR5SRDPF4SL2MR53DZ577IFAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBVGM4TIMRRGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Oh... If this is tied to an AWIPS version, and that AWIPS version is too old, then, yeah, that's a problem. I think your Python venv will include a version for Have a great evening, Hank |
Yeah no worries this can 100% wait until tomorrow. I tried the curl command
and it said "failed to connect connection timed out".
…On Tue, Oct 29, 2024 at 2:55 PM HankHerr-NOAA ***@***.***> wrote:
Oh... If this is tied to an AWIPS version, and that AWIPS version is too
old, then, yeah, that's a problem.
I think your Python venv will include a version for requests and urllib3.
Can you get those versions? My apologies for not being sure how best to do
that off the top of my head. I'm already being called down by my family, so
I really need to get off of the computer. Maybe I'll check back in later
this evening.
Have a great evening,
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RXUIVSZXKXQQJK77HDZ5775TAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBVGQYDCNJSGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Because this is bugging me so much, I decided to take a look at the server logs. First, I think the "SSLV3" stuff is just a distraction. I think that is some sort of general message Python returns when the handshake fails that doesn't actually indicate you are using SSL v3. Something else is going wrong, as indicated by the failure of your Server side, the HAProxy is logging the failed handshakes as exactly that with no details. Here are the messages around the time you provided:
Using the If you can, Anna, please share the complete, verbose logging you get from your At this point, there is nothing more I can investigate without more verbose output from Hank ==========
|
Hopefully it isn't that because I don't think we have the ability to do any
kind of python library upgrading due to NWS security things.
The log output by the curl command was fairly minimal but here it is.
Basically just saying it timed out.
…On Tue, Oct 29, 2024 at 4:14 PM HankHerr-NOAA ***@***.***> wrote:
Because this is bugging me so much, I decided to take a look at the server
logs.
First, I think the "SSLV3" stuff is just a distraction. I think that is
some sort of general message Python returns when the handshake fails that
doesn't actually indicate you are using SSL v3. Something else is going
wrong, as indicated by the failure of your curl command to connection.
Server side, the HAProxy is logging the failed handshakes as exactly that
with no details. Here are the messages around the time you provided:
<134>Oct 29 21:46:01 haproxy[8]: [IP omitted]:47068 [29/Oct/2024:21:46:01.207] https-in/1: SSL handshake failure
<134>Oct 29 21:46:34 haproxy[8]: [IP omitted]:47112 [29/Oct/2024:21:46:34.445] https-in/1: SSL handshake failure
<134>Oct 29 21:50:55 haproxy[8]: [IP omitted]:48044 [29/Oct/2024:21:50:55.830] https-in/1: SSL handshake failure
Using the curl command I shared, I saw the verbose logging below.
Connection was successful (as expected).
If you can, Anna, please share the complete, verbose logging you get from
your curl command (you may want to remove any references to IP addresses;
if you prefer you can email me the output).
At this point, there is nothing more I can investigate without more
verbose output from curl or Python explaining why the handshake failed.
Thanks,
Hank
==========
***@***.*** issue103897]$ curl -v --cacert cowres_dod_ca_chain.pem https://wres.nwc.nws.noaa.gov/job
* Trying [IP address omitted]...
* TCP_NODELAY set
* Connected to wres.nwc.nws.noaa.gov ([IP address omitted]) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: cowres_dod_ca_chain.pem
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; O=U.S. Government; OU=DoD; OU=PKI; OU=DISA; CN=wres.nwc.nws.noaa.gov
* start date: Mar 12 18:39:17 2024 GMT
* expire date: Mar 12 18:39:17 2027 GMT
* subjectAltName: host "wres.nwc.nws.noaa.gov" matched cert's "wres.nwc.nws.noaa.gov"
* issuer: C=US; O=U.S. Government; OU=DoD; OU=PKI; CN=DOD SW CA-66
* SSL certificate verify ok.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /job HTTP/1.1
> Host: wres.nwc.nws.noaa.gov
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 200 OK
< date: Tue, 29 Oct 2024 23:08:26 GMT
< content-type: text/plain;charset=utf-8
< vary: Accept-Encoding
< content-length: 2
< server: Jetty(11.0.24)
< x-frame-options: DENY
< x-xss-pz`rotection: 1;mode=block
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000; includeSubDomains; preload;
< cache-control: no-cache, max-age=0
<
* Connection #0 to host wres.nwc.nws.noaa.gov left intact
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RRAA4NCKFH4G3NHEETZ6AJGHAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBVGQ4TENZTGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
ls2-rsa.wr.awips.noaa.gov{ldad}102: curl -v --cacert cowres_dod_ca_chain.pem http://wres.nwc.nws.noaa.gov/job
* About to connect() to wres.nwc.nws.noaa.gov port 80 (#0)
* Trying [IP omitted]
* Connection timed out
* Failed connect to wres.nwc.nws.noaa.gov:80; Connection timed out
* Closing connection 0
curl: (7) Failed connect to wres.nwc.nws.noaa.gov:80; Connection timed out
|
Anna, Can you double-check that you are using
Thanks, James |
As an aside, bumping the major version of the proxy server software should entail a fairly rigorous staging/testing protocol because the major version bump implies breaking changes. Most likely, the major version bump has brought about some changes to legacy tls settings that are incompatible with an old version of python and/or the requests package. Educated guess. It may help to turn on debug logging within python requests. Most likely, the curl failure is a distraction here and that will succeed when |
Ah dang you’re right. Let me do that again
…On Tuesday, October 29, 2024, James Brown ***@***.***> wrote:
Anna,
Can you double-check that you are using https in your curl? The following
shows http, not https:
ls2-rsa.wr.awips.noaa.gov{ldad}102: curl -v --cacert
cowres_dod_ca_chain.pem http://wres.nwc.nws.noaa.gov/job
Thanks,
James
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RR2JWJDWXMF6U25D33Z6ALNTAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBVGUYTENJQHE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Here’s the new log file.
ls2-rsa.wr.awips.noaa.gov{ldad}102: curl -v --cacert
cowres_dod_ca_chain.pem https://wres.nwc.nws.noaa.gov/job
* About to connect() to wres.nwc.nws.noaa.gov port 443 (#0)
* Trying 140.172.117.41...
* Connected to wres.nwc.nws.noaa.gov (140.172.117.41) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: cowres_dod_ca_chain.pem
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=wres.nwc.nws.noaa.gov,OU=DISA,OU=PKI,OU=DoD,O=U.S.
Government,C=US
* start date: Mar 12 18:39:17 2024 GMT
* expire date: Mar 12 18:39:17 2027 GMT
* common name: wres.nwc.nws.noaa.gov
* issuer: CN=DOD SW CA-66,OU=PKI,OU=DoD,O=U.S. Government,C=US
GET /job HTTP/1.1
User-Agent: curl/7.29.0
Host: wres.nwc.nws.noaa.gov
Accept: */*
< HTTP/1.1 200 OK
< date: Wed, 30 Oct 2024 00:10:52 GMT
< content-type: text/plain;charset=utf-8
< vary: Accept-Encoding
< content-length: 2
< server: Jetty(11.0.24)
< x-frame-options: DENY
< x-xss-pz`rotection: 1;mode=block
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000; includeSubDomains; preload;
< cache-control: no-cache, max-age=0
<
* Connection #0 to host wres.nwc.nws.noaa.gov left intact
On Tuesday, October 29, 2024, Anna Schneider - NOAA Federal <
***@***.***> wrote:
… Ah dang you’re right. Let me do that again
On Tuesday, October 29, 2024, James Brown ***@***.***>
wrote:
> Anna,
>
> Can you double-check that you are using https in your curl? The
> following shows http, not https:
>
> ls2-rsa.wr.awips.noaa.gov{ldad}102: curl -v --cacert
> cowres_dod_ca_chain.pem http://wres.nwc.nws.noaa.gov/job
>
> Thanks,
>
> James
>
> —
> Reply to this email directly, view it on GitHub
> <#351 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BLNZ2RR2JWJDWXMF6U25D33Z6ALNTAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBVGUYTENJQHE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Correct. Thanks Hank!
…On Wed, Oct 30, 2024 at 9:39 AM HankHerr-NOAA ***@***.***> wrote:
James: Thanks. I had forgotten about that.
Anna: Got it. So, yeah, any communication with the COWRES is failing when
done through Python.
ITSG is going to be backing up the HAProxy version in production in a bit;
I'll let you know when its done. At that time, I'm hoping your scripts will
start working again. I'm going to try to reproduce your problem using our
staging environment, which uses the HAProxy version 2.4 that was
problematic.
Thanks,
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RRAZXXIPMKA2V4GBD3Z6EDSFAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBXG42TMMRSGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Ann: The older version of the HAProxy is back up and running. Please let me know if you are able to connect. If not, then it means something else has gone awry, Hank |
Thanks! We're in the middle of some awips reboots so I will give it a go as
soon as I have the ok to log back in.
…On Wed, Oct 30, 2024 at 9:54 AM HankHerr-NOAA ***@***.***> wrote:
Ann:
The older version of the HAProxy is back up and running. Please let me
know if you are able to connect. If not, then it means something else has
gone awry,
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RVJ5FZOSLPY5NMDNFDZ6EFM5AVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBXHAYDCMRSG4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Just tried again. I am now connecting via python.
On Wed, Oct 30, 2024 at 9:57 AM Anna Schneider - NOAA Federal <
***@***.***> wrote:
… Thanks! We're in the middle of some awips reboots so I will give it a go
as soon as I have the ok to log back in.
On Wed, Oct 30, 2024 at 9:54 AM HankHerr-NOAA ***@***.***>
wrote:
> Ann:
>
> The older version of the HAProxy is back up and running. Please let me
> know if you are able to connect. If not, then it means something else has
> gone awry,
>
> Hank
>
> —
> Reply to this email directly, view it on GitHub
> <#351 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BLNZ2RVJ5FZOSLPY5NMDNFDZ6EFM5AVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBXHAYDCMRSG4>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Thanks. I'm glad your scripts are working again. That confirms it was something with the HAProxy upgrade combined with your AWIPS environment that is causing the issue. Hank |
This web search would appear to agree largely with what James was saying about incompatible ciphers (based on the AI summary): Protocol mismatch should not be an issue so long as the HAProxy 1.8 (now running) respects this line in its configuration:
That line disallows anything before TLS v1.2. Since Anna's scripts are now working, they must be using v1.2 or later. The certificate should also be valid. I was able to use the file However, we still need to confirm ciphers is the cause by reproducing the issue locally. Since this is a user support ticket, I'll create a development ticket tomorrow and start trying to reproduce the issue. I'll obtain an RHEL 7 image (trying to match versions with whatever AWIPS uses), install the correct version of Python on it, hop into the container, and try to reproduce the problem. Next time I comment in this ticket, it should be to confirm that we reproduced the problem locally. Thanks, Hank |
We have a ticket open with NCF to update the python requests library. I'll
let you know if they are able to do it.
…On Wed, Oct 30, 2024 at 11:28 AM HankHerr-NOAA ***@***.***> wrote:
This web search would appear to agree largely with what James was saying
about incompatible ciphers (based on the AI summary):
https://www.google.com/search?q=SSLV3_ALERT_HANDSHAKE_FAILURE+Python+HAProxy&rlz=1C1GCEA_enUS1057US1057&oq=SSLV3_ALERT_HANDSHAKE_FAILURE+Python+HAProxy&gs_lcrp=EgZjaHJvbWUyCQgAEEUYORigATIHCAEQIRigATIHCAIQIRigAdIBCTIwNDQwajBqNKgCALACAA&sourceid=chrome&ie=UTF-8
Protocol mismatch should not be an issue so long as the HAProxy 1.8 (now
running) respects this line in its configuration:
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
That line disallows anything before TLS v1.2. Since Anna's scripts are now
working, they must be using v1.2 or later.
The certificate should also be valid. I was able to use the file
cowres_dod_ca_chain.pem to authenticate the COWRES, as was Anna (now that
we've reverted to 1.8).
However, we still need to confirm ciphers is the cause by reproducing the
issue locally. Since this is a user support ticket, I'll create a
development ticket tomorrow and start trying to reproduce the issue. I'll
obtain an RHEL 7 image (trying to match versions with whatever AWIPS uses),
install the correct version of Python on it, hop into the container, and
try to reproduce the problem.
Next time I comment in this ticket, it should be to confirm that we
reproduced the problem locally.
Thanks,
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RWW2UICAIY4JYYWIFDZ6EQM5AVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGAZDSNZXHE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Anna, I'm not certain just updating requests will be enough. In fact, it may have something do with the AWIPS machine, itself. I was unable to reproduce your error when deploying Python 3.8.18 and requests 2.22.0 in a Docker container, so I'm not entirely sure the ciphers are coming from Python/requests or coming from the AWIPS machine. My investigation has barely begun, so I could be wrong, and there certainly is value in updating requests as well as Python, if they allow it, regardless of this ticket. So I'm not saying don't ask for it; I'm just not sure that will fix this issue. Thanks, |
Gotcha, that's good to know. We are also still on red hat 7 which may be
another issue I'm told.
…On Wed, Oct 30, 2024 at 1:06 PM HankHerr-NOAA ***@***.***> wrote:
Anna,
I'm not certain just updating requests will be enough. In fact, it may
have something do with the AWIPS machine, itself. I was unable to reproduce
your error when deploying Python 3.8.18 and requests 2.22.0 in a Docker
container, so I'm not entirely sure the ciphers are coming from
Python/requests or coming from the AWIPS machine.
My investigation has barely begun, so I could be wrong, and there
certainly is value in updating requests as well as Python, if they allow
it, regardless of this ticket. So I'm not saying don't ask for it; I'm just
not sure that will fix this issue.
Thanks,
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RR5QE2S7U5KH2J2UL3Z6E3Z3AVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGI3DEMJRGQ>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
The problem reported here is very much OWP/NWC specific, so I'm tracking reproducing and fixing it in a VLab ticket #136136. Anna: Since I think this may be a cipher issue, when you can, please identify for me the cipher suite being used by your Python My hope is to somehow force my tests to use your cipher suite to determine if that is the cause of the handshake failure. Thanks, Hank |
The negotiation of the cipher should be part of the debug logging information for the handshake on both sides and I would expect a message somewhere about the failure to negotiate a common cipher between the client and server if that is indeed the issue. In general, it can be hard to debug tls because limited information is provided for obvious reasons, but the debug logging of the client and server should reveal a lot. I am not sure how this is turned on in python requests, however. |
True. Anna could turn on debugging on her end and run the same evaluation to see what ciphers are reported in the logging. Whether that is easier than running the command Google identified in Python is up to her. I'll see what I can do on our end to identify the HAProxy cipher suite in staging. Hopefully, it matches what would be in the DMZ using HAProxy 2.4, but I can't be absolutely certain. Reproducing the issue using Anna's cipher would give me more confidence, however. Hank |
Anna: I have a series of Python lines-of-code for you to run that should allow you to identify the cipher. Please run them when you have time. I'm hoping you'll see a different cipher than I'm seeing. So, I tried a few different ways to get at the cipher being used in Python. It wasn't as easy as just turning on debug, but that may be because of my own ignorance of Python. Eventually, I settled on this approach found via web search that wraps around the request to dump the cipher:
You can see that the cipher being used is Thanks, Hank |
Hi Hank,
I typed in the commands as instructed. The response I got is:
('ECDHE-RSA-AES256-GCM-SHA384', 'TLSv1/SSLv3', 256)
…On Thu, Oct 31, 2024 at 5:39 AM HankHerr-NOAA ***@***.***> wrote:
*Anna: I have a series of Python lines-of-code for you to run that should
allow you to identify the cipher. Please run them when you have time.*
I'm hoping you'll see a different cipher than I'm seeing.
So, I tried a few different ways to get at the cipher being used in
Python. It wasn't as easy as just turning on debug, but that may be because
of my own ignorance of Python. Eventually, I settled on this approach found
via web search that wraps around the request to dump the cipher:
$ python
Python 3.8.18 (default, Mar 12 2024, 12:46:23)
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from http.client import HTTPConnection
>>> from http.client import HTTPConnection
>>>
>>>
>>> def request(self, method, url, body=None, headers={}, *,
... encode_chunked=False):
... self._send_request(method, url, body, headers, encode_chunked)
... print(self.sock.cipher())
...
>>> HTTPConnection.request = request
>>> response = requests.get("https://wres.nwc.nws.noaa.gov/job",verify='cowres_dod_ca_chain.pem')
('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)
>>>
You can see that the cipher being used is ('TLS_AES_256_GCM_SHA384',
'TLSv1.3', 256) and that seems to work for accessing either the old or
the new proxy. I did confirm by running `print(response)' that I get a 200
response, so it connects. I also get a 200 visiting the staging proxy,
which is still using the new HAProxy 2.4.
Thanks,
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RWAINGE6QWBRUNMOB3Z6IQHTAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBZG42DQOBZGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Not sure if that is reporting accurately, but TLSv1.0 is going to be a problem as that is ancient/insecure. I wonder if there is a setting in Python to enforce/request TLS1.2 or above. We only support TLS1.2 and TLS1.3, I assume for the proxy too, but certainly the backend. The cipher looks fine, but I'm not even sure that meshes with TLSv1.0. |
Tried it a second time and it still says TLSv1/SSLv3.
…On Thu, Oct 31, 2024 at 6:02 AM James Brown ***@***.***> wrote:
Not sure if that is reporting accurately, but TLSv1.0 is going to be a
problem as that is ancient/insecure. I wonder if there is a setting in
Python to enforce/request TLS1.2 or above. We only support TLS1.2 and
TLS1.3, I assume for the proxy too, but certainly the backend. The cipher
looks fine, but I'm not even sure that meshes with TLSv1.0.
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RRM6CCWAUJJALJUOCTZ6IS7DAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBZG44TCNZQGM>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Thanks, Anna! I think we are definitely closer to identifying the problem now. TLSv1/SSLv3 should be a problem. Given that we are disallowing access to the proxy using TLS older the v1.2 in the configuration,
I'm surprised that you are even able to connect to the old proxy. I think Brock from ITSG wondered if the 1.8 HAProxy is respecting that configuration line; now I'm wondering the same. I'll look into that. I'm hoping its possible set the cipher and TLS version in Python, as James says. I'll start by trying to use your cipher and TLS to see what happens. Hank |
I googled and it seems like you are able to in theory. But, I wonder if
this is more applying to going back to an old TLS version rather than
trying to force it to upgrade. I imagine if I try and force it to upgrade
like this it will yell at me for not having permission to do any kind of
upgrading. I also imagine that all the other RFCs are on the same version
given that we are using the same older requests library.
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
import ssl
class TLSAdapter(HTTPAdapter):
def __init__(self, ssl_version=None, **kwargs):
self.ssl_version = ssl_version
super().__init__(**kwargs)
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(
num_pools=connections,
maxsize=maxsize,
block=block,
ssl_version=self.ssl_version
)
# Create a session and mount the adapter
session = requests.Session()
session.mount('https://', TLSAdapter(ssl_version=ssl.PROTOCOL_TLSv1_2))
# Make a request
response = session.get('https://www.example.com')
…On Thu, Oct 31, 2024 at 6:11 AM HankHerr-NOAA ***@***.***> wrote:
Thanks, Anna! I think we are definitely closer to identifying the problem
now.
TLSv1/SSLv3 should be a problem. Given that we are disallowing access to
the proxy using TLS older the v1.2 in the configuration,
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
I'm surprised that you are even able to connect to the old proxy. I think
Brock from ITSG wondered if the 1.8 HAProxy is respecting that
configuration line; now I'm wondering the same. I'll look into that.
I'm hoping its possible set the cipher and TLS version in Python, as James
says. I'll start by trying to use your cipher and TLS to see what happens.
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RSQVXYOGPW5UTJQF4TZ6IT6VAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBZHAYDSMJTGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Our ITO talked to NCF and they said the TLS version we have is the most
currently available for this version of awips, so there is no chance of me
being able to do the requests update if that is what is needed to solve
this.
On Thu, Oct 31, 2024 at 6:13 AM Anna Schneider - NOAA Federal <
***@***.***> wrote:
… I googled and it seems like you are able to in theory. But, I wonder if
this is more applying to going back to an old TLS version rather than
trying to force it to upgrade. I imagine if I try and force it to upgrade
like this it will yell at me for not having permission to do any kind of
upgrading. I also imagine that all the other RFCs are on the same version
given that we are using the same older requests library.
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
import ssl
class TLSAdapter(HTTPAdapter):
def __init__(self, ssl_version=None, **kwargs):
self.ssl_version = ssl_version
super().__init__(**kwargs)
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(
num_pools=connections,
maxsize=maxsize,
block=block,
ssl_version=self.ssl_version
)
# Create a session and mount the adapter
session = requests.Session()
session.mount('https://', TLSAdapter(ssl_version=ssl.PROTOCOL_TLSv1_2))
# Make a request
response = session.get('https://www.example.com')
On Thu, Oct 31, 2024 at 6:11 AM HankHerr-NOAA ***@***.***>
wrote:
> Thanks, Anna! I think we are definitely closer to identifying the problem
> now.
>
> TLSv1/SSLv3 should be a problem. Given that we are disallowing access to
> the proxy using TLS older the v1.2 in the configuration,
>
> ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
>
> I'm surprised that you are even able to connect to the old proxy. I think
> Brock from ITSG wondered if the 1.8 HAProxy is respecting that
> configuration line; now I'm wondering the same. I'll look into that.
>
> I'm hoping its possible set the cipher and TLS version in Python, as
> James says. I'll start by trying to use your cipher and TLS to see what
> happens.
>
> Hank
>
> —
> Reply to this email directly, view it on GitHub
> <#351 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BLNZ2RSQVXYOGPW5UTJQF4TZ6IT6VAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBZHAYDSMJTGU>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Anna, I don't think this is an issue with your TLS version. I was able to prove that the proxy does reject requests that are using an old TLS. So, if you were using SSLv3/TLSv1, then you would have been encountering issues all along. Hence, I think this is cipher specific. I've been trying to reproduce your problem using your cipher, but have not had luck so far. You can visit VLab ticket #136136 if you want to follow my progress. Hank |
Ok gotcha. I tried to force it to use TLSv1.2 and it didn't yell at me, but
I can't confirm it worked either. When I asked the request to print out the
cipher when connecting the response it gave me was "None". I got that when
doing the request without the adapter the normal way as well. If that's of
any use.
…On Thu, Oct 31, 2024 at 11:52 AM HankHerr-NOAA ***@***.***> wrote:
Anna,
I don't think this is an issue with your TLS version. I was able to prove
that the proxy does reject requests that are using an old TLS. So, if you
were using SSLv3/TLSv1, then you would have been encountering issues all
along.
Hence, I think this is cipher specific. I've been trying to reproduce your
problem using your cipher, but have not had luck so far. You can visit VLab
ticket #136136 if you want to follow my progress.
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RR4AOYSWMWG4R5YLP3Z6J37FAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJQGYYDGOJSGI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Okay. I'm not really sure what to make of the None response when using the adapter as you did before. One thing that might be worth testing is forcing it to use "TLSv1" instead of 1.2. If I'm right, based on my tests, you should fail to access the services. Anyway, I'm going to spend a bit more time on this tomorrow, but it might point at us having to be patient and wait for AWIPS to upgrade to RHEL 8 (in hopes that the Python cipher is updated) before updating HAProxy to 2.4. More tomorrow, Hank |
It spit back none even when I was not using the adaptor which I found odd.
I did try again with v1 instead of 1.2 and it did fail with a
TLSV1_ALERT_PROTOCOL_VERSION error. So I think in spite of what was spit
out by the code you had me test before you're right and it is using 1.2.
…On Thu, Oct 31, 2024 at 1:19 PM HankHerr-NOAA ***@***.***> wrote:
Okay. I'm not really sure what to make of the None response when using the
adapter as you did before. One thing that might be worth testing is forcing
it to use "TLSv1" instead of 1.2. If I'm right, based on my tests, you
should fail to access the services.
Anyway, I'm going to spend a bit more time on this tomorrow, but it might
point at us having to be patient and wait for AWIPS to upgrade to RHEL 8
(in hopes that the Python cipher is updated) before updating HAProxy to
2.4. More tomorrow,
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RRMDAEJHUK2JTDIA23Z6KGFPAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJQG42DENJSGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Also no rush, I'm off tomorrow so I'll be back Monday.
On Thu, Oct 31, 2024 at 1:29 PM Anna Schneider - NOAA Federal <
***@***.***> wrote:
… It spit back none even when I was not using the adaptor which I found odd.
I did try again with v1 instead of 1.2 and it did fail with a
TLSV1_ALERT_PROTOCOL_VERSION error. So I think in spite of what was spit
out by the code you had me test before you're right and it is using 1.2.
On Thu, Oct 31, 2024 at 1:19 PM HankHerr-NOAA ***@***.***>
wrote:
> Okay. I'm not really sure what to make of the None response when using
> the adapter as you did before. One thing that might be worth testing is
> forcing it to use "TLSv1" instead of 1.2. If I'm right, based on my tests,
> you should fail to access the services.
>
> Anyway, I'm going to spend a bit more time on this tomorrow, but it might
> point at us having to be patient and wait for AWIPS to upgrade to RHEL 8
> (in hopes that the Python cipher is updated) before updating HAProxy to
> 2.4. More tomorrow,
>
> Hank
>
> —
> Reply to this email directly, view it on GitHub
> <#351 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BLNZ2RRMDAEJHUK2JTDIA23Z6KGFPAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJQG42DENJSGA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Thanks, Anna. So you are using a correct version of TLS (1.2 or higher), but the cipher used is one associated with TLSv1/SSLv3 that HAProxy 2.4 will not allow, but HAProxy 1.8 does allow. At least, that's my theory. I just wish I could reproduce the error so that I could explore fixes and mitigations with 2.4. Have a good weekend, Hank |
I'm not so sure, tbh. The ECDHE-RSA key exchange algorithms (so, not actually a cipher in itself, but the key exchange part of a cipher suite) are supported in TLSv1.2 and are FIPS compliant, based on my searching (ECDHE = Diffie Hellman Exchange with Eliptic Curve). I think the ECDHE-RSA algorithms are actually supported in all versions of TLS. So, actually, I doubt that is the issue and it may be something more subtle. Either way, it will probably disappear when upgrading Python/Requests/RHEL7, but that may be some time away. This may be a situation where you make a note of the problem and don't sink a ton more time into it until absolutely necessary as these issues can be very hard to reproduce and debug. |
Thanks, James, for digging deeper. That would explain why its so hard for me to reproduce the error. There must be some other check involved with the handshake that is failing under the specific conditions in which the call is made from the CNRFC AWIPS machine. We've confirmed the TLS version is okay and the cipher appears usable (both by your research and what I did using Docker running Python). Some other part of that handshake is failing, and I'm not sure what. And it appears HAProxy can't communicate it from its end without turning on an option that wasn't available until sometime after 2.4. I saw folks using wire shark or other tools to identify issues, but that's way beyond me and would require we put 2.4 back in production so that Anna can run against it. I agree that spending more time investigating this from my end is probably not worth it. It would be too hard to mirror the CNRFC AWIPS machine precisely enough in Docker. I'm going to check with the CHPS team to see when the RFCs are slated to move to RHEL 8 so that we can try HAProxy 2.4 at that time. Until then, we won't have proper traffic logging in production. We'll have to cross our fingers that we are able to work around it for a while longer. Hank |
Thanks for looking into it. I was told we would be upgrading to RedHat 8
some time next year.
…On Fri, Nov 1, 2024 at 5:58 AM HankHerr-NOAA ***@***.***> wrote:
Thanks, James, for digging deeper. That would explain why its so hard for
me to reproduce the error. There must be some other check involved with the
handshake that is failing under the specific conditions in which the call
is made from the CNRFC AWIPS machine. We've confirmed the TLS version is
okay and the cipher appears usable (both by your research and what I did
using Docker running Python). Some other part of that handshake is failing,
and I'm not sure what. And it appears HAProxy can't communicate it from its
end without turning on an option that wasn't available until sometime after
2.4. I saw folks using wire shark or other tools to identify issues, but
that's way beyond me and would require we put 2.4 back in production so
that Anna can run against it.
I agree that spending more time investigating this from my end is probably
not worth it. It would be too hard to mirror the CNRFC AWIPS machine
precisely enough in Docker.
I'm going to check with the CHPS team to see when the RFCs are slated to
move to RHEL 8 so that we can try HAProxy 2.4 at that time. Until then, we
won't have proper traffic logging in production. We'll have to cross our
fingers that we are able to work around it for a while longer.
Hank
—
Reply to this email directly, view it on GitHub
<#351 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BLNZ2RVUGVHNDSNQVQPOTU3Z6N3IXAVCNFSM6AAAAABQYHTUXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJRHAZTCOJZGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
*-------------------------------------------------------------------*
*Anna Schneider* | Meteorologist
NOAA/National Weather Service
*California Nevada River Forecast Center*
(p) 916.979.3056
--------------------------------------------------------------------
|
Moving this ticket to the backlog (i.e., removing the milestone). The HAProxy 2.4 was backed up to 1.8 to support access from the CNRFC AWIPS machine. Until the RFCs move to RHEL 8, and hopefully get a newer version of Python with newer ciphers, we will not be able to return to 2.4. That is, unless IT security folks force us to stop accepting the old ciphers. Shaif said that the RHEL 8 migration schedule is still up in the air, but hopefully it happens in Anna's time frame of "next year". I guess we'll see. Once all RFCs are on RHEL 8, we'll move to HAProxy 2.4, and ask Anna to test access again via the ticket. Thanks, Anna, for reporting this issue and let us know if you have any questions or new information, Hank |
My evaluations through the COWRES suddenly stopped working. I thought that there may be a problem connecting to the server but I am able to access the WRES website in windows. When I run my script I am getting an SSLV3_ALERT_HANDSHAKE_FAILURE error. I have not changed the certificate or the location of the certificate. Is anything going on in the back end with the COWRES?
The text was updated successfully, but these errors were encountered: