Redirect issue requests not passing proxies through to redirect

[toasteez]

I'm trying to retrieve data via Quandl from behind a proxy.

The reason I have dropped into requests is that the Quandl library does not allow a proxy dictionary to be passed.

I can successfully retrieve datasets where there is no redirect however when you try to retrieve a complete database there is a redirect

from https://www.quandl.com/api/v3/databases/CLSM/data?api_key ... (cannot provide my private key)

to - https://quandl-bulk-download.s3.amazonaws.com/CLSM.zip? ..... (Amazon credentials are created here)

Code

response = requests.get(url, proxies=proxies, allow_redirects=False) # This lets me see the headers, without failing 

response = requests.get(url, proxies=proxies) # This fails OSError: Tunnel connection failed: 407 Proxy Authentication Required

The initial request returns a status 302.

{'X-Runtime': '0.079339', 'X-XSS-Protection': '1; mode=block', 'Connection': 'keep-alive', 'Content-Length': '1059', 'Cache-Control': 'no-cache', 'X-RateLimit-Remaining': '999901', 'Vary': 'Origin', 'X-Content-Type-Options': 'nosniff', 'Location': 'https://quandl-bulk-download.s3.amazonaws.com/CLSM.zip?...SignedHeaders=host&X-Amz-Signature=... , 'X-Frame-Options': 'SAMEORIGIN', 'X-Rack-CORS': 'preflight-hit; no-origin', 'CF-RAY': '...., 'Set-Cookie': '__cfduid=...; expires=Wed, 28-Feb-18 10:00:16 GMT; path=/; domain=.quandl.com; HttpOnly', 'Server': 'cloudflare-nginx', 'Content-Type': 'text/html; charset=utf-8', 'Date': 'Tue, 28 Feb 2017 10:00:17 GMT', 'X-RateLimit-Limit': '1000000', 'X-Request-Id': ...}

Is there still a bug in this area or should I be doing something differently?

Note that IE prompts for download of file and Chrome just works.

Version
Python 3.5.2
requests 2.13.0

[Lukasa]

It seems like your proxy requires that you authorize to it. Do you have credentials for that proxy?

[toasteez]

The proxy auto authorizes for a simple dataset request via quandl but not for the redirect. I tested this by removing the proxies=proxies (the request fails) witha MaxRetryError: HTTPSConnectionPool(host='www.quandl.com', port=443): With the proxy dict it is successful. (this is without a redirect to amazon)

I'm on windows so it uses either Kerberos or NTLM not sure which?

[Lukasa]

Requests does not transparently support Kerberos or NTLM. You'd need to use something like requests-kerberos or requests-ntlm to support those auth challenges. However, I don't think either support authing to CONNECT tunnelling proxies via those protocols.

[toasteez]

Why would 1st connection to quandl.com authenticate via proxy automatically but the redirect does not?

[Lukasa]

@toasteez Can you show me the proxy dictionary?

[toasteez]

proxies = {'http': 'http://proxy.company.org:8080',
'https': 'http://proxy.company.org:8080',}

[Lukasa]

So, my best guess is that the first request does not need auth because the proxy server has decided that requests to that hostname (www.quandl.com) do not need authorization, while requests to the second hostname (which belongs to Amazon) do. That is, that this is a proxy concern.

[toasteez]

OK. Thanks. How do I pass the auth and pw via requests to try this out? My preferred option is to make this seamless without hard coding my credentials anywhere though.

[Lukasa]

If you can auth to the proxy using basic auth you need to pass them in the URL to the proxy, e.g. {'https': 'http://username:password@proxy.company.org:8080'}. If you need to auth using Kerberos or NTLM you'll need to consult the documentation for those modules.

[toasteez]

Or could this be wrapped up somehow to work?

import win32com.client

url = 'https://...'

h = win32com.client.Dispatch('WinHTTP.WinHTTPRequest.5.1')
h.SetAutoLogonPolicy(0)
h.Open('GET', url, False)
h.Send()
result = h.responseText
result

[Lukasa]

It's not at all clear to me what you're trying to do with the above code.

[toasteez]

Ignoring the above the adding the user:password to the proxy results in:

SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

[Lukasa]

Ok, so it looks very much like the proxy is attempting to place itself as a man-in-the-middle on your HTTPS connection by decrypting the traffic to Amazon. To allow that you'll need to get the certificate authority your proxy is using to build its TLS certificates and pass the path to that cert to verify.

[toasteez]

OK thanks, is there a way to not have to have the user:pass in any script?

[Lukasa]

Yes. Requests supports reading proxy data from the environment variables HTTP_PROXY and HTTPS_PROXY. You can set those up to have the same strings in them that you have as the values of the proxy dictionary. The proxy dictionary then becomes unnecessary. Otherwise, you should write code to read them from config files.