requests 2.32.3 with IPv6 link local address fails with error: [Errno -2] Name or service not known

[pytech66]

When using an IPv6 link local address with requests > v2.31.0, seeing an error [Errno -2] Name or service not known. The same works with requests 2.31.0.
urllib3 2.2.1 also works successfully with IPv6 link local address .

urllib3 2.2.1 with IPv6 link local address

>>> import urllib3
>>> http = urllib3.PoolManager(cert_reqs='CERT_NONE')
>>> res = http.request("GET", "https://[fe80::5eed:8cff:fe00:0da4%ens192]/redfish/v1")
/root/new-requests/venv/lib/python3.10/site-packages/urllib3/connectionpool.py:1103: InsecureRequestWarning: Unverified HTTPS request is being made to host 'fe80::5eed:8cff:fe00:0da4%ens192'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
>>> res.status
200

requests 2.31.0 with IPv6 link local address

>>> import requests
>>> res = requests.get("https://[fe80::5eed:8cff:fe00:0da4%ens192]/redfish/v1", verify=False)
>>> res.status_code
200

Expected Result

Successful connection with versions 2.32.x

Actual Result

>>> import requests
>>> res = requests.get("https://[fe80::5eed:8cff:fe00:0da4%ens192]/redfish/v1", verify=False)
Traceback (most recent call last):
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/connection.py", line 616, in connect
    self.sock = sock = self._new_conn()
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/connection.py", line 205, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x716b63d403d0>: Failed to resolve 'fe80::5eed:8cf00:0da4%25ens192' ([Errno -2] Name or service not known)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/new-requests/venv/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "/root/new-requests/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='fe80::5eed:8cff:fe00:0da4%25ens192', port=443): Max retries exceeded with urredfish/v1 (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x716b63d403d0>: Failed to resolve 'fe80::5eed::fe00:0da4%25ens192' ([Errno -2] Name or service not known)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/new-requests/venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/root/new-requests/venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/root/new-requests/venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/new-requests/venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/root/new-requests/venv/lib/python3.10/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='fe80::5eed:8cff:fe00:0da4%25ens192', port=443): Max retries exceeded with: /redfish/v1 (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x716b63d403d0>: Failed to resolve 'fe80::5ecff:fe00:0da4%25ens192' ([Errno -2] Name or service not known)"))

Reproduction Steps

import requests
res = requests.get("https://[fe80::5eed:8cff:fe00:0da4%ens192]/redfish/v1", verify=False)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.12"
  },
  "platform": {
    "release": "6.5.0-27-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.32.3"
  },
  "system_ssl": {
    "version": "30000020"
  },
  "urllib3": {
    "version": "2.2.1"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

[mattiaswal]

I worked around the bug with:

# Workaround for bug in requests 2.32.x: https://github.com/psf/requests/issues/6735
def requests_workaround(method, url, json, headers, auth, verify=False):
    # Create a session
        session=requests.Session()

        # Prepare the request
        request=requests.Request(method, url, json=json, headers=headers, auth=auth)
        prepared_request=session.prepare_request(request)
        prepared_request.url=prepared_request.url.replace('%25', '%')
        return session.send(prepared_request, verify=verify)

def requests_workaround_put(url, json, headers, auth, verify=False):
    return requests_workaround('PUT', url, json, headers, auth, verify=False)

def requests_workaround_delete(url, headers, auth, verify=False):
    return requests_workaround('DELETE', url, None, headers, auth, verify=False)

def requests_workaround_post(url, json, headers, auth, verify=False):
    return requests_workaround('POST', url, json, headers, auth, verify=False)

def requests_workaround_get(url, headers, auth, verify=False):
    return requests_workaround('GET', url, None, headers, auth, verify=False)

[xiachen-rh]

Hello, I hit the problem too when using python3-requests-2.32.3-2.
Reproduced steps (# python testre.py )

import requests
res = requests.get("http://[fe80::a9fe:a9fe%25enp3s0]/openstack", verify=False)

I found the problem is caused by the changes in this commit c0813a2

requests/adapters.py
def _urllib3_request_context(
...
    parsed_request_url = urlparse(request.url)
...
    host_params = {
        "scheme": scheme,
        "host": parsed_request_url.hostname,
        "port": port,
    }

When the url contains IPv6 address, urllib.urlparse.hostname does not contain the chars "[" "]", but urllib3.util.url._normalize_host matches ipv6 host with chars "[" "]". so the exception is thrown out.

The fix could be using urllib3.util.parse_url() instead of urlparse()

parsed_request_url = parse_url(request.url) # from urllib3.util

OR using netloc instead of hostname

    host_params = { 
        "scheme": scheme,
        "host": parsed_request_url.netloc,
        "port": port,
    }

@sigmavirus24 @nateprewitt What do you think of it?

Here are my system information

# python -m requests.help
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.12.8"
  },
  "platform": {
    "release": "6.12.0-35.el*.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.32.3"
  },
  "system_ssl": {
    "version": "30200020"
  },
  "urllib3": {
    "version": "1.26.19"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

and my debug logs, I adding some print messages into code to debug the issue.

# python testre.py 
url._normalize_host host: [fe80::a9fe:a9fe%25enp3s0]
url._normalize_host is_ipv6: <re.Match object; span=(0, 26), match='[fe80::a9fe:a9fe%25enp3s0]'>
checking request http://[fe80::a9fe:a9fe%25enp3s0]/openstack
checking send
checking adapter: <requests.adapters.HTTPAdapter object at 0x7f4ddf5d0ad0>
if using urllib.urlparse: ParseResult(scheme='http', netloc='[fe80::a9fe:a9fe%25enp3s0]', path='/openstack', params='', query='', fragment='')
requests.adapter parsed_request_url: ParseResult(scheme='http', netloc='[fe80::a9fe:a9fe%25enp3s0]', path='/openstack', params='', query='', fragment='')
**the problem is here: {'scheme': 'http', 'host': 'fe80::a9fe:a9fe%25enp3s0', 'port': None}**
HttpConnectionPool init: fe80::a9fe:a9fe%25enp3s0
###init host: fe80::a9fe:a9fe%25enp3s0
url._normalize_host host: fe80::a9fe:a9fe%25enp3s0
url._normalize_host is_ipv6: None
connection host: fe80::a9fe:a9fe%25enp3s0
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/socket.py", line 976, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 417, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.12/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib64/python3.12/http/client.py", line 1336, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.12/http/client.py", line 1382, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.12/http/client.py", line 1035, in send
    self.connect()
  File "/usr/lib/python3.12/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f4ddf5d0980>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/requests/adapters.py", line 672, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 802, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/urllib3/util/retry.py", line 594, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='fe80::a9fe:a9fe%25enp3s0', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4ddf5d0980>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/testre.py", line 2, in <module>
    res = requests.get("http://[fe80::a9fe:a9fe%25enp3s0]/openstack", verify=False)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/requests/sessions.py", line 590, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/requests/sessions.py", line 705, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/requests/adapters.py", line 705, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='fe80::a9fe:a9fe%25enp3s0', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4ddf5d0980>: Failed to establish a new connection: [Errno -2] Name or service not known'))

When using urllib3.util.parse_url, the logs are

# python testre.py 
url._normalize_host host: [fe80::a9fe:a9fe%25enp3s0]
url._normalize_host is_ipv6: <re.Match object; span=(0, 26), match='[fe80::a9fe:a9fe%25enp3s0]'>
checking request http://[fe80::a9fe:a9fe%25enp3s0]/openstack
checking send
checking adapter: <requests.adapters.HTTPAdapter object at 0x7f2635e258e0>
url._normalize_host host: [fe80::a9fe:a9fe%25enp3s0]
url._normalize_host is_ipv6: <re.Match object; span=(0, 26), match='[fe80::a9fe:a9fe%25enp3s0]'>
if using urllib3.util.parse_url: http://[fe80::a9fe:a9fe%enp3s0]/openstack
requests.adapter parsed_request_url: http://[fe80::a9fe:a9fe%enp3s0]/openstack
the problem is fixed: {'scheme': 'http', 'host': '[fe80::a9fe:a9fe%enp3s0]', 'port': None}
HttpConnectionPool init: [fe80::a9fe:a9fe%enp3s0]
###init host: [fe80::a9fe:a9fe%enp3s0]
url._normalize_host host: [fe80::a9fe:a9fe%enp3s0]
url._normalize_host is_ipv6: <re.Match object; span=(0, 24), match='[fe80::a9fe:a9fe%enp3s0]'>
connectionpool._normalize_host: fe80::a9fe:a9fe%enp3s0
connection host: fe80::a9fe:a9fe%enp3s0

When using netloc instead of hostname, the logs are

# python testre.py 
url._normalize_host host: [fe80::a9fe:a9fe%25enp3s0]
url._normalize_host is_ipv6: <re.Match object; span=(0, 26), match='[fe80::a9fe:a9fe%25enp3s0]'>
checking request http://[fe80::a9fe:a9fe%25enp3s0]/openstack
checking send
checking adapter: <requests.adapters.HTTPAdapter object at 0x7f0aacced580>
if using urllib.urlparse: ParseResult(scheme='http', netloc='[fe80::a9fe:a9fe%25enp3s0]', path='/openstack', params='', query='', fragment='')
requests.adapter parsed_request_url: ParseResult(scheme='http', netloc='[fe80::a9fe:a9fe%25enp3s0]', path='/openstack', params='', query='', fragment='')
the problem is fixed: {'scheme': 'http', 'host': '[fe80::a9fe:a9fe%25enp3s0]', 'port': None}
HttpConnectionPool init: [fe80::a9fe:a9fe%25enp3s0]
###init host: [fe80::a9fe:a9fe%25enp3s0]
url._normalize_host host: [fe80::a9fe:a9fe%25enp3s0]
url._normalize_host is_ipv6: <re.Match object; span=(0, 26), match='[fe80::a9fe:a9fe%25enp3s0]'>
connectionpool._normalize_host: fe80::a9fe:a9fe%enp3s0
connection host: fe80::a9fe:a9fe%enp3s0