Haproxy + DNSDIST (DoT, DoH) enableProxyProtocol : exception Calling tryRead() with a too small buffer

[mineexchange89]

dnsdist --version

Enabled features: cdb dns-over-quic dns-over-http3 dns-over-tls(gnutls openssl) dns-over-https(h2o nghttp2) dnscrypt ebpf fstrm ipcipher libedit libsodium lmdb protobuf re2 recvmmsg/sendmmsg snmp systemd

haproxy -vv

Status: long-term supported branch - will stop receiving fixes around Q2 2029.
Known bugs: http://www.haproxy.org/bugs/bugs-3.0.2.html
Running on: Linux 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64
Build options :
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_OT=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL +OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 3.0.2 15 Mar 2022
Running on OpenSSL version : OpenSSL 3.0.2 15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTracing support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[  OT] opentracing
	[SPOE] spoe
	[TRACE] trace

/etc/dnsdist/dnsdist.conf

-- The IPs and ports to listen on
addLocal("0.0.0.0:53")

-- Allow all IPs access
setACL({'0.0.0.0/0', '::/0'})
addACL('2001:db8::/32')  -- Add IPv6 range


-- Logging configuration
addAction(AllRule(), LogAction("/var/log/dnsdist.log", false, true, false, true, true))
setVerboseHealthChecks(true)

-- Downstream Servers for DNS resolving
newServer("1.1.1.3")
newServer("1.1.1.1")

-- Set ACL for PROXY protocol
setProxyProtocolACL("127.0.0.1/32")

-- Set a longer timeout
setTCPRecvTimeout(10)
setTCPSendTimeout(10)

addTLSLocal('127.0.0.1:8853', "/etc/dnsdist/ssl/cert.pem", "/etc/dnsdist/ssl/privkey.pem", {
    provider="openssl",
    minTLSVersion="tls1.2",
    maxTLSVersion="tls1.3",
    ciphers="ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305",
    ciphersTLS13="TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256",
    enableProxyProtocol=true,
    proxyProtocolOutsideTLS=true,

})
addDOHLocal("127.0.0.1:8053", nil, nil, "/dns-query", { reusePort=true })
includeDirectory("/etc/dnsdist/conf.d")

/etc/haproxy/haproxy.cfg

    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    maxconn 100000
    lua-load /etc/haproxy/client_ip.lua
    tune.ssl.default-dh-param 2048
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
defaults
    log     global
    mode    http
    option  dontlognull
    timeout  connect 10s
    timeout  client 1m
    timeout  server 1m
    timeout  http-request 10s
    timeout  queue 1m
    timeout  check 10s
    log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ac/%fc/%bc/%sc/%rc %sq/%bq SNI=%[ssl_fc_sni]"

frontend DoH
    bind :::443 ssl crt /etc/haproxy/ssl/domain.tech.pem ca-file /etc/haproxy/ssl/ca.pem verify required alpn h2,http/1.1
    mode tcp
    option tcplog
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }
    tcp-request content set-var(txn.original_sni) req.ssl_sni
    tcp-request content set-var(txn.client_ip) src
    use_backend DOHDNS
    log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ac/%fc/%bc/%sc/%rc %sq/%bq SNI=%[ssl_fc_sni] SSL_VERSION=%[ssl_fc_protocol] CIPHER=%[ssl_fc_cipher]"

frontend DoT
    bind :::853 ssl crt /etc/haproxy/ssl/domain.tech.pem ca-file /etc/haproxy/ssl/ca.pem verify required alpn h2,http/1.1
    mode tcp
    option tcplog
    mode tcp
    option tcplog
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }
    tcp-request content set-var(txn.original_sni) req.ssl_sni
    tcp-request content set-var(txn.client_ip) src
    use_backend DOTDNS
    log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ac/%fc/%bc/%sc/%rc %sq/%bq SNI=%[ssl_fc_sni] SSL_VERSION=%[ssl_fc_protocol] CIPHER=%[ssl_fc_cipher]"
backend DOHDNS
    mode tcp
    option ssl-hello-chk
    server dnsdist 127.0.0.1:8053 sni ssl_fc_sni # Working but haproxy loopback IP is getting instead of  of real client 
    # server dnsdist 127.0.0.1:8053 sni ssl_fc_sni send-proxy-v2 # Working as expected with Ithe real client IP forwarding.
backend DOTDNS
    mode tcp
    option ssl-hello-chk
    #server dnsdist 127.0.0.1:8853 ssl verify none sni ssl_fc_sni # Working but haproxy loopback IP is getting instead of  of real client 
    server dnsdist 127.0.0.1:8853 ssl verify none sni ssl_fc_sni send-proxy-v2 # Not working getting ssl error 

Logs :

==> /var/log/dnsdist.log <==
Got TCP connection from 127.0.0.1:59426
Got an exception while handling (reading) TCP query from 127.0.0.1:59426: Calling tryRead() with a too small buffer (2) for a read of 18446744073709551566 bytes starting at 52

==> /var/log/haproxy.log <==
Sep 7 09:01:50 SERVER1 haproxy[4948]: ::ffff:106.213.87.184:32523 [07/Sep/2024:09:01:49.531] DoT~ DOTDNS/dnsdist 300/2/498 0 1/1/0/0/0 0/0 SNI=client.domain.tech SSL_VERSION=TLSv1.3 CIPHER=TLS_AES_256_GCM_SHA384

When using send-proxy-v2 with the DoT backend, we are encountering an exception, but the DoH backend works as expected. The goal is to pass the real client IP address to DNSDist through HAProxy.

Can anyone please suggest me what is wrong here.

[rgacogne]

With setProxyProtocolACL() unset, dnsdist will not expect a proxy protocol payload from the client.

[rgacogne]

Thanks! I was able to reproduce the issue, and I just opened a pull request fixing the issue in my tests: #14636
Would you, by any chance, be able to test the fix? If you are not building from the source code and are using our repository, I can build a package including the fix for a specific OS version, just let me know.

[mineexchange89]

Thank you very much. I am sorry, but I’m not building from the source code. It would be great if you could provide the build for Ubuntu 22.04 LTS.

[rgacogne]

Not a problem! I built a package for Ubuntu 22.04 LTS based on the current 1.9.x branch + the patch from the pull request mentioned above, it is available here: https://downloads.powerdns.com/tmp/4c709a5f-63bf-43af-ba3e-6efa7e61880e/ The version number is weird, sorry about that: please note that it means you will likely have to force the installation of this package over the existing one, and you might want to check that the upgrade is automatically done when a new official 1.9.x version is released.
Thanks a lot for reporting this issue, and please let us know how it goes if you are able to test the package :-)

[mineexchange89]

Thanks for the quick turnaround! I've tested the package, and it’s working as expected. The installation went smoothly, and I appreciate the fix. I’ll keep an eye out for the official release and make sure to manage the upgrade process accordingly.

Thanks again for your support!

[rgacogne]

Wonderful, thank you for testing and reporting back, much appreciated!