Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rec: Duplicate NSEC and RRSIG in foo.cname-exists.phicoh.nl #14120

Open
2 tasks done
mnordhoff opened this issue Apr 30, 2024 · 6 comments · May be fixed by #14617
Open
2 tasks done

rec: Duplicate NSEC and RRSIG in foo.cname-exists.phicoh.nl #14120

mnordhoff opened this issue Apr 30, 2024 · 6 comments · May be fixed by #14617
Assignees
@omoerbeek
Labels
defect rec
Milestone
rec-5.3.0

Comments

@mnordhoff
Copy link
Contributor

Short description

Someone on the DNS-OARC Mattermost found that "dig +dnssec foo.cname-exists.phicoh.nl txt" on "a popular public resolver" returns two "does-exist.phicoh.nl. 3600 IN NSEC *.ent.phicoh.nl. A RRSIG NSEC" records. I don't know which resolver they meant, but I checked a few of them and PowerDNS Recursor does in fact return the NSEC and its corresponding RRSIG twice.

Beats me why, but there are wildcards and ENTs and possibly other horrors involved.

There may have been recent changes in this area, but I don't know if it's a new issue. It affects Quad9, so I doubt it's that new.

Environment

  • Operating system: Ubuntu 20.04 amd64
  • Software version: 5.1.0~alpha0+master.1387.g0b6eb67e1-1pdns.focal and 5.1.0~alpha0+master.1406.g5e386641f-1pdns.focal
  • Software source: PowerDNS repository

Steps to reproduce

  1. dig +dnssec foo.cname-exists.phicoh.nl txt

Expected behaviour

No duplicate records.

Actual behaviour

Extra NSEC and RRSIG records.

Other information

$ dig +dnssec foo.cname-exists.phicoh.nl txt
;; Truncated, retrying in TCP mode.

; <<>> DiG 9.19.22-1+ubuntu20.04.1+deb.sury.org+1-Ubuntu <<>> +dnssec foo.cname-exists.phicoh.nl txt
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46879
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 8, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;foo.cname-exists.phicoh.nl.    IN      TXT

;; ANSWER SECTION:
foo.cname-exists.phicoh.nl. 3600 IN     CNAME   does-exist.phicoh.nl.
foo.cname-exists.phicoh.nl. 3600 IN     RRSIG   CNAME 8 3 3600 20240523132800 20240425132800 63939 phicoh.nl. gGJOiNSXdI2IZzEeyx2JTnT+djjR9i2gIhRYShC3yVp++2va9qYBdivB gXZgEmMnNb1t2yA6b7DmVd9Vyi1XkBYQU6Zhr10ItwNZvNynC/kNmFAY T6GSP4AZZxX4wU1cNgOo5Rzm01Hk6VSG8ipDGNvxRA9HheZO7mZizkoz hphh9xhBCsqHbnj8liwyqwESJGhjJMuCBLDkouN4ak/AkII2xa+ycvno UK4rsdz4zT8jif0mN1kt8nfKu8WuXozcHKoUvWW03ZPvvCbmYaY8k/rq pSVO04nr7EZQG2E9yT72XhFnfLJg2+rYbY9Ryyfe/+aAHRHtfxkvIx8n obxgsA==

;; AUTHORITY SECTION:
*.cname-exists.phicoh.nl. 3600  IN      NSEC    cname-nxdomain.phicoh.nl. CNAME RRSIG NSEC
does-exist.phicoh.nl.   3600    IN      NSEC    *.ent.phicoh.nl. A RRSIG NSEC
phicoh.nl.              3600    IN      SOA     stereo.hq.phicoh.net. dnsmaster.hq.phicoh.net. 1714051679 7200 3600 2419200 3600
does-exist.phicoh.nl.   3600    IN      NSEC    *.ent.phicoh.nl. A RRSIG NSEC
*.cname-exists.phicoh.nl. 3600  IN      RRSIG   NSEC 8 3 3600 20240523132800 20240425132800 63939 phicoh.nl. qn63vY9jK1QAPmMGOk2igSdOWRYwg2oCwH7okDVPPWbKO0SMENdPLoSF slgR0+Ck2e6zV7Nf1ZawSRMFU0u1H2KML56iSELjQOJW34zCiwxcox/A 6BrZ+gld1Ak6PiT/X41yDbMlDwz4usmpRhP3nG6qU01Pjd9vy/38HP3V QEj9EMuFDJPzlavO3dE7uCMSjW/vzXPvqg295yQ1f3fvV6D55PVJWN4Q cDI6hSAyPG/dVq24zqklADrb2lh+JNwHR4WsgBjgI4pFBxV2SQnTcqYz /2lihopU6qD5b99hw51fKuYe/xpKLvFxHwQ9Ccud9TDK64wggJ6cjvef HwmUBg==
does-exist.phicoh.nl.   3600    IN      RRSIG   NSEC 8 3 3600 20240523132801 20240425132801 63939 phicoh.nl. AmkFNWd0GFouYmDjOsYaTpSirHH8n2UNThEb6IZ2m8CKkNEVUApDtP4N Jpb88orjpeDM9THZnD+/zQbx4Qpp4GsMRvUAtTizcZDpOQ1esN5zSTJM fJRDfyFDPFlm+NPyzDaqbBaFSyzTVrKECBVfDW3QVTW7trB03BZSg+Gf kLhti9jA0dPq9nauFzILeCdT0k8vhhrhf4Gc5repYy5otAAPJBnR3cgQ 8zSobeC/kEu8mvvSWjxStDsG5P86hj3mKdj9Yqx5Sj6/Piug4y2D6lva aIg0C+cSstePrwKbpKEgkycWQnIa4MWYAJ1auS7HJ5NJX9hAL/XNm64U ILAGSA==
phicoh.nl.              3600    IN      RRSIG   SOA 8 2 3600 20240523132759 20240425132759 63939 phicoh.nl. Ve5TIpO5DnxpWpsOOlYPNKSKHu1zAXQOESWKmrfv40qnBw/8FYBZCnPe usLafNFDtDZYy4+g2BMPTq5aHOluF+GtHfR/Tnb+9Y27r7F42LfgWI/S NZ3hZ+445PeDJIhJ55oJ+HjlWoNWIQGYNyLXRmCfteKhYEUwN3xQdz+t 0wgC6gXENpg4Dbk2WzEHNx6RtkwpvUSJd6HyvSsS+1VYoCL2NgteVhp0 paTYSYnn9fwAZDyGUUbMUS7Pbeot393lV+5OFG8OuwnTlO4i12Bx9Rk7 uo55fJiHmtWsgJCuv8lkFTLpoAKH0+9gDFZ7V80nZf2Re9a9/PJVs46d 4JKCyg==
does-exist.phicoh.nl.   3600    IN      RRSIG   NSEC 8 3 3600 20240523132801 20240425132801 63939 phicoh.nl. AmkFNWd0GFouYmDjOsYaTpSirHH8n2UNThEb6IZ2m8CKkNEVUApDtP4N J
pb88orjpeDM9THZnD+/zQbx4Qpp4GsMRvUAtTizcZDpOQ1esN5zSTJM fJRDfyFDPFlm+NPyzDaqbBaFSyzTVrKECBVfDW3QVTW7trB03BZSg+Gf kLhti9jA0dPq9nauFzILeCdT0k8vhhrhf4Gc5repYy5otAAPJBnR3cgQ 8zSobeC/kEu8mvvSWjxStDsG5P86hj3mKdj9Yqx5Sj6/Piug4y2D6lva aIg0C+cSstePrwKbpKEgkycWQnIa4MWYAJ1auS7HJ5NJX9hAL/XNm64U ILAGSA==

;; Query time: 4 msec
;; SERVER: ::1#53(::1) (TCP)
;; WHEN: Tue Apr 30 11:02:47 UTC 2024
;; MSG SIZE  rcvd: 1753
@omoerbeek omoerbeek added the rec label May 6, 2024
@omoerbeek omoerbeek added this to the rec-5.1.0 milestone May 6, 2024
@omoerbeek omoerbeek self-assigned this May 6, 2024
@omoerbeek
Copy link
Member

omoerbeek commented May 6, 2024 •
edited
Loading

Rec constructs the answer in two steps, first foo.cname-exists.phicoh.nl is resolved, resulting in a CNAME record and an AUTHORITY section containing the future dup NSEC and RRSIG records:

 dig @ns5.phicoh.nl foo.cname-exists.phicoh.nl txt +dnssec

; <<>> DiG 9.18.25 <<>> @ns5.phicoh.nl foo.cname-exists.phicoh.nl txt +dnssec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9313
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1280
;; QUESTION SECTION:
;foo.cname-exists.phicoh.nl.	IN	TXT

;; ANSWER SECTION:
foo.cname-exists.phicoh.nl. 3600 IN	CNAME	does-exist.phicoh.nl.
foo.cname-exists.phicoh.nl. 3600 IN	RRSIG	CNAME 8 3 3600 20240523132800 20240425132800 63939 phicoh.nl. gGJOiNSXdI2IZzEeyx2JTnT+djjR9i2gIhRYShC3yVp++2va9qYBdivB gXZgEmMnNb1t2yA6b7DmVd9Vyi1XkBYQU6Zhr10ItwNZvNynC/kNmFAY T6GSP4AZZxX4wU1cNgOo5Rzm01Hk6VSG8ipDGNvxRA9HheZO7mZizkoz hphh9xhBCsqHbnj8liwyqwESJGhjJMuCBLDkouN4ak/AkII2xa+ycvno UK4rsdz4zT8jif0mN1kt8nfKu8WuXozcHKoUvWW03ZPvvCbmYaY8k/rq pSVO04nr7EZQG2E9yT72XhFnfLJg2+rYbY9Ryyfe/+aAHRHtfxkvIx8n obxgsA==

;; AUTHORITY SECTION:
*.cname-exists.phicoh.nl. 3600	IN	NSEC	cname-nxdomain.phicoh.nl. CNAME RRSIG NSEC
*.cname-exists.phicoh.nl. 3600	IN	RRSIG	NSEC 8 3 3600 20240523132800 20240425132800 63939 phicoh.nl. qn63vY9jK1QAPmMGOk2igSdOWRYwg2oCwH7okDVPPWbKO0SMENdPLoSF slgR0+Ck2e6zV7Nf1ZawSRMFU0u1H2KML56iSELjQOJW34zCiwxcox/A 6BrZ+gld1Ak6PiT/X41yDbMlDwz4usmpRhP3nG6qU01Pjd9vy/38HP3V QEj9EMuFDJPzlavO3dE7uCMSjW/vzXPvqg295yQ1f3fvV6D55PVJWN4Q cDI6hSAyPG/dVq24zqklADrb2lh+JNwHR4WsgBjgI4pFBxV2SQnTcqYz /2lihopU6qD5b99hw51fKuYe/xpKLvFxHwQ9Ccud9TDK64wggJ6cjvef HwmUBg==
does-exist.phicoh.nl.	3600	IN	NSEC	*.ent.phicoh.nl. A RRSIG NSEC
does-exist.phicoh.nl.	3600	IN	RRSIG	NSEC 8 3 3600 20240523132801 20240425132801 63939 phicoh.nl. AmkFNWd0GFouYmDjOsYaTpSirHH8n2UNThEb6IZ2m8CKkNEVUApDtP4N Jpb88orjpeDM9THZnD+/zQbx4Qpp4GsMRvUAtTizcZDpOQ1esN5zSTJM fJRDfyFDPFlm+NPyzDaqbBaFSyzTVrKECBVfDW3QVTW7trB03BZSg+Gf kLhti9jA0dPq9nauFzILeCdT0k8vhhrhf4Gc5repYy5otAAPJBnR3cgQ 8zSobeC/kEu8mvvSWjxStDsG5P86hj3mKdj9Yqx5Sj6/Piug4y2D6lva aIg0C+cSstePrwKbpKEgkycWQnIa4MWYAJ1auS7HJ5NJX9hAL/XNm64U ILAGSA==

;; Query time: 24 msec
;; SERVER: 2a10:3781:2413:1:2a0:c9ff:fe9f:17a9#53(ns5.phicoh.nl) (UDP)
;; WHEN: Mon May 06 08:16:36 CEST 2024
;; MSG SIZE  rcvd: 1056

Then the CNAME target does-exist.phicoh.nl is resolved, resulting in an AUTHORITY section containing the already present records again:

$ dig @ns5.phicoh.nl does-exist.phicoh.nl txt +dnssec

; <<>> DiG 9.18.25 <<>> @ns5.phicoh.nl does-exist.phicoh.nl txt +dnssec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3911
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1280
;; QUESTION SECTION:
;does-exist.phicoh.nl.		IN	TXT

;; AUTHORITY SECTION:
phicoh.nl.		3600	IN	SOA	stereo.hq.phicoh.net. dnsmaster.hq.phicoh.net. 1714051679 7200 3600 2419200 3600
phicoh.nl.		3600	IN	RRSIG	SOA 8 2 3600 20240523132759 20240425132759 63939 phicoh.nl. Ve5TIpO5DnxpWpsOOlYPNKSKHu1zAXQOESWKmrfv40qnBw/8FYBZCnPe usLafNFDtDZYy4+g2BMPTq5aHOluF+GtHfR/Tnb+9Y27r7F42LfgWI/S NZ3hZ+445PeDJIhJ55oJ+HjlWoNWIQGYNyLXRmCfteKhYEUwN3xQdz+t 0wgC6gXENpg4Dbk2WzEHNx6RtkwpvUSJd6HyvSsS+1VYoCL2NgteVhp0 paTYSYnn9fwAZDyGUUbMUS7Pbeot393lV+5OFG8OuwnTlO4i12Bx9Rk7 uo55fJiHmtWsgJCuv8lkFTLpoAKH0+9gDFZ7V80nZf2Re9a9/PJVs46d 4JKCyg==
does-exist.phicoh.nl.	3600	IN	NSEC	*.ent.phicoh.nl. A RRSIG NSEC
does-exist.phicoh.nl.	3600	IN	RRSIG	NSEC 8 3 3600 20240523132801 20240425132801 63939 phicoh.nl. AmkFNWd0GFouYmDjOsYaTpSirHH8n2UNThEb6IZ2m8CKkNEVUApDtP4N Jpb88orjpeDM9THZnD+/zQbx4Qpp4GsMRvUAtTizcZDpOQ1esN5zSTJM fJRDfyFDPFlm+NPyzDaqbBaFSyzTVrKECBVfDW3QVTW7trB03BZSg+Gf kLhti9jA0dPq9nauFzILeCdT0k8vhhrhf4Gc5repYy5otAAPJBnR3cgQ 8zSobeC/kEu8mvvSWjxStDsG5P86hj3mKdj9Yqx5Sj6/Piug4y2D6lva aIg0C+cSstePrwKbpKEgkycWQnIa4MWYAJ1auS7HJ5NJX9hAL/XNm64U ILAGSA==

;; Query time: 25 msec
;; SERVER: 2a10:3781:2413:1:2a0:c9ff:fe9f:17a9#53(ns5.phicoh.nl) (UDP)
;; WHEN: Mon May 06 08:17:08 CEST 2024
;; MSG SIZE  rcvd: 746

The auth sections are then merged, but contain duplicates.

@rgacogne
Copy link
Member

rgacogne commented May 6, 2024

Right, the NSEC for *.cname-exists.phicoh.nl. both proves that foo.cname-exists.phicoh.nl. does not exist and that *.cname-exists.phicoh.nl. does not have a TXT.

@rgacogne
Copy link
Member

rgacogne commented May 6, 2024

I wonder if it would make sense to use a set instead of a vector for these?

@omoerbeek
Copy link
Member

I wonder if it would make sense to use a set instead of a vector for these?

Don't know yet, have to investigate, as we might be using the (implicit) ordering of the vector.

@omoerbeek
Copy link
Member

Currently the code builds the complete answer in a vector. The order of the answers is mostly preserved (after sorting in sections) by the shuffling code. If we want to start using a duplicate-avoiding data structure we still want to keep the order of inserts (like the order-preserving arrays/set hybrid available in some languages). So this is a no small change, as lots of code assume they just need to append to a vector.

As an experiment I created a diff to fix this particular case, but not very happy about it, it is very specific to the test case, though the duplicate record problem might be a general one. This raises the question: what if an auth sends us duplicates. Do we want to dedup these as well? Or only if the answer is constructed from several replies?

Conclusion: this needs more thought. Experimental diff attached.

x.diff.txt

@omoerbeek
Copy link
Member

See also #14362, another instance of an edge case that causes duplicate records.

@omoerbeek omoerbeek modified the milestones: rec-5.1.0, rec-5.2.0 Jul 17, 2024
@omoerbeek omoerbeek linked a pull request Sep 3, 2024 that will close this issue
Open
7 tasks
@omoerbeek omoerbeek modified the milestones: rec-5.2.0, rec-5.3.0 Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@omoerbeek omoerbeek

Labels
defect rec
Projects
None yet
Milestone
rec-5.3.0
Development

Successfully merging a pull request may close this issue.

rec: dedup records omoerbeek/pdns
3 participants
@mnordhoff @rgacogne @omoerbeek