Root zone NSEC should be returned for missing DS in zone, not NS referral

[pinheadmz]

See handshake-org/hsd#291 (comment)

Longer discussion went down on telegram but heres some important quotes:

DS is wierd
it belongs to the parent zone
so its the parent zone that needs to proove it doesn't exist

HNS root returns referral if DS is missing

(--ns-port=5300)

$ dig js ds +dnssec @127.0.0.1 -p 5300

; <<>> DiG 9.16.1-Ubuntu <<>> js ds +dnssec @127.0.0.1 -p 5300
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56349
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 5, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;js.				IN	DS

;; AUTHORITY SECTION:
js.			21600	IN	NS	ns2.nameserver.io.
js.			21600	IN	NS	ns1.nameserver.io.
js.			21600	IN	NS	ns3.nameserver.io.
js.			21600	IN	NS	ns4.nameserver.io.
js.			10800	IN	RRSIG	NS 13 1 21600 20210626153707 20200625153707 60944 . whxQmC/AXf90C/xY+z2kfUKRMpgIgxfMiVYB/HuzdrAon7Cf6xTRVzai P7jOdrn9unHpABmViJ+xK7k4Kbsatw==

;; SIG0 PSEUDOSECTION:
.			0	ANY	SIG	0 253 0 0 20200626213707 20200626093707 27687 . Kb1AOc4Jfyl2qXW3QgCjhfuVIBwXzVYTqCOtQRs2799yvNnat6hDqsD4 /Q9yrYYganNHLFfI1lYRbaGtoqL8Kg==

;; Query time: 184 msec
;; SERVER: 127.0.0.1#5300(127.0.0.1)
;; WHEN: Fri Jun 26 11:37:07 EDT 2020
;; MSG SIZE  rcvd: 305

Legacy DNS root returns NSEC from root if DS is missing

$ dig zw ds +dnssec @e.root-servers.net

; <<>> DiG 9.16.1-Ubuntu <<>> zw ds +dnssec @e.root-servers.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48613
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;zw.				IN	DS

;; AUTHORITY SECTION:
.			86400	IN	SOA	a.root-servers.net. nstld.verisign-grs.com. 2020062600 1800 900 604800 86400
.			86400	IN	RRSIG	SOA 8 0 86400 20200709050000 20200626040000 48903 . ZYoQbgv4WW1+/saL6BPSJjEpo3Ci7SFTgyK1AlRO62pogyJavdy+3U5e 5SXfT/ZWvj1wlFzz+I+fw093WOTwSs2zsqIRACzM0H4LfLJ8T6Fcltdi sHoLwWFPByWoMolOhRj2TvONrJZIld6DsbqPfpVzfZ3BKVNZ2V+tJF8q 5Hdxpn1uDwUndQcu58pzQ9sY0R0LTZJAIlRqQEai2HrvWgXXzuXzARkb KbdnBMi+JCYZMSisNxn/VHwAW6mYMmSufaPZ5CmpKCCNvgMMqzCorE1h VAPdsKsLnq5+UQmb114EhbE920yqEMHQpGI78a3gCPdHhun1IXkd9etC c758AQ==
zw.			86400	IN	NSEC	. NS RRSIG NSEC
zw.			86400	IN	RRSIG	NSEC 8 1 86400 20200709050000 20200626040000 48903 . XNkyjdhGGspX4ZgdqndU9YSWRs8ovA+QTraldzMvVfuUKfQ5yu5Jq9MG 12vxe0RwBzqZEWMfqQMS7Z9Nto0o2eqYl0d0UpE913kKiec4go7AthKS NSPp3t78y7r5Dd/K5cJnGxEHuI2Jvbms5DoZ6tVI5FvN+PHiFx6vmILb d+DErt/PkdnBR/SgmSQw8o0OoA3yGMTPX8D8R/Ocf7dLaDep7oiMdImo fFxbelB+imC/d7QI1K+K6EGfXOHuSNNhVY1voUGkUzQeIeWj+28QCSRz 8Ypm83G1UJK0uyfvfK2jAnqrT2jppy0F8CTyoeemQNvTL8xSBwcVfqSe v5sWyA==

;; Query time: 16 msec
;; SERVER: 192.203.230.10#53(192.203.230.10)
;; WHEN: Fri Jun 26 11:37:37 EDT 2020
;; MSG SIZE  rcvd: 700

More stuff to look in to:

i've not checked all the proofs, some may not mattter, as this is a delegation-only domain
there are lots of places that use all sorts of different resolver s/w - using a machanism that conforms the RFC would mean any could work
there are five different basic proofs, two only apply to wildcard records - so apart from NXDOMAIN & NODATA there is "unsigned referral" (no DS record)
for NXDOMAIN you also need to prove there is no wildcard record that could have matched
ADD_NEXT_CLOSER|ADD_CLOSEST_ENCLOSER|ADD_NEXT_CLOSER_WILDCARD <- these are the NXDOMAIN proofs required
NODATA is just ADD_MY_NSEC
PROOF_NSEC_UNSIGNED_REFERRAL is ADD_NEXT_CLOSER|ADD_CLOSEST_ENCLOSER