Can't get auto-primary/auto-secondary working

[bab5470]

I have the following config for my PowerDNS primary:

allow-axfr-ips=127.0.0.0/8,::1,10.60.10.11,208.80.120.32/32,208.80.120.39/32
api=yes
api-key=redacted
default-soa-content=dns1.ptp.media sysadmin.ptp.media. 0 10800 3600 604800 3600
default-ttl=3600
disable-axfr=no
expand-alias=yes
guardian=yes
launch=gmysql
gmysql-host=localhost
gmysql-user=pdns
gmysql-password=redacted
gmysql-dbname=pdns
gmysql-dnssec=yes
local-address=0.0.0.0, ::
local-port=53
log-timestamp=yes
loglevel=4
master=yes
primary=yes
resolver=127.0.0.1
secondary=no
security-poll-suffix=
setgid=pdns
setuid=pdns
slave=no
version-string=anonymous
webserver=yes

And the following on my secondary:

allow-axfr-ips=127.0.0.0/8,::1,10.6:0.10.10,208.80.120.32/32,208.80.120.39/32
allow-unsigned-autoprimary=yes
api=yes
api-key=redacted
autosecondary=yes
default-soa-content=dns1.ptp.media sysadmin.ptp.media. 0 10800 3600 604800 3600
default-ttl=3600
disable-axfr=no
expand-alias=yes
guardian=yes
launch=gmysql
gmysql-host=localhost
gmysql-user=pdns
gmysql-password=redacted
gmysql-dbname=pdns
gmysql-dnssec=yes
local-address=0.0.0.0, ::
local-port=53
log-timestamp=yes
loglevel=4
master=no
resolver=127.0.0.1
secondary=yes
security-poll-suffix=
setgid=pdns
setuid=pdns
slave=yes
slave-cycle-interval=60
superslave=yes
version-string=anonymous
webserver=yes

And I have the following in the MySQL database on my secondary:

$pdnsutil list-autoprimaries
IP=10.60.10.10, NS=dns1.ptp.media, account=
IP=10.60.10.10, NS=dns2.ptp.media, account=

I have over a hundred zones in PowerDNS but only one zone is currently pointed to PowerDNS (bumptv.com)

I have the following zone defined on the primary:

$pdnsutil list-zone bumptv.com
$ORIGIN .
bumptv.com      3600    IN      ALIAS   www.bumptv.com
bumptv.com      3600    IN      CAA     0 issue "letsencrypt.org"
bumptv.com      3600    IN      DNSKEY  257 3 13 BTYR1tb3ROu8aQF57DE0OEhAUo+x+12MsFwVdsUl4ZvweejCY9aR6+4WFDclm0l8L9f7Eoj60qaezxuuO90xCQ==
bumptv.com      3600    IN      DNSKEY  257 3 13 GhZ9bviORdd8vNEv/z6Coa4kwqXdSX9RGdpZimjjeOKDTqc9FhQ7ydiSM8Kp2u39PbYGB3Cb7ft2m7vJVj40Zw==
bumptv.com      60      IN      MX      10 aspmx.l.google.com.
bumptv.com      60      IN      MX      20 alt1.aspmx.l.google.com.
bumptv.com      60      IN      MX      20 alt2.aspmx.l.google.com.
bumptv.com      60      IN      MX      30 aspmx2.googlemail.com.
bumptv.com      60      IN      MX      30 aspmx3.googlemail.com.
bumptv.com      60      IN      NS      dns1.ptp.media.
bumptv.com      60      IN      NS      dns2.ptp.media.
bumptv.com      3600    IN      SOA     dns1.ptp.media sysadmin.ptp.media 2023092603 10800 3600 604800 3600
bumptv.com      60      IN      TXT     "google-site-verification=zeQ9xzxg9EWpjvmlgj1vWyx5gUCDYiQ7KinC448glcY"
bumptv.com      60      IN      TXT     "v=spf1 include:spf.pinktriangle.ca include:_spf.google.com include:spf.campaigner.com include:servers.mcsv.net  include:_spf.atlassian.net -all"
google._domainkey.bumptv.com    60      IN      TXT     "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuBOl7Lhrj8H+camSn+SKB/8m28vx+X7tnuSgJ+MCAy8PI8/z8jmsovLX71ufrHfgOTp1/ZdyCAYBloihtd+OR2Cq2DqBhd+E9bO3ii101pF171D4RJWcNDlvnpZdoSJzABf/X3+Kj26dL+aUzEpOHZ+IAUsD16nBgSczb3s7CcwBkpupkxh/ocHA0uLpzoCiJBcFC8GRMH/k3wmePFCIxWnL9sj5HLn4QwFXsDcrKofkYHMro5T6sv+hNUaW7CxQG1Qixb0hFMu+Piy1BQhLAot0auz7ZTIX+Cq99MRGzsMylrIDD61h6kA3QTA3vzZsqTcikY4/wx2gmWJ0BhQB2wIDAQAB"
test.bumptv.com 60      IN      A       1.2.3.4
www.bumptv.com  3600    IN      A       158.106.84.29

However, if I check the secondary:

$pdnsutil list-all-zones

There are no zones.

What am I doing wrong here? Checking the logs I see this:

Sep 26 11:50:07 DNS2 pdns_server[1444742]: Received NOTIFY for bumptv.com from 10.60.10.1 for which we are not authoritative, trying supermaster
Sep 26 11:50:11 DNS2 pdns_server[1444742]: Error resolving SOA or NS for bumptv.com at: 10.60.10.1: Timeout waiting for answer from 10.60.10.1

I don't get why it's trying 10.60.10.1 that's not what I specified as the address of the auto primary in MySQL. Where is it getting 10.60.10.1 from?

I did validate that it is 10.60.10.10 in the supermasters database table not 10.60.10.1. Unless PowerDNS is truncating the last octet for some reason?

10.60.10.1 is our gateway - but I am not sure why it would be trying to query the gateway.

Should the auto-primaries (supermasters) table on the secondaries have private or public IP addresses if using NAT - or does that not matter either way?

😕

[bab5470]

Does anyone have any advise on how I can troubleshoot this? :( I want to take my PowerDNS server live but would really like this working before hand.

[Habbie]

10.60.10.1 is our gateway - but I am not sure why it would be trying to query the gateway.

Because that's the IP it received the NOTIFY from. The supermasters table is not involved in the process yet, as your secondary fails to query SOA/NS from the IP it got a NOTIFY from.

[bab5470]

@Habbie The two PowerDNS servers are in the same network segment though - so the gateway shouldn't even get used. PowerDNS1 should speak directly to PowerDNS2 without any intermediaries.

The only way the gateway should get used is if they were in different network segments and they're not.

I appreciate the extra insights but I'm lost as to why we're seeing this behavior. Is there any way to debug this further?

[Habbie]

Perhaps use tcpdump on the primary to see how and where it is sending NOTIFY.

[bab5470]

For posterity/to help others in the future I had to do the following:

1. I had NAT incorrectly configured on my firewall/gateway. Instead of NATing the public IPs of our powerDNS servers to the private IPs of our powerDNS server we had them NAT'd to the gateway's default IP. Which caused NOTIFYs to appear to come from the gateway instead of the other server.
2. In the supermasters database table we had the internal IP of our primary powerdns server. I needed to change that to the external IP of the primary powerDNS server
3. On the primary powerDNS server I had the internal IP of our secondary powerDNS server in the allow-axfr-ips. I needed the external IP listed there. I ended up leaving the internal IP as well although I am not sure if that would actually ever get used.