VM instances get VR's IP as DNS server but cannot resolve through VR

[fnavidan]

Hi Everyone,

I'm on CS v4.18.2.1. All DNS servers on the zone including external and internal DNS servers are set to 8.8.8.8 and 8.8.4.4. However, every time an instance is booted, it gets the VR's Ip as DNS server in addition to above DNS resolvers but the VR doesn't act as a DNS resolver. As a result, the VM fails to resolve through VR and tries the working ones (e.g. 8.8.8.8). This causes a huge delay in resolving hosts as it tries the VR first.

How do I either prevent VMs to obtain VR's IP as DNS server or make the VR to act as DNS resolver?

Any idea is highly appreciated.

[weizhouapache]

@fnavidan
there is a global/zone setting use.external.dns (which is false by default)

[fnavidan]

This setting is already set to true for the zone and I also set it to true globally but even restarting the management server and the network didn't help. VMs are still obtaining VR's IP as their primary name server. I guess we should go for a solution where the VR acts as a DNS resolver / forwarder. I'm not sure why VR isn't acting as DNS server while DNS service is enabled in the network offering.

[mvds-vdsc]

Hi, I have a similar issue but after setting this parameter and restarting the VPC, servers still receive the VR's IP address. Should we restart the VR or how does this change kick in? I tried to renew the dhcp address but that doesn't do the trick

[fnavidan]

Upon further investigation inside the VR, I found that the dnsmasq service does not listen on port 53 which is causing both DNS service not to function and cloud-init inside VMs to fail get VM's password from VR.

By further looking into /etc/dnsmasq.conf file inside the VR, I have figured out that problem is caused by two lines below:

#interface=eth0 # Disables binding dnsmasq to the guest NIC
port=0 # Disables listening on port 53

Uncommenting first line and commenting the second one followed by restarting dnsmasq temporarily fixed the issue, making the VR a DNS resolver and accessible by cloud-init as a data source for passwords.

I confirm that the network offering that the guest network is built with, has all services including DHCP, DNS, and UserData enabled and mapped to VirtualRouter.

So, I'm not sure why CloudStack is writing these two lines in dnsmasq config file while DNS is enabled for the network offering. Also I have tried upgrading the VR version to v4.18.1 (latest version for 4.18.x) but still get the same result.

Any idea would be appreciated.

[weizhouapache]

Upon further investigation inside the VR, I found that the dnsmasq service does not listen on port 53 which is causing both DNS service not to function and cloud-init inside VMs to fail get VM's password from VR.

By further looking into /etc/dnsmasq.conf file inside the VR, I have figured out that problem is caused by two lines below:

#interface=eth0 # Disables binding dnsmasq to the guest NIC port=0 # Disables listening on port 53

Uncommenting first line and commenting the second one followed by restarting dnsmasq temporarily fixed the issue, making the VR a DNS resolver and accessible by cloud-init as a data source for passwords.

I confirm that the network offering that the guest network is built with, has all services including DHCP, DNS, and UserData enabled and mapped to VirtualRouter.

So, I'm not sure why CloudStack is writing these two lines in dnsmasq config file while DNS is enabled for the network offering. Also I have tried upgrading the VR version to v4.18.1 (latest version for 4.18.x) but still get the same result.

Any idea would be appreciated.

@fnavidan
can you share /etc/dnsmasq.d/cloud.conf inside the VR ?

[fnavidan]

There's only a single line in /etc/dnsmasq.d/cloud.conf

root@r-341-VM:# cat /etc/dnsmasq.d/cloud.conf
listen-address=127.0.0.1
root@r-341-VM:#

[weizhouapache]

There's only a single line in /etc/dnsmasq.d/cloud.conf

root@r-341-VM:# cat /etc/dnsmasq.d/cloud.conf listen-address=127.0.0.1 root@r-341-VM:#

this is wrong
do you use a build-in network offering or new network offering ?

[fnavidan]

I'm running on a custom shared network offering. Both UI and DB show DNS, DHCP, and UserData enabled and mapped to VirtualRouter. I't was previously working. I'm not sure when it has been started to break.

[weizhouapache]

I'm running on a custom shared network offering. Both UI and DB show DNS, DHCP, and UserData enabled and mapped to VirtualRouter. I't was previously working. I'm not sure when it has been started to break.

@fnavidan
can you share more details of the network offering ?

stopping/starting VR might help, as well as restarting network with cleanup

[fnavidan]

The network offering is a shared guest network one. I've already destroyed VR and have restarted the network with cleanup. Below is the network offering service map in DB: select service, provider from ntwk_offering_service_map where network_offering_id = ( select id from network_offerings where uuid = 'bbf17083-aee6-4904-9e69-e9362d3dc475' ); +----------+---------------+ | service | provider | +----------+---------------+ | Dhcp | VirtualRouter | | Dns | VirtualRouter | | Firewall | VirtualRouter | | Lb | VirtualRouter | | UserData | VirtualRouter | +----------+---------------+ 5 rows in set (0.00 sec)

…

On Mon, Oct 6, 2025 at 8:30 AM Wei Zhou ***@***.***> wrote: I'm running on a custom shared network offering. Both UI and DB show DNS, DHCP, and UserData enabled and mapped to VirtualRouter. I't was previously working. I'm not sure when it has been started to break. @fnavidan <https://github.com/fnavidan> can you share more details of the network offering ? stopping/starting VR might help, as well as restarting network with cleanup — Reply to this email directly, view it on GitHub <#11749 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKIF7YXQ7K7L77HIX4I2OKL3WID7XAVCNFSM6AAAAACHYVEXL2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTINRQGEZTKMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[weizhouapache]

The network offering is a shared guest network one. I've already destroyed VR and have restarted the network with cleanup. Below is the network offering service map in DB: select service, provider from ntwk_offering_service_map where network_offering_id = ( select id from network_offerings where uuid = 'bbf17083-aee6-4904-9e69-e9362d3dc475' ); +----------+---------------+ | service | provider | +----------+---------------+ | Dhcp | VirtualRouter | | Dns | VirtualRouter | | Firewall | VirtualRouter | | Lb | VirtualRouter | | UserData | VirtualRouter | +----------+---------------+ 5 rows in set (0.00 sec)
…
On Mon, Oct 6, 2025 at 8:30 AM Wei Zhou @.> wrote: I'm running on a custom shared network offering. Both UI and DB show DNS, DHCP, and UserData enabled and mapped to VirtualRouter. I't was previously working. I'm not sure when it has been started to break. @fnavidan https://github.com/fnavidan can you share more details of the network offering ? stopping/starting VR might help, as well as restarting network with cleanup — Reply to this email directly, view it on GitHub <#11749 (reply in thread)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKIF7YXQ7K7L77HIX4I2OKL3WID7XAVCNFSM6AAAAACHYVEXL2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTINRQGEZTKMQ . You are receiving this because you were mentioned.Message ID: @.>

@fnavidan
I cannot reproduce the issue in 4.20

network offering

mysql> select * from ntwk_offering_service_map;
...
| 131 |                  31 | Lb             | VirtualRouter         | 2025-10-07 06:44:01 |
| 132 |                  31 | UserData       | VirtualRouter         | 2025-10-07 06:44:01 |
| 133 |                  31 | Dns            | VirtualRouter         | 2025-10-07 06:44:01 |
| 134 |                  31 | Dhcp           | VirtualRouter         | 2025-10-07 06:44:01 |
| 135 |                  31 | Firewall       | VirtualRouter         | 2025-10-07 06:44:01 |

however, network only has 3 services

mysql> select * from ntwk_service_map;
...
| 107 |        220 | Dhcp           | VirtualRouter | 2025-10-07 06:44:32 |
| 108 |        220 | UserData       | VirtualRouter | 2025-10-07 06:44:32 |
| 109 |        220 | Dns            | VirtualRouter | 2025-10-07 06:44:32 |

VR 1

root@r-47-VM:~# cat /etc/dnsmasq.d/cloud.conf 
bind-interfaces
listen-address=127.0.0.1,172.16.101.10
dhcp-range=set:interface-eth0-0,172.16.101.10,static
dhcp-option=tag:interface-eth0-0,15,cs1cloud.internal
dhcp-option=tag:interface-eth0-0,6,172.16.101.10,10.0.32.1,8.8.8.8
dhcp-option=tag:interface-eth0-0,3,172.16.101.1
dhcp-option=eth0,26,1500
dhcp-option=tag:interface-eth0-0,1,255.255.255.0
root@r-47-VM:~# 

VR 2 (after restarting network with cleanup)

root@r-48-VM:~# cat /etc/dnsmasq.d/cloud.conf 
bind-interfaces
listen-address=127.0.0.1,172.16.101.10
dhcp-range=set:interface-eth0-0,172.16.101.10,static
dhcp-option=tag:interface-eth0-0,15,cs1cloud.internal
dhcp-option=tag:interface-eth0-0,6,172.16.101.10,10.0.32.1,8.8.8.8
dhcp-option=tag:interface-eth0-0,3,172.16.101.1
dhcp-option=eth0,26,1500
dhcp-option=tag:interface-eth0-0,1,255.255.255.0
root@r-48-VM:~# 

[fnavidan]

Hi, I'm on v4.18.2.1 Regards.

…

On Tue, Oct 7, 2025 at 8:53 AM Wei Zhou ***@***.***> wrote: The network offering is a shared guest network one. I've already destroyed VR and have restarted the network with cleanup. Below is the network offering service map in DB: select service, provider from ntwk_offering_service_map where network_offering_id = ( select id from network_offerings where uuid = 'bbf17083-aee6-4904-9e69-e9362d3dc475' ); +----------+---------------+ | service | provider | +----------+---------------+ | Dhcp | VirtualRouter | | Dns | VirtualRouter | | Firewall | VirtualRouter | | Lb | VirtualRouter | | UserData | VirtualRouter | +----------+---------------+ 5 rows in set (0.00 sec) … <#m_-3845268203206475123_> On Mon, Oct 6, 2025 at 8:30 AM Wei Zhou *@*.*> wrote: I'm running on a custom shared network offering. Both UI and DB show DNS, DHCP, and UserData enabled and mapped to VirtualRouter. I't was previously working. I'm not sure when it has been started to break. @fnavidan <https://github.com/fnavidan> https://github.com/fnavidan <https://github.com/fnavidan> can you share more details of the network offering ? stopping/starting VR might help, as well as restarting network with cleanup — Reply to this email directly, view it on GitHub <#11749 (reply in thread) <#11749 (reply in thread)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKIF7YXQ7K7L77HIX4I2OKL3WID7XAVCNFSM6AAAAACHYVEXL2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTINRQGEZTKMQ <https://github.com/notifications/unsubscribe-auth/AKIF7YXQ7K7L77HIX4I2OKL3WID7XAVCNFSM6AAAAACHYVEXL2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTINRQGEZTKMQ> . You are receiving this because you were mentioned.Message ID: @.*> @fnavidan <https://github.com/fnavidan> I cannot reproduce the issue in 4.20 network offering mysql> select * from ntwk_offering_service_map; ... | 131 | 31 | Lb | VirtualRouter | 2025-10-07 06:44:01 | | 132 | 31 | UserData | VirtualRouter | 2025-10-07 06:44:01 | | 133 | 31 | Dns | VirtualRouter | 2025-10-07 06:44:01 | | 134 | 31 | Dhcp | VirtualRouter | 2025-10-07 06:44:01 | | 135 | 31 | Firewall | VirtualRouter | 2025-10-07 06:44:01 | however, network only has 3 services mysql> select * from ntwk_service_map; ... | 107 | 220 | Dhcp | VirtualRouter | 2025-10-07 06:44:32 | | 108 | 220 | UserData | VirtualRouter | 2025-10-07 06:44:32 | | 109 | 220 | Dns | VirtualRouter | 2025-10-07 06:44:32 | VR 1 ***@***.***:~# cat /etc/dnsmasq.d/cloud.conf bind-interfaces listen-address=127.0.0.1,172.16.101.10 dhcp-range=set:interface-eth0-0,172.16.101.10,static dhcp-option=tag:interface-eth0-0,15,cs1cloud.internal dhcp-option=tag:interface-eth0-0,6,172.16.101.10,10.0.32.1,8.8.8.8 dhcp-option=tag:interface-eth0-0,3,172.16.101.1 dhcp-option=eth0,26,1500 dhcp-option=tag:interface-eth0-0,1,255.255.255.0 ***@***.***:~# VR 2 (after restarting network with cleanup) ***@***.***:~# cat /etc/dnsmasq.d/cloud.conf bind-interfaces listen-address=127.0.0.1,172.16.101.10 dhcp-range=set:interface-eth0-0,172.16.101.10,static dhcp-option=tag:interface-eth0-0,15,cs1cloud.internal dhcp-option=tag:interface-eth0-0,6,172.16.101.10,10.0.32.1,8.8.8.8 dhcp-option=tag:interface-eth0-0,3,172.16.101.1 dhcp-option=eth0,26,1500 dhcp-option=tag:interface-eth0-0,1,255.255.255.0 ***@***.***:~# — Reply to this email directly, view it on GitHub <#11749 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKIF7YSCS4H7QHEJPJKQIAT3WNPN5AVCNFSM6AAAAACHYVEXL2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTINRRGE4DMMI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[weizhouapache]

@weizhouapache Yes, using an external dns server inside VR and VMs works fine. The issue is that when VR is deployed it sets port=0 in /etc/dnsmasq.conf which causes dnsmasq not to listen on port 53. Manually uncommenting this line followed by dnsmasq restart makes it working fine. This issue matters to us because on latest templates we have used cloud-init which sticks on boot because it cannot find cloudstack data source. Logs on that VM shows it is looking for data-server.cloud.internal for password and user data.

@fnavidan
I am able to reproduce the issue that data-server is not resolvable when use.external.dns is set to true

[weizhouapache]

@fnavidan
as a workaround, can you change use.external.dns to false, and restart network with cleanup ?

[weizhouapache]

btw, It still worked in my testing with ubuntu 24.04
when data-server is not found (after 6 seconds), cloud-init got the VR IP through dhcp leases.

2025-10-28 09:57:49,469 - sources[DEBUG]: Seeing if we can get any data from <class 'cloudinit.sources.DataSourceCloudStack.DataSourceCloudStack'>
2025-10-28 09:57:55,614 - DataSourceCloudStack.py[DEBUG]: DNS Entry data-server not found
2025-10-28 09:57:55,615 - util.py[DEBUG]: Reading from /run/systemd/netif/leases/2 (quiet=False)
2025-10-28 09:57:55,615 - util.py[DEBUG]: Read 328 bytes from /run/systemd/netif/leases/2
2025-10-28 09:57:55,616 - DataSourceCloudStack.py[DEBUG]: Found SERVER_ADDRESS '10.100.102.10' via networkd_leases
2025-10-28 09:57:55,616 - sources[DEBUG]: Update datasource metadata and network config due to events: boot-new-instance
2025-10-28 09:57:55,617 - sources[DEBUG]: Detected platform: DataSourceCloudStack. Checking for active instance data
2025-10-28 09:57:55,619 - url_helper.py[DEBUG]: [0/1] open 'http://10.100.102.10/latest/meta-data/instance-id' with {'url': 'http://10.100.102.10/latest/meta-data/instance-id', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 50.0, 'headers': {'User-Agent': 'Cloud-Init/24.1.3-0ubuntu3.3'}} configuration

@fnavidan
which guest OS do you use ?

[fnavidan]

After upgrading to 4.20.2.0, I was able to get around this problem by setting use.external.dns back to false followed by destroying VR and restarting network. It now works fine.!

Thank you for all your helpful info.