libvert issue

[addisflava]

"msg": "\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.fd\n\n with module.master_nodes.libvirt_domain.service-vm[0],\n on .terraform/modules/master_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n 47: resource "libvirt_domain" "service-vm" {\n\n\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.fd\n\n with module.worker_nodes.libvirt_domain.service-vm[0],\n on .terraform/modules/worker_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n 47: resource "libvirt_domain" "service-vm" {",
"rc": 1,
"stderr": "\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.fd\n\n with module.master_nodes.libvirt_domain.service-vm[0],\n on .terraform/modules/master_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n 47: resource "libvirt_domain" "service-vm" {\n\n\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.fd\n\n with module.worker_nodes.libvirt_domain.service-vm[0],\n on .terraform/modules/worker_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n 47: resource "libvirt_domain" "service-vm" {\n\n",
"stderr_lines": [
"",
"Error: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.fd",
"",
" with module.master_nodes.libvirt_domain.service-vm[0],",
" on .terraform/modules/master_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":",
" 47: resource "libvirt_domain" "service-vm" {",
"",
"",
"Error: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.fd",
"",
" with module.worker_nodes.libvirt_domain.service-vm[0],",
" on .terraform/modules/worker_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":",
" 47: resource "libvirt_domain" "service-vm" {",
""
],
"stdout": "module.libvirt_pool.libvirt_pool.vm_pool: Creating...\nmodule.libvirt_network.libvirt_network.vm_network: Creating...\nmodule.libvirt_pool.libvirt_pool.vm_pool: Creation complete after 5s [id=71027022-1700-4007-9dc4-0d2d17c23e2e]\nmodule.libvirt_network.libvirt_network.vm_network: Creation complete after 5s [id=a1f12fab-2aca-4b4b-8fc2-861ec807a64c]\nmodule.master_nodes.data.template_file.user_data[0]: Reading...\nmodule.worker_nodes.data.template_file.user_data[0]: Reading...\nmodule.master_nodes.libvirt_volume.os_image[0]: Creating...\nmodule.worker_nodes.libvirt_volume.os_image[0]: Creating...\nmodule.worker_nodes.data.template_file.user_data[0]: Read complete after 0s [id=34f0436f48849fa59eb76b98d30720376f110444df95237e3adcc96c7f0d8d52]\nmodule.master_nodes.data.template_file.user_data[0]: Read complete after 0s [id=d12f168eb9e443c541d4177bb394b0294ce1dca493cc2131735408e6ccfa06d2]\nmodule.worker_nodes.libvirt_cloudinit_disk.commoninit[0]: Creating...\nmodule.master_nodes.libvirt_cloudinit_disk.commoninit[0]: Creating...\nmodule.master_nodes.libvirt_volume.os_image[0]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-os_image]\nmodule.master_nodes.libvirt_volume.os_disk[0]: Creating...\nmodule.worker_nodes.libvirt_volume.os_image[0]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0-os_image]\nmodule.worker_nodes.libvirt_volume.os_disk[0]: Creating...\nmodule.worker_nodes.libvirt_cloudinit_disk.commoninit[0]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0-commoninit.iso;77011195-611e-4f46-bc26-54ade61e309a]\nmodule.master_nodes.libvirt_cloudinit_disk.commoninit[0]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-commoninit.iso;2de59958-e79e-41bd-acac-5899cf09e398]\nmodule.master_nodes.libvirt_volume.os_disk[0]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0]\nmodule.master_nodes.libvirt_domain.service-vm[0]: Creating...\nmodule.worker_nodes.libvirt_volume.os_disk[0]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0]\nmodule.worker_nodes.libvirt_domain.service-vm[0]: Creating...\n",
"stdout_lines": [
"module.libvirt_pool.libvirt_pool.vm_pool: Creating...",
"module.libvirt_network.libvirt_network.vm_network: Creating...",
"module.libvirt_pool.libvirt_pool.vm_pool: Creation complete after 5s [id=71027022-1700-4007-9dc4-0d2d17c23e2e]",
"module.libvirt_network.libvirt_network.vm_network: Creation complete after 5s [id=a1f12fab-2aca-4b4b-8fc2-861ec807a64c]",
"module.master_nodes.data.template_file.user_data[0]: Reading...",
"module.worker_nodes.data.template_file.user_data[0]: Reading...",
"module.master_nodes.libvirt_volume.os_image[0]: Creating...",
"module.worker_nodes.libvirt_volume.os_image[0]: Creating...",
"module.worker_nodes.data.template_file.user_data[0]: Read complete after 0s [id=34f0436f48849fa59eb76b98d30720376f110444df95237e3adcc96c7f0d8d52]",
"module.master_nodes.data.template_file.user_data[0]: Read complete after 0s [id=d12f168eb9e443c541d4177bb394b0294ce1dca493cc2131735408e6ccfa06d2]",
"module.worker_nodes.libvirt_cloudinit_disk.commoninit[0]: Creating...",
"module.master_nodes.libvirt_cloudinit_disk.commoninit[0]: Creating...",
"module.master_nodes.libvirt_volume.os_image[0]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-os_image]",
"module.master_nodes.libvirt_volume.os_disk[0]: Creating...",
"module.worker_nodes.libvirt_volume.os_image[0]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0-os_image]",
"module.worker_nodes.libvirt_volume.os_disk[0]: Creating...",
"module.worker_nodes.libvirt_cloudinit_disk.commoninit[0]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0-commoninit.iso;77011195-611e-4f46-bc26-54ade61e309a]",
"module.master_nodes.libvirt_cloudinit_disk.commoninit[0]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-commoninit.iso;2de59958-e79e-41bd-acac-5899cf09e398]",
"module.master_nodes.libvirt_volume.os_disk[0]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0]",
"module.master_nodes.libvirt_domain.service-vm[0]: Creating...",
"module.worker_nodes.libvirt_volume.os_disk[0]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0]",
"module.worker_nodes.libvirt_domain.service-vm[0]: Creating..."
]
}

[kubealex]

Hi @addisflava, thank you for reporting this, are you trying to install it on Ubuntu?

[kubealex]

I updated the provisioner to use the latest version of the module.
The previous one had an issue of hardcoded UEFI module, preventing ubuntu users to create VMs due to mismatching paths.
Let me know if this solves your issue or please reopen it.

[romyt]

Hi @kubealex I have the same issue with Ubuntu 22.04. Exact same output as above. I noticed that on Ubuntu the file is located under /usr/share/OVMF/OVMF_CODE.fd but the provisioner is looking for it /usr/share/edk2/ovmf/OVMF_CODE.fd
I copied the file under /usr/share/edk2/ovmf/OVMF_CODE.fd but it doesn't solve the problem.

FAILED! => {"changed": false, "cmd": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmpjt848myj.tfplan", "msg": "\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.fd\n\n with module.loadbalancer[0].libvirt_domain.service-vm[0],\n on .terraform/modules/loadbalancer/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n 47: resource "libvirt_domain" "service-vm" {\n\n\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/edk2/ovmf/OVMF_CODE.fd\n\n with module.master_nodes.libvirt_domain.service-vm[0],\n on .terraform/modules/master_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n

[kubealex]

hi @romyt can you please double check with the latest changes? I added a variable to globally set the path of the Firmware based on the host OS, I tested it now and it works, can you please confirm on your end?
Thank you!

[romyt]

hi @romyt can you please double check with the latest changes? I added a variable to globally set the path of the Firmware based on the host OS, I tested it now and it works, can you please confirm on your end? Thank you!

Hi @kubealex,
thank you for taking the time to looking into this. I cloned a fresh copy and tried but I still have the same issue.
I'm trying to deploy it against an ubuntu 22.04 server in my local network from a Mac client.

Full logs bellow:

TASK [Ensure control plane VMs are in place] **********************************************************************************
fatal: [192.168.1.158]: FAILED! => {"changed": false, "cmd": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmp43fz2mzf.tfplan", "msg": "\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/OVMF/OVMF_CODE.fd\n\n with module.master_nodes.libvirt_domain.service-vm[0],\n on .terraform/modules/master_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n 47: resource "libvirt_domain" "service-vm" {\n\n\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/OVMF/OVMF_CODE.fd\n\n with module.worker_nodes.libvirt_domain.service-vm[0],\n on .terraform/modules/worker_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n 47: resource "libvirt_domain" "service-vm" {", "rc": 1, "stderr": "\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/OVMF/OVMF_CODE.fd\n\n with module.master_nodes.libvirt_domain.service-vm[0],\n on .terraform/modules/master_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n 47: resource "libvirt_domain" "service-vm" {\n\n\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/OVMF/OVMF_CODE.fd\n\n with module.worker_nodes.libvirt_domain.service-vm[0],\n on .terraform/modules/worker_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":\n 47: resource "libvirt_domain" "service-vm" {\n\n", "stderr_lines": ["", "Error: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/OVMF/OVMF_CODE.fd", "", " with module.master_nodes.libvirt_domain.service-vm[0],", " on .terraform/modules/master_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":", " 47: resource "libvirt_domain" "service-vm" {", "", "", "Error: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/OVMF/OVMF_CODE.fd", "", " with module.worker_nodes.libvirt_domain.service-vm[0],", " on .terraform/modules/worker_nodes/modules/terraform-libvirt-instance/main.tf line 47, in resource "libvirt_domain" "service-vm":", " 47: resource "libvirt_domain" "service-vm" {", ""], "stdout": "module.libvirt_network.libvirt_network.vm_network: Creating...\nmodule.libvirt_network.libvirt_network.vm_network: Creation complete after 5s [id=eb4c81b7-7e6d-45e6-bf86-1259e6eee87e]\nmodule.master_nodes.data.template_file.user_data[0]: Reading...\nmodule.worker_nodes.data.template_file.user_data[0]: Reading...\nmodule.master_nodes.libvirt_volume.os_image[0]: Creating...\nmodule.worker_nodes.libvirt_volume.os_image[0]: Creating...\nmodule.worker_nodes.data.template_file.user_data[0]: Read complete after 0s [id=9281018d6f7ee61970ef9e20c4c73b7ebec8f9ebc4ee9fca23e6eb1ecca1d7e2]\nmodule.master_nodes.data.template_file.user_data[0]: Read complete after 0s [id=258b29d03ebdcad9d18afebd51d3dfb0e9f7ddc2ca368a8419c5a402d5b6a78b]\nmodule.master_nodes.libvirt_cloudinit_disk.commoninit[0]: Creating...\nmodule.worker_nodes.libvirt_cloudinit_disk.commoninit[0]: Creating...\nmodule.master_nodes.libvirt_volume.os_image[0]: Creation complete after 3s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-os_image]\nmodule.master_nodes.libvirt_volume.os_disk[0]: Creating...\nmodule.worker_nodes.libvirt_volume.os_image[0]: Creation complete after 4s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0-os_image]\nmodule.worker_nodes.libvirt_volume.os_disk[0]: Creating...\nmodule.master_nodes.libvirt_cloudinit_disk.commoninit[0]: Creation complete after 4s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-commoninit.iso;7b704ab1-e56a-414e-ad5c-498d6b49b5d8]\nmodule.worker_nodes.libvirt_cloudinit_disk.commoninit[0]: Creation complete after 4s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0-commoninit.iso;8f6863bc-d853-4be4-bc17-bec49fc74e18]\nmodule.master_nodes.libvirt_volume.os_disk[0]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0]\nmodule.master_nodes.libvirt_domain.service-vm[0]: Creating...\nmodule.worker_nodes.libvirt_volume.os_disk[0]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0]\nmodule.worker_nodes.libvirt_domain.service-vm[0]: Creating...\n", "stdout_lines": ["module.libvirt_network.libvirt_network.vm_network: Creating...", "module.libvirt_network.libvirt_network.vm_network: Creation complete after 5s [id=eb4c81b7-7e6d-45e6-bf86-1259e6eee87e]", "module.master_nodes.data.template_file.user_data[0]: Reading...", "module.worker_nodes.data.template_file.user_data[0]: Reading...", "module.master_nodes.libvirt_volume.os_image[0]: Creating...", "module.worker_nodes.libvirt_volume.os_image[0]: Creating...", "module.worker_nodes.data.template_file.user_data[0]: Read complete after 0s [id=9281018d6f7ee61970ef9e20c4c73b7ebec8f9ebc4ee9fca23e6eb1ecca1d7e2]", "module.master_nodes.data.template_file.user_data[0]: Read complete after 0s [id=258b29d03ebdcad9d18afebd51d3dfb0e9f7ddc2ca368a8419c5a402d5b6a78b]", "module.master_nodes.libvirt_cloudinit_disk.commoninit[0]: Creating...", "module.worker_nodes.libvirt_cloudinit_disk.commoninit[0]: Creating...", "module.master_nodes.libvirt_volume.os_image[0]: Creation complete after 3s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-os_image]", "module.master_nodes.libvirt_volume.os_disk[0]: Creating...", "module.worker_nodes.libvirt_volume.os_image[0]: Creation complete after 4s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0-os_image]", "module.worker_nodes.libvirt_volume.os_disk[0]: Creating...", "module.master_nodes.libvirt_cloudinit_disk.commoninit[0]: Creation complete after 4s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-commoninit.iso;7b704ab1-e56a-414e-ad5c-498d6b49b5d8]", "module.worker_nodes.libvirt_cloudinit_disk.commoninit[0]: Creation complete after 4s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0-commoninit.iso;8f6863bc-d853-4be4-bc17-bec49fc74e18]", "module.master_nodes.libvirt_volume.os_disk[0]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0]", "module.master_nodes.libvirt_domain.service-vm[0]: Creating...", "module.worker_nodes.libvirt_volume.os_disk[0]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-worker-0]", "module.worker_nodes.libvirt_domain.service-vm[0]: Creating..."]}

PLAY RECAP ********************************************************************************************************************
192.168.1.158 : ok=33 changed=3 unreachable=0 failed=1 skipped=16 rescued=0 ignored=0

[kubealex]

Mmmhhh @romyt, that's interesting...when you have time can you show me the content (if any) of the /use/share/OVMF folder? Maybe the name of the file changed 🤯

Thank you!

[romyt]

I read here that libvirt on Ubuntu 22.04 has a permission bug. I wonder if this is not the reason. What version of libvirt do you use for your test?

The content of:
ls -al /usr/share/OVMF/
total 12824
drwxr-xr-x 2 root root 4096 Apr 9 06:14 .
drwxr-xr-x 168 root root 4096 Apr 21 02:58 ..
-rw-r--r-- 1 root root 1966080 Feb 12 20:19 OVMF_CODE.fd
lrwxrwxrwx 1 root root 20 Feb 12 20:19 OVMF_CODE.ms.fd -> OVMF_CODE.secboot.fd
-rw-r--r-- 1 root root 1966080 Feb 12 20:19 OVMF_CODE.secboot.fd
-rw-r--r-- 1 root root 3653632 Feb 12 20:19 OVMF_CODE_4M.fd
lrwxrwxrwx 1 root root 23 Feb 12 20:19 OVMF_CODE_4M.ms.fd -> OVMF_CODE_4M.secboot.fd
-rw-r--r-- 1 root root 3653632 Feb 12 20:19 OVMF_CODE_4M.secboot.fd
lrwxrwxrwx 1 root root 23 Feb 12 20:19 OVMF_CODE_4M.snakeoil.fd -> OVMF_CODE_4M.secboot.fd
-rw-r--r-- 1 root root 131072 Feb 12 20:19 OVMF_VARS.fd
-rw-r--r-- 1 root root 131072 Feb 12 20:19 OVMF_VARS.ms.fd
-rw-r--r-- 1 root root 540672 Feb 12 20:19 OVMF_VARS_4M.fd
-rw-r--r-- 1 root root 540672 Feb 12 20:19 OVMF_VARS_4M.ms.fd
-rw-r--r-- 1 root root 540672 Feb 12 20:19 OVMF_VARS_4M.snakeoil.fd

[kubealex]

thank you @romyt
I just tried again and it completed the setup successfully...that's super weird!

sysadmin@ubuntu-server:~/libvirt-k8s-provisioner$ sudo virsh version
Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: QEMU 8.0.0
Running hypervisor: QEMU 6.2.0

sysadmin@ubuntu-server:~/libvirt-k8s-provisioner$ libvirtd --version
libvirtd (libvirt) 8.0.0

sysadmin@ubuntu-server:~/libvirt-k8s-provisioner$ uname -a
Linux ubuntu-server.rh-lab.labs 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

I'm using a fresh cloud image for the setup:

And this is the outcome of a fresh run:

PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
k8s-test-loadbalancer.k8s.test : ok=22 changed=12 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0
k8s-test-master-0.k8s.test : ok=49 changed=28 unreachable=0 failed=0 skipped=20 rescued=0 ignored=0
k8s-test-master-1.k8s.test : ok=38 changed=24 unreachable=0 failed=0 skipped=20 rescued=0 ignored=0
k8s-test-worker.k8s.test : ok=38 changed=24 unreachable=0 failed=0 skipped=20 rescued=0 ignored=0
localhost : ok=64 changed=29 unreachable=0 failed=0 skipped=37 rescued=0 ignored=0

[romyt]

The only difference is
Linux ubuntu-server 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux.

Let me try to move the code on the server itself and run it from there using root account to see if I still have the same issue.

[romyt]

Same problem running on the server! There is definitely something not right here...

TASK [Ensure cluster VMs are in place] ****************************************************************************************
[DEPRECATION WARNING]: Encryption using the Python crypt module is deprecated. The Python crypt module is deprecated and will
be removed from Python 3.13. Install the passlib library for continued encryption functionality. This feature will be removed
in version 2.17. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/usr/bin/terraform apply -no-color -input=false -auto-approve -lock=true /tmp/tmpqn2cx2zp.tfplan", "msg": "\nError: error creating libvirt domain: operation failed: unable to find any master var store for loader: /usr/share/OVMF/OVMF_CODE.fd\n\n with module.master_nodes.libvirt_domain.service-vm[0],

PLAY RECAP ********************************************************************************************************************
localhost : ok=37 changed=21 unreachable=0 failed=1 skipped=14 rescued=0 ignored=0