☂️ Migrate VM resource implementation to Plugin Framework

[bpg]

This is an "umbrella" ticket (call it an epic, or whatever) to track the migration of the proxmox_virtual_environment_vm resource to the Plugin Framework.

Motivation

Check out the benefits of Plugin Framework from HashiCorp. There are quite a lot.

Specific to this provider:

• The Plugin Framework gives better control over Optional and Computed fields and their defaults, which is extremely useful in handling the clone operation. During cloning, we inject lots of attributes from a template that are not present in the clone while also overwriting some of the template parameters from the clone.

  Related:

  • Adding disk causes VM to be re-created #103
  • Creating VM from Clone Does Not Manage New Disks #247
  • Disk discard parameter doesn't work for cloned VM #360
  • Disable default values? #566
  • Multiple cdroms #718
  • Disk resize not properly applied in clone #781
  • boot_order is ignored when clonning a VM #850
  • iothread and ssd emulation not enabled during resource creation and modification #873
  • Initialization settings cannot be cleared when cloning. #1106
  • fix(vm): do not overwrite cpu attributes with defaults when cloning #1219
  • vga changed on second run with unchanged source #1224
  • Idempotence issues in "proxmox_virtual_environment_vm" after cloning from another VM #1299
  • Align Defaults with Proxmox Defaults #1304
  • and others

• Ability to manage data transformations via custom data types (various conventions for lists/sets in the Proxmox API).

• Better support for collection data types (lists, and specifically sets, to handle unordered collections of items).

Challenges

• There's a vast amount of functionality to port, and some (or most?) of it needs to be re-implemented to leverage the benefits of the Provider Framework data model/custom types.

• Severe lack of the provider's code test coverage. The changes/re-implementation will introduce new bugs that will be hard to catch.

Compatibility

Inevitably, some attribute formats and semantics will be incompatible with the old version. Also, some attributes will be renamed/removed. Known incompatibilities are marked with ❗ below.

Implementation

The proxmox_virtual_environment_vm2 resource contains the new implementation.

Status of features/attributes:

• Individual attributes
  • acpi
  • boot_order
  • description
  • id ❗
  • hook_script_file_id
  • machine
  • migrate
  • name
  • node_name
  • on_boot
  • pool_id
  • protection
  • reboot
  • smbios
  • started
  • stop_on_destroy
  • tablet_device
  • tags
  • template
• Blocks
  • agent
  • audio_device
  • bios
  • cdrom ❗
  • clone
    • simple clone
    • migrate between nodes
  • cpu
  • disk
  • efi_disk
  • hostpci
  • initialization
  • keyboard_layout
  • kvm_arguments
  • memory
  • network_device
  • numa
  • operating_system
  • serial_device
  • scsi_hardware
  • startup
  • timeouts ❗
    • CRUD operations
    • specific API calls (if supported/needed)
  • tpm_state
  • usb
  • vga
  • watchdog

[sergelogvinov]

Hi, i think that we need to add reboot flag. which allow/skip rebooting the VM if it needed after changes.
(Currently, I usually comment out the reboot block in the code...)

Thanks.

[maksimsamt]

@bpg ,
What is the approximate forecast on this new implementation?
Unfortunately, the current implementation of proxmox_virtual_environment_vm cannot be used for clone operations, especially in prod environments. As you mentioned in your task description, there are some important issues with disk provisioning/changing/adding etc from clone vm or template. And basically in almost all scenarios (disk modify or adding new one) the issue is that the current main disk is destroyed and a new one is created.

[bpg]

Hi @maksimsamt!

It's difficult to estimate, as my availability for this project is quite unpredictable. I work on it in my free time, but that's essentially it.

I'm still establishing the best approach to re-implementing the functionality, common reusable patterns, test harness, etc., and this initial stage is almost done. After that, I'll formalize the design decisions in some contribution guidelines so anyone willing to help (and speed thing up) can get on board more easily.

But for now, as you can see from the status of the checkboxes in the list above, I've only scratched the surface in terms of functionality coverage. The VM2 resource is really far away from being functionally complete.

[svengreb]

It's important to take you time on such large topics.
"Forecasts" or "ETA" are terms that should be avoided in (non-commercial) open-source as it only creates (mental) pressure and slows down development processes.

Are all of list entries above possible to be implemented on their own or are there some dependencies between each other? I'd like to help when my (also limited) free-time allows it so when it is possible to simply pick one of those it's easy to distribute contributions.

[bpg]

Are all of list entries above possible to be implemented on their own or are there some dependencies between each other? I'd like to help when my (also limited) free-time allows it so when it is possible to simply pick one of those it's easy to distribute contributions.

@svengreb Thanks for the offer, much appreciated!!
Those blocks / attributes are pretty much independent, but I still need to iron out a few general things, like code structure, conventions, and similar. I'll ping you when it's ready.