Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

swap issues while starting machines on copr-hv-02 x86_64 #3490

Closed
praiskup opened this issue Oct 23, 2024 · 2 comments
Closed

swap issues while starting machines on copr-hv-02 x86_64 #3490

praiskup opened this issue Oct 23, 2024 · 2 comments
Assignees
@nikromen

Comments

@praiskup
Copy link
Member 


TASK [print out the enable-swap failure logs] **********************************
Wednesday 23 October 2024  04:56:11 +0000 (0:00:00.616)       0:00:47.034 ***** 
fatal: [2620:52:3:1:dead:beef:cafe:c148]: FAILED! => {
    "failed_when_result": true,
    "journalctl_output.stdout_lines": [
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 systemd[1]: Started enable-swap.service - Enable swap on /dev/nvme0n1.",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + set -e",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + part_suffix=p",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + swap_device=",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + test -f /config/resalloc-vars.sh",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + generic_mount",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + for vol in /dev/vdb /dev/vdc /dev/vdd /dev/vda",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1435]: + mount",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1436]: + grep /dev/vdb",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1438]: ++ blockdev --getsize64 /dev/vdb",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + size=168000000000",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + test 168000000000 -le 150000000000",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + swap_device=/dev/vdb",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + part_suffix=",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + break",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + test -n /dev/vdb",
        "Oct 23 04:55:43 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + systemctl unmask tmp.mount",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + systemctl start tmp.mount",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1481]: + echo 'n",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1481]: p",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1481]: +16G",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1481]: n",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1481]: p",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1481]: 2",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1481]: w",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1481]: '",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: + fdisk /dev/vdb",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Welcome to fdisk (util-linux 2.39.3).",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Changes will remain in memory only, until you decide to write them.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Be careful before using the write command.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Device does not contain a recognized partition table.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Created a new DOS (MBR) disklabel with disk identifier 0xe937fb49.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Command (m for help): Partition type",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]:    p   primary (0 primary, 0 extended, 4 free)",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]:    e   extended (container for logical partitions)",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Select (default p): Partition number (1-4, default 1): First sector (2048-328124999, default 2048): Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-328124999, default 328124999):",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Created a new partition 1 of type 'Linux' and of size 16 GiB.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Command (m for help): Partition type",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]:    p   primary (1 primary, 0 extended, 3 free)",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]:    e   extended (container for logical partitions)",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Select (default p): Partition number (2-4, default 2): First sector (33556480-328124999, default 33556480): Last sector, +/-sectors or +/-size{K,M,G,T,P} (33556480-328124999, default 328124999):",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Created a new partition 2 of type 'Linux' and of size 140.5 GiB.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Command (m for help): The partition table has been altered.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Calling ioctl() to re-read partition table.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1482]: Syncing disks.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1433]: + mkfs.ext4 /dev/vdb1",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1486]: mke2fs 1.47.0 (5-Feb-2023)",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 enable-swap.sh[1486]: The file /dev/vdb1 does not exist and no size was specified.",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 systemd[1]: enable-swap.service: Main process exited, code=exited, status=1/FAILURE",
        "Oct 23 04:55:44 copr-hv-x86-64-02-prod-10308972-20241023-045500 systemd[1]: enable-swap.service: Failed with result 'exit-code'."
    ]
}

PLAY RECAP *********************************************************************
2620:52:3:1:dead:beef:cafe:c148 : ok=8    changed=5    unreachable=0    failed=1    skipped=12   rescued=0    ignored=0
@nikromen nikromen self-assigned this Oct 29, 2024
@nikromen
Copy link
Member

can't repro - mkswap on hypervisors currently works

we enable swap - mainly thanks to IBM - generically enough choosing from pools of /dev/vdb /dev/vdc /dev/vdd /dev/vda and then testing whether swap device and partition exists in endless loop - once partition exists we try to do mkswap. So I think partition was created and then something removed/destroyed forcefully the partition before mkswap. Perhaps some ewird glitch.

This shouldn't happen (or at least very very rarely :D), reopen if happens again

@praiskup
Copy link
Member Author

can't repro - mkswap on hypervisors currently works

I can confirm.

This shouldn't happen (or at least very very rarely :D)

Yeah, at that point when I reported it was 100% reproducible (no VM was started in copr_hv_x86_64_02_prod). Weird.

Anyway, it works now - good - and thanks for checking!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nikromen nikromen

Labels
None yet
Projects
CPT Kanban
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@praiskup @nikromen