upgrade-test: add 2 upgrades test #4806

HuijingHei · 2024-01-29T14:51:17Z

Enhance 2 upgrade tests:

kolainst: Do client-side layering tests on old build, then upgrade
rpm-ostree to new version, then upgrade to another new update.
prow/e2e-upgrades: Start old build with upgraded rpm-ostree,
do layering tests, then upgrade to another new update.

tests/kolainst/upgrades/misc-tests

HuijingHei · 2024-01-30T03:43:32Z

The basic tests include override kernel, kernel args, initramfs-etc, remove package, overlay, and the whole test took almost 30 minutes, seems too long, maybe should skip some reboots?

[2024-01-30T02:45:29.909Z] === RUN   ext.rpm-ostree.upgrades.misc-tests
[2024-01-30T03:15:37.825Z] --- PASS: ext.rpm-ostree.upgrades.misc-tests (1797.84s)

tests/common/libtest.sh

tests/kolainst/upgrades/client-layering

tests/kolainst/upgrades/upgrade-rpmostree

tests/kolainst/upgrades/client-layering

tests/kolainst/upgrades/upgrade-rpmostree

jlebon

This is really great! This kind of testing was sorely needed.

jlebon · 2024-01-30T21:24:25Z

.cci.jenkinsfile

+      make -C tests/kolainst upgrades-install
+      mv rpm-ostree-[0-9]*.rpm rpm-ostree-libs-[0-9]*.rpm /usr/lib/coreos-assembler/tests/kola/rpm-ostree/upgrades/data/      
+    """)
+    kola(cosaDir: "${env.WORKSPACE}", extraArgs: "--qemu-image ./fedora-coreos*.qcow2 ext.rpm-ostree.upgrades.*", parallel: 2)


I was thinking more of having the OS rollback to an earlier version on its first boot, but yeah this is more proper even if more wordy. :) (Kola also has support for downloading the previous image, but as part of the run-upgrade suite; I think we should really fold that back into kola run proper and support external tests.)

(Kola also has support for downloading the previous image, but as part of the run-upgrade suite; I think we should really fold that back into kola run proper and support external tests.)

Good suggestion, then this will make upgrade tests more easier, will change that later.

jlebon · 2024-01-30T21:25:52Z

tests/kolainst/Makefile

@@ -22,3 +22,9 @@ install: all
 localinstall: all
 	rm -rf ../kola
 	make install KOLA_TESTDIR=../kola
+
+upgrades-install: all


Instead of this, we could make it part of the destructive suite, but add a e.g. requiredTag: rpm-ostree-upgrade to it so that it doesn't run in the default testsuite run, and will only run if you pass --tag rpm-ostree-upgrade.

SGTM, thanks!

.cci.jenkinsfile

jlebon · 2024-01-30T21:40:01Z

tests/kolainst/upgrades/upgrade-rpmostree

+##   minMemory: 2048
+##   tags: needs-internet
+##   description: Start old build, upgrade rpm-ostree to new version,
+##     do client-side layering tests, then upgrade to another new update.


Sorry, can you expand on #4784 (comment) ? Isn't fedora-silverblue/issue-tracker#523 covered by your first test where we do an override remove on the old version and then upgrade to the new version and then upgrade again from the new version to another version (with the same cached RPM content)?

Yes, I meant to add there like that, but then I thought if we add such tests in kolainst that can cover more tests instead of just remove, no need to change in e2e-upgrades.

But agree with you that it would be better to add the tests to e2e-upgrades instead of kolainst.

tests/kolainst/upgrades/client-layering

HuijingHei · 2024-01-31T09:48:03Z

/test kola-upgrade

HuijingHei · 2024-01-31T09:51:36Z

Seems + cosa kola run 'ext.rpm-ostree.*' triggers test ext.rpm-ostree.destructive.client-layering-upgrade that has requiredTag: rpm-ostree-upgrade, will do more research.

jlebon · 2024-01-31T15:20:25Z

OK yeah, I think I understand better now. Basically we want to test new rpm-ostree on a system composed by old rpm-ostree. There's a subtle difference from the scenario in fedora-silverblue/issue-tracker#523 though: the override remove was done with the old rpm-ostree first and then upgraded to a compose with the new rpm-ostree (but that's still composed by old rpm-ostree).

I don't think it matters for catching #4746, but does for #4727 since the pkgcache was generated by old rpm-ostree in one case but new rpm-ostree in the other. Do you agree?

I think it does make sense to test both cases, which is what you're doing with the Prow version and the kolainst version of these tests. But man... this stuff is complex. Maybe let's add a link to this PR (or maybe even this comment directly) in the tests?

jlebon

Some comments, but LGTM overall! Thank you so much for working on this.

tests/kolainst/destructive/client-layering-upgrade

Before, even though a test had a required tag, explicitly passing a pattern which matched that test name would still include it. I thought that would be convenient, but it makes the logic less predictable. Since we don't use many required tags today, let's instead opt for being more explicit and always actually require the tag. Motivated by coreos/rpm-ostree#4806 (comment).

jlebon · 2024-01-31T15:51:01Z

Seems + cosa kola run 'ext.rpm-ostree.*' triggers test ext.rpm-ostree.destructive.client-layering-upgrade that has requiredTag: rpm-ostree-upgrade, will do more research.

Should be fixed by coreos/coreos-assembler#3710!

Before, even though a test had a required tag, explicitly passing a pattern which matched that test name would still include it. I thought that would be convenient, but it makes the logic less predictable. Since we don't use many required tags today, let's instead opt for being more explicit and always actually require the tag. Motivated by coreos/rpm-ostree#4806 (comment).

HuijingHei · 2024-02-01T02:37:02Z

I don't think it matters for catching #4746, but does for #4727 since the pkgcache was generated by old rpm-ostree in one case but new rpm-ostree in the other. Do you agree?

Yes, yes, you are totally correct.
For some reason, we build fcos-rawhide with coreos-assembler (using f39 for example) which has old rpm-ostree, but the built image include the new release rpm-ostree, likes upgrade rpm-ostree on old (previous) build. Is this the same as prow test that sync new rpm-ostree to old quay.io/fedora/fedora-coreos:testing-devel (refer to https://github.com/coreos/rpm-ostree/blob/main/ci/prow/Dockerfile.fcos#L10) ?

So we should have tests:

kolainst: Do client-side layering tests on old build, then upgrade
rpm-ostree to new version, then upgrade to another new update.
prow/e2e-upgrades: Start old build with upgraded rpm-ostree,
do layering tests, then upgrade to another new update.

Is my understanding is correct?

- `kolainst`: Do client-side layering tests on old build, then upgrade rpm-ostree to new version, then upgrade to another update. - `prow/e2e-upgrades`: Start old build with upgraded rpm-ostree, add layering tests, then upgrade to another new update.

jlebon · 2024-02-01T18:58:51Z

.cci.jenkinsfile

+      coreos-installer download -p qemu -f qcow2.xz -s testing --decompress
+      mv rpm-ostree-[0-9]*.rpm rpm-ostree-libs-[0-9]*.rpm /usr/lib/coreos-assembler/tests/kola/rpm-ostree/destructive/data/      
+    """)
+    kola(cosaDir: "${env.WORKSPACE}", extraArgs: "--qemu-image ./fedora-coreos*.qcow2 --tag rpm-ostree-upgrade ext.rpm-ostree.destructive.client-layering-upgrade")


BTW, this could now just be

kola(cosaDir: "${env.WORKSPACE}", extraArgs: "--qemu-image ./fedora-coreos*.qcow2 --tag rpm-ostree-upgrade")

since it's the only test with that tag.

Also, we probably want skipUpgrade: true here.

And finally, it doesn't matter much, though re. --qemu-image ....qcow2, that would belong better as a platformArgs argument. (It would matter if we e.g. didn't disable the default upgrade tests.)

jlebon · 2024-02-01T18:59:41Z

tests/kolainst/destructive/client-layering-upgrade

+
+    /tmp/autopkgtest-reboot 2    
+    ;;
+  "2")


We should verify here that the booted ref is kola to make sure e.g. stage finalization didn't fail.

Xerf to Jonathan's comment coreos#4806 (comment)

HuijingHei added the do-not-merge/work-in-progress label Jan 29, 2024

github-advanced-security bot found potential problems Jan 29, 2024

View reviewed changes

tests/kolainst/upgrades/misc-tests Fixed Show fixed Hide fixed

HuijingHei changed the title ~~upgrade-test: do basic tests on old builds before upgrade rpm-ostree to new version, then upgrade to another new update~~ upgrade-test: do basic tests on old builds before upgrade Jan 29, 2024

HuijingHei force-pushed the upgrades branch 2 times, most recently from dc6b0a1 to 97657b0 Compare January 30, 2024 01:24

HuijingHei force-pushed the upgrades branch 2 times, most recently from d9ed6f7 to 07f72de Compare January 30, 2024 10:27

github-advanced-security bot found potential problems Jan 30, 2024

View reviewed changes

openshift-ci bot removed the do-not-merge/work-in-progress label Jan 30, 2024

HuijingHei changed the title ~~upgrade-test: do basic tests on old builds before upgrade~~ upgrade-test: add 2 upgrades test Jan 30, 2024

HuijingHei force-pushed the upgrades branch 2 times, most recently from 885cab9 to 0586cc0 Compare January 30, 2024 10:47

github-advanced-security bot found potential problems Jan 30, 2024

View reviewed changes

tests/kolainst/upgrades/client-layering Fixed Show fixed Hide fixed

tests/kolainst/upgrades/upgrade-rpmostree Fixed Show fixed Hide fixed

HuijingHei force-pushed the upgrades branch 2 times, most recently from e11c68e to fb38b73 Compare January 30, 2024 13:50

HuijingHei mentioned this pull request Jan 30, 2024

e2e-upgrades: add basic tests with new rpm-ostree version #4784

Closed

HuijingHei requested a review from jlebon January 30, 2024 14:26

jlebon reviewed Jan 30, 2024

View reviewed changes

jlebon mentioned this pull request Jan 30, 2024

Rebase to container image failed (with local package installed) #4805

Closed

HuijingHei force-pushed the upgrades branch from fb38b73 to 89505ab Compare January 31, 2024 07:43

HuijingHei added the do-not-merge/work-in-progress label Jan 31, 2024

HuijingHei force-pushed the upgrades branch from 89505ab to d80fb47 Compare January 31, 2024 08:16

jlebon approved these changes Jan 31, 2024

View reviewed changes

tests/kolainst/destructive/client-layering-upgrade Outdated Show resolved Hide resolved

tests/kolainst/destructive/client-layering-upgrade Outdated Show resolved Hide resolved

jlebon mentioned this pull request Jan 31, 2024

kola: really make requiredTag required coreos/coreos-assembler#3710

Merged

openshift-ci bot removed the do-not-merge/work-in-progress label Feb 1, 2024

HuijingHei force-pushed the upgrades branch 4 times, most recently from 08ff36f to 667cf69 Compare February 1, 2024 06:14

Enhance 2 upgrade tests

0521f79

- `kolainst`: Do client-side layering tests on old build, then upgrade rpm-ostree to new version, then upgrade to another update. - `prow/e2e-upgrades`: Start old build with upgraded rpm-ostree, add layering tests, then upgrade to another new update.

HuijingHei force-pushed the upgrades branch from 667cf69 to 0521f79 Compare February 1, 2024 12:39

jmarrero approved these changes Feb 1, 2024

View reviewed changes

jmarrero merged commit aa4caed into coreos:main Feb 1, 2024
17 checks passed

jlebon reviewed Feb 1, 2024

View reviewed changes

HuijingHei deleted the upgrades branch February 2, 2024 02:27

HuijingHei added a commit to HuijingHei/rpm-ostree that referenced this pull request Feb 2, 2024

tests: minor fix for client-layering-upgrade

401b28e

Xerf to Jonathan's comment coreos#4806 (comment)

HuijingHei added a commit to HuijingHei/rpm-ostree that referenced this pull request Feb 2, 2024

tests: minor fix for client-layering-upgrade

b92f95c

Xerf to Jonathan's comment coreos#4806 (comment)

This was referenced Feb 2, 2024

tests: minor fix for client-layering-upgrade #4813

Merged

[RFE] kola should support to start previous build to do external tests coreos/coreos-assembler#3714

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade-test: add 2 upgrades test #4806

upgrade-test: add 2 upgrades test #4806

HuijingHei commented Jan 29, 2024 •

edited

Loading

HuijingHei commented Jan 30, 2024 •

edited

Loading

jlebon left a comment

jlebon Jan 30, 2024

HuijingHei Jan 31, 2024

jlebon Jan 30, 2024 •

edited

Loading

HuijingHei Jan 31, 2024

jlebon Jan 30, 2024

HuijingHei Jan 31, 2024

HuijingHei commented Jan 31, 2024

HuijingHei commented Jan 31, 2024 •

edited

Loading

jlebon commented Jan 31, 2024

jlebon left a comment

jlebon commented Jan 31, 2024

HuijingHei commented Feb 1, 2024 •

edited

Loading

jlebon Feb 1, 2024

jlebon Feb 1, 2024

upgrade-test: add 2 upgrades test #4806

upgrade-test: add 2 upgrades test #4806

Conversation

HuijingHei commented Jan 29, 2024 • edited Loading

HuijingHei commented Jan 30, 2024 • edited Loading

jlebon left a comment

Choose a reason for hiding this comment

jlebon Jan 30, 2024

Choose a reason for hiding this comment

HuijingHei Jan 31, 2024

Choose a reason for hiding this comment

jlebon Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

HuijingHei Jan 31, 2024

Choose a reason for hiding this comment

jlebon Jan 30, 2024

Choose a reason for hiding this comment

HuijingHei Jan 31, 2024

Choose a reason for hiding this comment

HuijingHei commented Jan 31, 2024

HuijingHei commented Jan 31, 2024 • edited Loading

jlebon commented Jan 31, 2024

jlebon left a comment

Choose a reason for hiding this comment

jlebon commented Jan 31, 2024

HuijingHei commented Feb 1, 2024 • edited Loading

jlebon Feb 1, 2024

Choose a reason for hiding this comment

jlebon Feb 1, 2024

Choose a reason for hiding this comment

HuijingHei commented Jan 29, 2024 •

edited

Loading

HuijingHei commented Jan 30, 2024 •

edited

Loading

jlebon Jan 30, 2024 •

edited

Loading

HuijingHei commented Jan 31, 2024 •

edited

Loading

HuijingHei commented Feb 1, 2024 •

edited

Loading