incorrectly versioned OS bits leads to potentially fatal update

[jclulow]

I was trying to update an old LTS system and pkg wanted me to update pkg. This is fine, except when I actually did it, it also tried to pull in system/library:

$ pfexec pkg install -v pkg
            Packages to update:         3
     Estimated space available:  18.71 GB
Estimated space to be consumed: 193.74 MB
       Create boot environment:       Yes
     Activate boot environment:       Yes
Create backup boot environment:        No
          Rebuild boot archive:        No

Changed packages:
omnios
  package/pkg
    0.5.11-151038.0:20210530T145223Z -> 0.5.11-151038.0:20230421T140529Z
  runtime/python-39
    3.9.9-151038.0 -> 3.9.15-151038.0
  system/library
    0.5.11-151038.0:20210705T094101Z -> 0.5.11-151038.0:20220220T121934Z

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                                3/3     3157/3157    33.9/33.9  1.3M/s

PHASE                                          ITEMS
Removing old actions                             9/9
Installing new actions                         12/12
Updating modified actions                  4039/4039
Updating package state database                 Done
Updating package cache                           3/3
Updating image state                            Done
Creating fast lookup database                   Done
Reading search index                            Done
Updating search index                            3/3
Updating package cache                           2/2

A clone of 00dcb0bd exists and has been updated and activated.
On the next boot the Boot Environment 00dcb0bd-1 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

*** Reboot required ***
New BE: 00dcb0bd-1

I was super confused, as the version for system/library has not been bumped, so how could have happened? Inspecting a bit closer, it seems that version of python-39 has a dependency on a timestamped system/library:

$ pfexec pkg contents -r -t depend [email protected]
TYPE    FMRI
require compress/bzip2
require database/sqlite-3
require developer/object-file
require library/expat
require library/libffi
require library/libxml2
require library/ncurses
require library/readline
require library/security/openssl
require library/zlib
require system/library/gcc-runtime
require system/[email protected]:20220220T121934Z

This is pretty unusual, and I don't think it's right. pkg will not allow the whole system to be updated without updating pkg first, which requires a reboot into a new BE -- but that BE will contain an updated system/library without a matching update to system/kernel and other things with which it shares an incorporation requirement. The OS bits incorporation does not specify the package timestamps for OS bits, just the version.

When new OS bits (anything in the osnet-incorporation) come out, they should really get a version roll so that they can be handled correctly by other bits like this; e.g., 151038.0.13 or whatever.

To get out of the hole I have used pkg freeze python-39 so that I can do pkg update -v first, and reboot into that boot environment, prior to trying to update pkg.

[jclulow]

Ironically this did lead to an unbootable system in the end which I have been unpicking. Because of, I guess, the partial rebuild stuff and the busted osnet-incorporation, the system updated everything except for ZFS, which apparently depends on a newer version of some out-of-gate packages because it was rebuilt later than everything else:

$ pkg contents -r -t depend system/file-system/[email protected]:20221018T212343Z
TYPE    FMRI
require pkg:/library/[email protected]
require pkg:/runtime/[email protected]
require pkg:/system/library/[email protected]
require pkg:/[email protected]
require pkg:/library/[email protected]
require pkg:/system/[email protected]
require pkg:/shell/[email protected]
require pkg:/system/library/[email protected]
require consolidation/osnet/osnet-incorporation
require system/library/python/zfs-39
require pkg:/system/[email protected]
require pkg:/system/library/python/[email protected]
require pkg:/system/library/python/[email protected]

When the system came back up, it did not have the fix for 14593, and hung trying to expand the pool.

[citrus-it]

I was super confused, as the version for system/library has not been bumped, so how could have happened? Inspecting a bit closer, it seems that version of python-39 has a dependency on a timestamped system/library:

If I remember correctly this was added because the new python-39 picked up a dependency on a new symbol that appeared in libc (the perils of backporting).

Thanks for reporting. The quickest way out of it is to pkg update -f from the old BE, which will do everything in one step, including pkg itself.

To get out of the hole I have used pkg freeze python-39 so that I can do pkg update -v first, and reboot into that boot environment, prior to trying to update pkg.

Did the new BE (with just the three updated packages) not work then? I would have expected it to because backports are done carefully, and the updated system/library should still have been compatible with the rest of osnet.

[jclulow]

I was hesitant to try to boot it because there is no console access on the machine.

I guess it may well have worked in practice, but in general it is (to me, at least) pretty counter-intuitive that a pkg install pkg operation could require a reboot, then only update one of the inter-dependent OS packages, but still somehow work.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.