After upgrade to 4.22.0.0 snapshot.delta.max not respected

[nxsbi]

problem

I upgraded to 4.22.0.0 on Nov 11. I have many volumes that are set to 3 snapshots max stored by my users.

I noticed that all snapshots since Nov 11 (all existing and new ones) seem to create a massive chain of snapshots.
I had 60+ days snapshot files on the disk for every volume with daily snapshots, while the snapshot.delta.max is only set to 7. And snapshot policy is 3 snapshots only.

It seems the full snapshot before Nov 11 in my case was the last full snapshot. After that there is no more full snapshots, only incremental snapshots.

What I see in database:

in snapshots table:

And in snapshot_store_ref (have hidden rows in between - but they all show as "Hidden"

Please help! I cannot even clear my storage which is now hit 95% used capacity and about to blow up!! Its tens of extra terabytes of data!! with no way to clear it other than deleting all snapshots which is a huge problem for users...

versions

ACS 4.22.0
XCP-NG 8.2

The steps to reproduce the bug

1.Upgraded from 4.19 .1 to 4.22
2.
3.
...

What to do about it?

Identify what is causing the snapshot.delta.max to not be respected.

[nvazquez]

Hi @nxsbi thanks for reporting the issue. An additional question, before the upgrade did you see these 'Hidden' entries on database, or were them removed?

On 4.21 the KVM incremental snapshot feature has been introduced (#8907) which seems to do a similar approach leaving Hidden entries on database but not removing them from storage, is it possible that its also the default behavior now for Xen/XCP? @sureshanaparti @JoaoJandre @DaanHoogland

[nxsbi]

Hi @nvazquez
Before upgrade the data reflected only two values - Destroyed or BackedUp (or Ready)

I did not do any extra steps around this.

Related to this - I had originally reported a related issue - #9501
This one is same issue, except that now there are no Full snapshots every 7 days at all. I used to be able to determine which is a Full snapshot based on Parent ID = 0 (in snapshot_store_ref) and now I no longer see that.
I also reviewed the actual snapshot sizes and there is no large files except the one from before upgrade.

Not sure if any other xcp-ng + cloudstack users can comment