Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

options: Report image format and virtual size #6

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

nirs
Copy link
Member

@nirs nirs commented Dec 12, 2021

Report image virtual size in OPTIONS so clients can get the image size
without a possibly slow extent call.

Report the format since OPTIONS can report the virtual size only when
the backend provides raw format. This is used when using the http
backend to report OPTIONS to the client.

Reporting virtual size is easy with the nbd and memory backends since
they always use raw format.

When using file backend and qcow2 image, we don't have access to the
image virtual size, and this size is not helpful to the user uploading
or downloading data. Currently we don't know about the image format
since engine does not report it in the ticket.

The http backend reports the info from the remote server, so it depends
on the backend used by the remote server, and on having new server
reporting the format and size.

To keep code and the API simple, we report virtual size only when using
the nbd and memory backends. When engine will report the image format
for the file backend, we can also report the size for raw images access
via the file backend.

This is work in progress, exploring the needed changes. I will post a
cleaner version later.

@nirs nirs marked this pull request as draft January 15, 2022 11:32
@aesteve-rh aesteve-rh added this to the ovirt-4.5.2 milestone Jul 20, 2022
docs/images.md Outdated Show resolved Hide resolved
docs/images.md Outdated Show resolved Hide resolved
docs/images.md Show resolved Hide resolved
docs/images.md Outdated Show resolved Hide resolved
ovirt_imageio/_internal/backends/http.py Outdated Show resolved Hide resolved
docs/images.md Outdated Show resolved Hide resolved
docs/images.md Outdated Show resolved Hide resolved
docs/images.md Outdated Show resolved Hide resolved
docs/images.md Outdated Show resolved Hide resolved
docs/images.md Outdated Show resolved Hide resolved
ovirt_imageio/_internal/backends/file.py Show resolved Hide resolved
test/handlers/images_test.py Outdated Show resolved Hide resolved
@nirs nirs assigned aesteve-rh and unassigned nirs Jul 26, 2022
@nirs nirs added enhancement Enhancing the system by adding new feature or improving performance or reliability performance labels Jul 26, 2022
@aesteve-rh aesteve-rh force-pushed the options-size branch 2 times, most recently from c647a60 to 79b3aee Compare July 29, 2022 11:11
@nirs nirs modified the milestones: ovirt-4.5.2, ovirt-4.5.3 Jul 29, 2022
Import and use units constants in the test to improve
readability.

Signed-off-by: Albert Esteve <[email protected]>
aesteve-rh and others added 2 commits August 8, 2022 10:10
Fix small typo on extents example.

Signed-off-by: Albert Esteve <[email protected]>
Report image virtual size in OPTIONS so clients can get the image size
without a possibly slow extent call.

Report the format since OPTIONS can report the virtual size only when
the backend provides raw format. This is used when using the http
backend to report OPTIONS to the client.

Reporting virtual size is easy with the nbd and memory backends since
they always use raw format.

When using file backend and qcow2 image, we don't have access to the
image virtual size, and this size is not helpful to the user uploading
or downloading data. Currently we don't know about the image format
since engine does not report it in the ticket.

The http backend reports the info from the remote server, so it depends
on the backend used by the remote server, and on having new server
reporting the format and size.

To keep code and the API simple, we report virtual size only when using
the nbd and memory backends. When engine will report the image format
for the file backend, we can also report the size for raw images access
via the file backend.

Change-Id: I89118301c98dc2d11c25a4d1e7ef83df26336f01
Related: oVirt#67
Bug-Url: https://bugzilla.redhat.com/1924945
Signed-off-by: Nir Soffer <[email protected]>
@aesteve-rh
Copy link
Member

@nirs I have done some tests and can see the new options available when I run an example server locally, with a nbd backend. It looks correct.

But unfortunately I haven't been able to do any test showing any relevant speedup. It downloads at roughly the same speed as master branch. Tried two methods: with direct download with a GET request, and through the client using the imageio-client in the examples folder. Also tried with empty 8TiB images, and with fedora-36 image, resized to 8 TiB.

I would like to have some data to share before ready this PR up, but I'm not sure what else can I try to obtain it, or what I may be missing in my own local testing.

@nirs
Copy link
Member Author

nirs commented Aug 8, 2022

@aesteve-rh with download you will not see any difference, since we always get extents once.

You should see a difference with incremental backup. With current release we get extent twice,
once with context=zero to get the size of the source, and then with context=dirty to get dirty blocks information. With 8TiB image the extents call takes 24 seconds (on my system) so calling twice will spend 48 seconds only on the extents calls. With the new feature we will call extents only once (with context=dirty), since image size is available via OPTIONS.

@sandrobonazzola sandrobonazzola removed this from the ovirt-4.5.4 milestone Jan 13, 2023
@nirs
Copy link
Member Author

nirs commented Feb 2, 2024

Let’s close since it stale for long time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancing the system by adding new feature or improving performance or reliability performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants