Improved progress reporting for Xet uploads #3096
Conversation
This PR adds detailed progress reporting for upload_files when hf_xet is used, showing both per-file progress and accurate total progress. Total progress speed, which includes both deduplication and data transfer, is also separated out into separate bars. Requries xet-core / hf_xet at commit 4faec0b or later.
|
Here is a video of the current version of it in action: Screen.Recording.2025-05-20.at.12.50.49.PM.mov |
rajatarya
requested review from
hanouticelina
and removed request for
hanouticelina
|
@hanouticelina @Wauplin : @hoytak has worked on improving the UX with xet uploads, and this draft PR shows how more accurate reporting can be achieved on the In this he replaced The main motivation for changing the UX with xet is due to the additional step of processing / chunking files. As you have seen, folks have been confused by seeing only file-level progress, and the current UX has 'choppy' upload behavior (the progress bar looks stalled and then jumps). By having that tracked explicitly with this new implementation it should be a lot clearer to users when their overall transfer is complete (TOTAL line), and the processing step (). Before embarking on refactoring the download path with similar UX changes Hoyt wanted to get your feedback / approval with this direction of changes. He's also open to UX changes - naming - colors - whatever with this approach. cc: @bpronan Also, see Slack thread here: https://huggingface.slack.com/archives/C02V5EA0A95/p1747771966873299 |
setup.py
Outdated
| @@ -19,6 +19,7 @@ def get_version() -> str: | |||
| "requests", | |||
| "tqdm>=4.42.1", | |||
| "typing-extensions>=3.7.4.3", # to be able to import TypeAlias | |||
| "rich>=14.0.0" | |||
This PR adds detailed progress reporting for upload_files when hf_xet is used, showing both per-file progress and accurate total progress. Total progress speed, which includes both deduplication and data transfer, is also separated out into separate bars.
Requries xet-core / hf_xet at commit 4faec0b or later, which can be obtained using
pip install --pre "hf-xet==1.1.3dev0"