Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use boxo with provider queue input buffer #10758

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Conversation

gammazero
Copy link
Contributor

No description provided.

@gammazero gammazero added the skip/changelog This change does NOT require a changelog entry label Mar 19, 2025
@lidel
Copy link
Member

lidel commented Mar 20, 2025

Notes form colo:

  • right now, by default provider queue is being persisted in leveldb datastore, which introduces extra disk writes and reads, as noted in Improve the perceived performance of data onboarding boxo#888 (comment)
  • we want Kubo to have an option to avoid doing disk writes when provide system works
    • (idea 1) we could ensure provider queue is persisted under deterministic queue
      • this would allow user to control where provide queue is stored, for example, user could mount in-memory datastore under something like /provides – similar to how flatfs is mounted for `/blocks
    • (idea 2) Reprovider.QueuePersistence Flag that allows user to control if queue is persisted in datastore, or not (prioritizing performance at the cost or slight risk in case of power outage and increased memory usage)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip/changelog This change does NOT require a changelog entry
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants