Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
terrywbrady authored Apr 2, 2024
1 parent 4939fb8 commit 4bd0436
Showing 1 changed file with 1 addition and 30 deletions.
31 changes: 1 addition & 30 deletions design/queue-2023/README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1 @@
# Merritt Queue Redesign 2023

## Objectives
- Match ingest workload to available resources (compute, memory, working storage)
- dynamically provision resources to match demand
- dynamically manage running thread count based on processing load
- Hold jobs based on temporary holds (collection lock, storage node lock, queue hold)
- Graceful resumption of processing in progress
- allow processing to be resumed on a different ingest host
- Accurate notification of ingest completion (including inventory recording)
- send accurate summary email on completion of a batch regardless of any interruption that occurred while processing

## Design Details
- [Merritt ZK API](api.md)
- [Queue State Transitions](states.md)
- [Queue Entry Data Storage](data.md)
- [State Transition Details](transition.md)
- [Underlying Queue Service](service.md)
- [Queue Manager](manager.md)
- [Admin Function Mapping](queue-admin.md)
- [Batch and Queue State Enums](https://github.com/CDLUC3/merritt-tinker/tree/main/state-transition)
- How is the queue item id number conveyed?
- How does priority work int the queue item sort?

## Zookeeper provides a "Watcher" object. Could this be helpful for the Batch Queue?
- https://zookeeper.apache.org/doc/r3.3.3/zookeeperProgrammers.html#ch_zkWatches

## Links
- https://github.com/CDLUC3/mrt-doc/issues/1062 - Background issue that led tot hsi design
- https://github.com/CDLUC3/mrt-doc/issues/1753 - Admin Manager
See https://github.com/CDLUC3/mrt-zk/blob/main/README.md

0 comments on commit 4bd0436

Please sign in to comment.