Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spec Scheduler #1487

Open
wants to merge 569 commits into
base: inference
Choose a base branch
from
Open

Spec Scheduler #1487

wants to merge 569 commits into from

Conversation

goliaro
Copy link
Collaborator

@goliaro goliaro commented Sep 4, 2024 •
edited by wmdi
Loading

Description of changes:

Related Issues:

Linked Issues:

  • Issue #

Issues closed by this PR:

  • Closes #

This change is Reviewable

zikun-li and others added 30 commits May 27, 2024 13:48
@zikun-li
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
e0451b5 
…cscheduler
@zikun-li
Optimized metadata transfer from host to GPU.
d3e76d6
@chenzhuofu
chore: add sort in argtopk
0acdff2
@zikun-li
Added a version of async specinfer. The old sync version is rewrite i…
b1dac12 
…nto the API serve_spec_infer_sync.
@zikun-li
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
f106b70 
…cscheduler
@zikun-li
Fixed load token task.
e674f71
@chenzhuofu
chore: minor output format for alignment checking
cf18e8a
@zikun-li
Fixed a bug.
0f66b08
@zikun-li
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
1feb73d 
…cscheduler
@chenzhuofu
chore: minor output format for alignment checking
6c608ac
@Flechman
Merge branch 'specscheduler' into specschedul_profiling
d93a412
@Flechman
Update write_to_output_file()
a654262
@chenzhuofu
feat: partial enable cudaGraph
31abfbf
@chenzhuofu
chore: align tree_inc_attn w/ inc_attn
c9bff3c
@chenzhuofu
chore: time measurement in tree_inc
afb9624
@zikun-li
Move the bitmask to shared memory to improve performance.
042d6a9
@zikun-li
Removed an unused line.
b68ce45
@zikun-li
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
f25122c 
…cscheduler
@chenzhuofu
chore: remove unused
10c4f65
@chenzhuofu
chore: minor
d4b0baf
Fix raft patch for Docker
653069b
@Flechman
Update config.linux
040525b
@zikun-li
Added a copy constructor for BatchConfig.
67117e2
@zikun-li
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
b6450f4 
…cscheduler
@zikun-li
Optimized mask loading.
3edf56f
@zikun-li
Removed the parameter speculative_decoding from arg_top_k. Added a pa…
6ac1d7a 
…rameter renormalize to arg_top_k.
@zikun-li
Support renormalize for arg_topk.
8aa38fb
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
900d994 
…cscheduler
@zikun-li
Modified the InferenceResult class. Added constructors for it.
378b598
@zikun-li
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
2f37d5e 
…cscheduler
@chenzhuofu
chore: minor
5ddeb11
@chenzhuofu
Reimplemented add_tokens_to_spec_token_tree.
fd6eb7b
@chenzhuofu
merge
4b4d55c
@chenzhuofu
chore: refactor lock
5623fc5
@chenzhuofu
fix: request per batch
f524aac
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
d42c6ce 
…cscheduler
@chenzhuofu
Optimizes CPU performance of the scheduler
0b7a02f
@chenzhuofu
chore: incr decode add slo attainment
fa13afa
@chenzhuofu
Optimized some usage of priority queues.
86f95dc
@chenzhuofu
feat: support slo ratio sampling
f169812
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
e1f711b 
…cscheduler
@chenzhuofu
fix: incr_decode doesn'y have slo attainment metric
7ae7edd
@chenzhuofu
feat: support early_drop switch
ff3af26
@chenzhuofu
chore: add request_per_second param
1a1dc56
@chenzhuofu
chore: change early drop logic
9f034a4
@chenzhuofu
feat: add emission output
fe55382
@chenzhuofu
Dynamically control tree width to not exceed max_tokens_per_ssm_batch.
0420199
@chenzhuofu
Simplified the method to add tokens to the token trees.
7c7376a
@chenzhuofu
Dynamic max tree depth control
4396fc9
@chenzhuofu
feat: update raft dependency (select_k)
eee85fe
@chenzhuofu
feat: raft build file
7caaf72
@chenzhuofu
chore: minor
2ab10b1
@chenzhuofu
feat: update argTopk op
57f6378
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
47be784 
…cscheduler
@chenzhuofu
chore: update emission trace
9fa1f4e
@chenzhuofu
feat: add TraceEmissionMachine
0a516c6
@chenzhuofu
Add back old scheduler
2071273
@chenzhuofu
feat: add trace generator
79f9130
@chenzhuofu
fix: initialization issue; read microsecond
f224b5e
@chenzhuofu
Merge branch 'specscheduler' of github.com:flexflow/FlexFlow into spe…
1fe612b 
…cscheduler
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@goliaro @zikun-li @chenzhuofu @Flechman