Build ProfileArgs in BenchmarkGenerativeTextArgs by dbutenhof · Pull Request #774 · vllm-project/guidellm

dbutenhof · 2026-06-05T11:43:34Z

Summary

Let BenchmarkGenerativeTextArgs build the ProfileArgs embedded object.

Details

This is a refactoring that follows up on a discussion during review of #753.

The key was to shuffle data/data_samples from ReplayProfileArgs to ReplayProfile so that we can rely on a built loader by the time we need it. Now we can fully build ProfileArgs up front.

Test Plan

Ensure proper parsing of profile parameters, particularly,

merging of global rate & rampup into ProfileArgs dict
proper handling of global --rate with aliased ProfileArgs parameters

E.g.:

uv run guidellm benchmark --target http:// --max-seconds 10 --profile kind=replay,time_scale=0.3 --data kind=trace_synthetic,path=${HOME}/replay.jsonl --rate 5
uv run guidellm benchmark --target http:// --max-seconds 10 --data "kind=synthetic_text,prompt_tokens=256,output_tokens=128" --rate 5 --profile kind=async

Related Issues

N/A

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes code generated or substantially modified by an AI agent
Includes tests generated or substantially modified by an AI agent

NOTE: the Generated-by or Assisted-by trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's DEVELOPING.md file.

git log

commit 3e27713
Author: David Butenhof dbutenho@redhat.com
Date: Fri Jun 5 02:03:41 2026 -0400

Build ProfileArgs in BenchmarkGenerativeTextArgs

This follows up on a discussion in #753.

Shuffle data/data_samples from ReplayProfileArgs to ReplayProfile so that we
can rely on built loader. Now we can fully build ProfileArgs up front.

Assisted-by: Cursor
Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit 66d9025
Author: David Butenhof dbutenho@redhat.com
Date: Fri Jun 5 15:04:53 2026 -0400

Pull random_seed back out of kwargs

Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit c3cf2d0
Author: David Butenhof dbutenho@redhat.com
Date: Fri Jun 5 20:12:36 2026 -0400

Fix a typo

Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit a11f2dd
Author: David Butenhof dbutenho@redhat.com
Date: Mon Jun 8 08:45:01 2026 -0400

Complain about duplicate rates

More consistent behavior than ignoring one.

Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit f240157
Author: David Butenhof dbutenho@redhat.com
Date: Tue Jun 9 16:21:31 2026 -0400

Fix merge queue test breakage

(Yeah, that's why we *have* a merge queue!)

Signed-off-by: David Butenhof <dbutenho@redhat.com>

Assisted-by: Cursor
Signed-off-by: David Butenhof dbutenho@redhat.com

sjmonson

Small nit but not really blocking. Otherwise looks good.

jaredoconnell · 2026-06-05T20:59:15Z

        profile=args.profile,
-        rate=args.rate,
-        random_seed=args.random_seed,
-        rampup=args.rampup,


Is the rampup field this was referencing still used anywhere?

It's referenced in ProfileArgs. (Which, as Sam pointed out earlier, is really weird because it's a completely different path from warmup & cooldown, and shouldn't be -- but that's beyond the scope of this PR.) Anyway, it was needed because we built ProfileArgs here; but we no longer do.

It probably makes sense to create an issue then to note the odd design and retirement of that pathway.

Yeah, we'd talked about unifying the paths for all of these; an issue makes sense, but when I started to think about writing one I got hung up on details and didn't get far.

mergify · 2026-06-09T18:54:36Z

sjmonson · 2026-06-09T18:57:01Z

@Mergifyio dequeue

sjmonson · 2026-06-09T18:58:23Z

^ @dbutenhof Sorry it auto queued since we have two approvals but I am not sure if you are done. Just requeue if you're ready.

dbutenhof · 2026-06-09T19:36:19Z

@Mergifyio queue

mergify · 2026-06-09T19:36:41Z

This follows up on a discussion in vllm-project#753. Shuffle data/data_samples from ReplayProfileArgs to ReplayProfile so that we can rely on built loader. Now we can fully build ProfileArgs up front. Assisted-by: Cursor Signed-off-by: David Butenhof <dbutenho@redhat.com>

Signed-off-by: David Butenhof <dbutenho@redhat.com>

More consistent behavior than ignoring one. Signed-off-by: David Butenhof <dbutenho@redhat.com>

(Yeah, that's why we *have* a merge queue!) Signed-off-by: David Butenhof <dbutenho@redhat.com>

dbutenhof · 2026-06-09T20:29:15Z

@Mergifyio requeue

mergify · 2026-06-09T20:29:18Z

Merge Queue Status

🛑 Queue command has been cancelled

mergify · 2026-06-09T20:44:04Z

dbutenhof requested review from jaredoconnell and sjmonson June 5, 2026 11:43

dbutenhof self-assigned this Jun 5, 2026

dbutenhof added internal filed by core contributor or associate cleanup Internal refactoring or improvement, including CI, that's not directly user-visible. labels Jun 5, 2026

sjmonson previously approved these changes Jun 5, 2026

View reviewed changes

Comment thread src/guidellm/benchmark/profiles/asynchronous.py Outdated

dbutenhof dismissed sjmonson’s stale review via 38e6251 June 5, 2026 19:05

dbutenhof force-pushed the refactor/redata branch from 38e6251 to fecc715 Compare June 5, 2026 19:08

dbutenhof requested a review from sjmonson June 5, 2026 20:33

jaredoconnell reviewed Jun 5, 2026

View reviewed changes

dbutenhof force-pushed the refactor/redata branch from fecc715 to e4792f6 Compare June 6, 2026 00:20

dbutenhof requested a review from jaredoconnell June 6, 2026 00:21

jaredoconnell previously approved these changes Jun 6, 2026

View reviewed changes

Comment thread src/guidellm/benchmark/profiles/concurrent.py Outdated

Comment thread src/guidellm/benchmark/profiles/replay.py Outdated

dbutenhof dismissed jaredoconnell’s stale review via a98c255 June 8, 2026 12:45

dbutenhof requested a review from jaredoconnell June 8, 2026 13:06

jaredoconnell previously approved these changes Jun 8, 2026

View reviewed changes

dbutenhof force-pushed the refactor/redata branch from a98c255 to fc00640 Compare June 9, 2026 13:22

sjmonson previously approved these changes Jun 9, 2026

View reviewed changes

mergify Bot added a commit that referenced this pull request Jun 9, 2026

Merge of #774

77a775c

mergify Bot mentioned this pull request Jun 9, 2026

merge queue: embarking main (48affcf) and #774 together #784

Closed

34 tasks

mergify Bot added the queued label Jun 9, 2026

mergify Bot added dequeued and removed queued labels Jun 9, 2026

mergify Bot added the queued label Jun 9, 2026

mergify Bot added a commit that referenced this pull request Jun 9, 2026

Merge of #774

5707f69

mergify Bot mentioned this pull request Jun 9, 2026

merge queue: embarking main (80f00e2) and #774 together #785

Closed

34 tasks

mergify Bot added dequeued and removed dequeued queued labels Jun 9, 2026

dbutenhof added 5 commits June 9, 2026 16:07

Pull random_seed back out of kwargs

66d9025

Signed-off-by: David Butenhof <dbutenho@redhat.com>

Fix a typo

c3cf2d0

Signed-off-by: David Butenhof <dbutenho@redhat.com>

Complain about duplicate rates

a11f2dd

More consistent behavior than ignoring one. Signed-off-by: David Butenhof <dbutenho@redhat.com>

Fix merge queue test breakage

f240157

(Yeah, that's why we *have* a merge queue!) Signed-off-by: David Butenhof <dbutenho@redhat.com>

dbutenhof dismissed stale reviews from sjmonson and jaredoconnell via f240157 June 9, 2026 20:22

dbutenhof force-pushed the refactor/redata branch from fc00640 to f240157 Compare June 9, 2026 20:22

mergify Bot removed the dequeued label Jun 9, 2026

dbutenhof requested review from jaredoconnell and sjmonson June 9, 2026 20:30

sjmonson approved these changes Jun 9, 2026

View reviewed changes

mergify Bot added the queued label Jun 9, 2026

mergify Bot merged commit 501c0c6 into vllm-project:main Jun 9, 2026
12 checks passed

mergify Bot removed the queued label Jun 9, 2026

dbutenhof deleted the refactor/redata branch June 9, 2026 20:45

Uh oh!

Conversation

dbutenhof commented Jun 5, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Test Plan

Related Issues

Use of AI

git log

Uh oh!

sjmonson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jaredoconnell Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

dbutenhof Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

jaredoconnell Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

dbutenhof Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

sjmonson commented Jun 9, 2026

Uh oh!

sjmonson commented Jun 9, 2026

Uh oh!

dbutenhof commented Jun 9, 2026

Uh oh!

mergify Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

dbutenhof commented Jun 9, 2026

Uh oh!

mergify Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

mergify Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dbutenhof commented Jun 5, 2026 •

edited by github-actions Bot

Loading

mergify Bot commented Jun 9, 2026 •

edited

Loading

mergify Bot commented Jun 9, 2026 •

edited

Loading

mergify Bot commented Jun 9, 2026 •

edited

Loading

mergify Bot commented Jun 9, 2026 •

edited

Loading