CLI profile refactor

[dbutenhof]

Summary

Continuing CLI refactoring: modify the Profile classes to use ProfileArgs instances built from kind=<name> payloads like backends and datasets.

Details

Use CLI to construct a ProfileArgs instance, which is then used to build the Profile.

Like datasets and backends, we now identify a profile "kind", as --profile kind=constant[,param=value...]; directly specified parameters supercede inherited global CLI values like --rate.

Use more meaningful aliases for the global "rate" to match the previous internal names (e.g., "sweep_size"). This also exposes previously hidden parameters like the sweep's strategy_type to run poisson instead of constant async.

Test Plan

Manually tested all profiles, including Replay.

Related Issues

• Related to Refactor CLI/Config in v0.7.0 #724

---

• "I certify that all code in this PR is my own, except as noted below."

Use of AI

• Includes code generated or substantially modified by an AI agent
• Includes tests generated or substantially modified by an AI agent

NOTE: the Generated-by or Assisted-by trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's DEVELOPING.md file.

Assisted-by: Cursor

---

git log

commit 6b3d675
Author: David Butenhof dbutenho@redhat.com
Date: Tue May 26 15:24:51 2026 -0400

CLI profile refactor

Use CLI to construct a ProfileArgs instance, which is then used to build
the Profile.

Like datasets and backends, we now identify a profile "kind", as
`--profile kind=constant[,param=value...]`; directly specified
parameters supercede inherited global CLI values like `--rate`.

Use more meaningful aliases for the global "rate" to match the previous
internal names (e.g., "sweep_size"). This also exposes previously hidden
parameters like the sweep's `strategy_type` to run poisson instead of
constant async.

Manually tested all profiles, including Replay.

Assisted-by: Cursor
Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit 9f24445
Author: David Butenhof dbutenho@redhat.com
Date: Tue Jun 2 09:27:53 2026 -0400

Review comments & cleanup

- Remove Pydantic aspect of Profile class, holding a reference to ProfileArgs.
- Remove constraints from ProfileArgs, to isolate construction and remove some
  report serialization issues.
- Change ProfileArgs deserialization to "forbid" unknown keys. We have two
  classes of profile parameter that aren't "universal":
  - data and data_samples are used only by Replay: we remove these from other
    ProfileArgs -- this will be refactored later.
  - Although random_seed isn't used by all profiles, it's a global used
    throughout GuideLLM, so we promote this to the base ProfileArgs for
    convenience.

Assisted-by: Cursor
Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit f49f3b7
Author: David Butenhof dbutenho@redhat.com
Date: Tue Jun 2 12:13:06 2026 -0400

Odd ..

CI "pre-commit" check found something local pre-commit didn't.

Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit 2c64d1a
Author: David Butenhof dbutenho@redhat.com
Date: Tue Jun 2 19:54:56 2026 -0400

More review changes

- move random_seed to Profile
- formalize abstraction of Profile with ABC

Signed-off-by: David Butenhof <dbutenho@redhat.com>

commit 744b3ec
Author: David Butenhof dbutenho@redhat.com
Date: Tue Jun 2 19:59:17 2026 -0400

Just in case the CI went crazy due to pre-commit updates ...

Signed-off-by: David Butenhof <dbutenho@redhat.com>

---

Assisted-by: Cursor
Signed-off-by: David Butenhof dbutenho@redhat.com

[sjmonson]

The way this is currently written each ProfileArgs is essentially a duplicate of its Profile without its business logic. That is going to be hard to maintain going forward because we will have to maintain identical fields between Args and Profiles. Instead I recommend looking at converting Profiles to a normal non-pydantic registry. See BackendRegistry or DataLoaderRegistry (after #754) for examples.

[sjmonson]

Forgot one thing. There is a bunch of migration code in the CLI run function:

guidellm/src/guidellm/cli/benchmark/run.py

Lines 345 to 364 in 8fdbd28

	# NOTE: Hardcode some of the defaults for now
	# data loader
	loader_args = kwargs.pop("data_loader", {})
	loader_args["kind"] = "pytorch"
	kwargs["data_loader"] = loader_args
	# column mapper
	column_mapper = kwargs.pop("data_column_mapper", {})
	kind = column_mapper.get("kind", "generative_column_mapper")
	column_mapper["kind"] = kind
	kwargs["data_column_mapper"] = column_mapper
	# preprocessors
	preprocessors = kwargs.pop("data_preprocessors", [])
	if not preprocessors:
	preprocessors = [{"kind": "encode_media"}]
	kwargs["data_preprocessors"] = preprocessors
	# finalizer
	finalizer = kwargs.pop("data_finalizer", {})
	if not finalizer:
	finalizer = {"kind": "generative"}
	kwargs["data_finalizer"] = finalizer

If you add the following then you can update BenchmarkGenerativeTextArgs to store the profile args rather then profile name and avoid changing the existing CLI usage.

kwargs["profile"] = {"kind": kwargs.pop("profile", sweep)}

[dbutenhof]

The way this is currently written each ProfileArgs is essentially a duplicate of its Profile without its business logic.

Yeah; I'd started out turning Profile into a plain class, but ran into problems ... and without noticing I'd forgotten to remove the Pydantic registry mixin, I let Cursor try to fix them. Which it did by completing the Pydantic infrastructure. I let it stand to get some other things straightened out "first", and then never remembered to fix it.

store the profile args rather then profile name and avoid changing the existing CLI usage.

Well, I started out with code to transparently promote a str to {"kind": str}, and decided it made more sense to require the kind to be explicit. I suppose it's not entirely similar to the new data deserializer since you can specify a profile without additional parameters.

Anyway, I spent today doing most of the rework, which got messier than I'd expected. I'll resume tomorrow morning, and hopefully have another tested commit up ...

[sjmonson]

Few more nits. Also I forgot to mention earlier but in the original proposal I moved warmup and cooldown config into the ProfileArgs (see #724) since currently they are fed directly into the scheduler. We can tackle that later though if we just want to get this in to unblock other work.

[sjmonson]

@Mergifyio queue

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-06-03 19:52 UTC · Rule: default
• ✅ Checks passed · on draft merge queue: embarking main (06cef29), #750 and #753 together #769
• ✅ Merged — 2026-06-03 20:00 UTC · at 7a424b044101f197b9613b150a7f5653649e3c10 · squash

This pull request spent 7 minutes 18 seconds in the queue, including 6 minutes 53 seconds running CI.

Required conditions to merge

• any of [🛡 GitHub repository ruleset rule Merge Requirements]:
  • check-success = quality (3.10) / type-checks
  • check-neutral = quality (3.10) / type-checks
  • check-skipped = quality (3.10) / type-checks
• any of [🛡 GitHub repository ruleset rule Merge Requirements]:
  • check-success = quality (3.10) / precommit-checks
  • check-neutral = quality (3.10) / precommit-checks
  • check-skipped = quality (3.10) / precommit-checks
• any of [🛡 GitHub repository ruleset rule Merge Requirements]:
  • check-success = quality (3.10) / quality-checks
  • check-neutral = quality (3.10) / quality-checks
  • check-skipped = quality (3.10) / quality-checks
• any of [🛡 GitHub repository ruleset rule Merge Requirements]:
  • check-success = tests (3.10) / e2e-tests
  • check-neutral = tests (3.10) / e2e-tests
  • check-skipped = tests (3.10) / e2e-tests
• any of [🛡 GitHub repository ruleset rule Merge Requirements]:
  • check-success = tests (3.10) / integration-tests
  • check-neutral = tests (3.10) / integration-tests
  • check-skipped = tests (3.10) / integration-tests
• any of [🛡 GitHub repository ruleset rule Merge Requirements]:
  • check-success = tests (3.10) / unit-tests
  • check-neutral = tests (3.10) / unit-tests
  • check-skipped = tests (3.10) / unit-tests
• any of [🛡 GitHub repository ruleset rule Merge Requirements]:
  • check-success = update-description
  • check-neutral = update-description
  • check-skipped = update-description