Rules clarification to prevent sorting beyond datset boundaries in LLMs

[nvashutoshd]

This PR addresses and clarifies the rules for dataset sorting in LLM benchmarks, specifically focusing on the offline scenario. The changes aim to ensure fair and consistent performance evaluation across submissions.

Proposed changes for LLM benchmarks-

1. Dataset Sorting Limitation: Sorting is restricted to within the boundaries of a single dataset copy. This prevents the creation of batches with identical queries from multiple dataset copies, which could lead to artificial performance advantages.
2. Clarification on Runtime: Submitters are still allowed to run beyond the minimum duration time, and sorting of queries within a dataset copy is permitted.
3. Addressing Performance Bias: The new rule eliminates the potential unfair advantage gained from batching identical queries with identical sequence lengths across multiple dataset copies. The refined rules better reflect real-world scenarios, where identical queries are not typically grouped together

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[nvashutoshd]

cc @nv-ananjappa @nvzhihanj

[mrmhodak]

WG Notes:

1. Sorting outside of the dataset boundary prohibited (regardless of runtime).
2. LLMs instead of list of benchmarks

[nvashutoshd]

recheck

[mrmhodak]

@nvashutoshd @ashwin:
This seems to be impractical to implement if somebody wants to sort. Loadgen sends one query with all the data, there are no markers where dataset ends and begins. To sort, one would need to hardcode how long the dataset is (or similar), which is explicitly prohibited.

[nvzhihanj]

This seems to be impractical to implement if somebody wants to sort.

@mrmhodak the scenario you mentioned only happens when: 1) offline scenarios, where all queries are dumped to the SUT in one shot; 2) requested query count is larger than the dataset size (e.g. if you specify a larger min_duration or min_query_count).

I do agree it's impractical to implement sorting in such scenario, but sorting across boundary will enable exploitative actions like batching same queries which should not be allowed. In the future, LoadGen should be the one implementing and accepting sorting lambdas. @pgmpablo157321 FYI.

[nvashutoshd]

@mrmhodak @nv-ananjappa @nvzhihanj - I've updated the proposed rule change based on our discussion in the WG meeting this morning.

The rules now cover any/all LLMs, and limits sorting outside dataset boundary irrespective of runtime.