Avoid deepcopy when submitting graph

[fjetter]

This avoids at least one deepcopy of the graph and therefore reduces overhead during submit

Typically, one would simply add ToPickle/Serialize to the dict value and pass the graph through directly. However, this would make it impossible to get clear error messages on (de-)serialization exceptions which is why this was pulled out to a manual call back then.

However, passing the header and frames as dictionary values directly causes msgpack to simply copy the bytes into the msgpack bytestream instead of us passing through the frames implicitly.

This avoids an unnecessary copy

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files  ±0      27 suites  ±0   11h 18m 56s ⏱️ + 5m 39s
 4 100 tests ±0   3 984 ✅ ±0    112 💤 ±0  4 ❌ ±0 
51 400 runs  ±0  49 084 ✅  - 3  2 309 💤 ±0  7 ❌ +3 

For more details on these failures, see this check.

Results for commit ef57b4a. ± Comparison against base commit 16aa189.

♻️ This comment has been updated with latest results.

[hendrikmakait]

Thanks, @fjetter! One suggestion, feel free to ignore.

[fjetter]

well, that's disappointing

I hope/suspect that's the Serialize. I'll try with sticking to ToPickle

[hendrikmakait]

What's the status of this PR?

[fjetter]

I rebased and removed the Scheduler->Worker msg change to isolate the graph submission change. I'll rerun benchmarks on this shortly

[fjetter]

Perf tests on this look good for now (these are the ordinary tests). I also ran tpch but tpch didn't show any signal. Looks like most are not actually affected but there are some with strong gains. This is pretty much what I'd expect.

The strong red bar is test_set_index_on_uber_lyft which has a significant error

the noisy rechunking bars are also noise.

static-dashboard - 2025-04-01T174251.730.zip

From a perf perspective I consider this a win but CI seems to be extra flaky. I assume this changes some timings which makes the entire thing more fragile.

[fjetter]

It seems like CI overall is currently pretty broken. Timing somewhat aligns with the HLG change but nor perfectly.