Remote Python Support for Benchmark Runner

[SujeethJinesh]

Description

Adds a recipe for testing and eventually benchmarking the Pathways Remote Python feature.

This is an initial PR that will set the precedent for future recipes. It is designed to be shareable and have common functionality (like flag parsing) that can be shared across recipes, but flexible and standard so that multiple team members can simply run the file with minimal configuration changes. It's designed to avoid superfluous flag adding and continue pythonically using configs.

This recipe in particular is for testing functionality for remote python.

Tests

Tested this change on a test remote python sidecar server on a v6e and a v5e shared test cluster.

Sibling XPK change: AI-Hypercomputer/xpk#326

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.

[Obliviour]

I am wondering if we can create a programmatic way of adding pathways params without needing to duplicate the config - for example default_basic_1 is sharable with mcjax and pathways and when the user has a pathways_config applied, it sets the pathways specific flags? I think the main goal to help solve is share a config between mcjax and pathways - and avoid the need to duplicate updates.

Curious what you think of this idea?

[SujeethJinesh]

For right now there are only a few flags luckily that are pathways specific (e.g. enable_pathways_goodput), but we do often test using a lot more features enabled. A rough ideas I have is to separate the specific model tuning params from the workload params - e.g. like goodput, checkpointing, metrics, etc. That way the models can be the same, but the workload params can be suited to the specific type. Maybe worth expanding on if you think it's a good initial idea?