Engineering: Custom Configs for Testing

[eleanorjboyd]

This is a meta-issue tracking the variety of engineering related work on this proposal #21845 for custom args in testing.

Current status: basic MVP of features has been created in a prototype for test.

To Do Items:

• get test config environment variables to work with debugging
• support env file argument in settings.json config
• remove params from launch.json that aren't possible to be put for a test debug config
• put all new commands about configurations in the command palette
• handle pytest detection to make extension reactive to the current user environment
• switch off of the python.testing.pytestArgs and python.testing.pytestEnabled settings

Outstanding Bugs:

• test config env vars not working in debugging
• when a user deletes a discovery config, that test explorer tree does not get removed until a full reload

Outstanding Questions:

• to confirm, the python path environment variable defined by the user in the settings.json config is appended to the python path and should not fully replace that path right?
• How will we handle settings sync with configs?
• Can users set these configs in their user settings? What would this mean?
• Can a user have duplicate arguments in their settings.json and launch.json? If not how can we add a restriction or surface this?
• Do we want framework in the individual configs? Do we want it set at a workspace level? How does switching between configs with different frameworks go? (frameworks being pytest and unittest)
• Given that discovery configs create a different test tree in the explorer with their own buttons, when discovery configs >2 should those run buttons use their correlated config or the default?
• What are we calling these? Configs? Profiles? How do we differentiate between the simplest test config and the default test profile as defined by the testing API.
• When creating a new config (like from the Manage Test Configs menu) should we ask framework or assume?

Multiroot scenario:

We are still in the beginning stages of exploring multiroot for this project. The MVP looks better than expected but still have bugs / questions

bugs:

• when you run a config from a given workspace B it will run that config across all workspaces and their tests
• when you create configs in workspace A then B, only those in B show up in the dropdown
• configs are not clearly associated with a workspace

outstanding questions:

• How does it work to create configs across workspaces?
• How should the test tree look in the explore panel? Will it include all roots, just the currently active one?
• When using the run button menu will this always prompt for the workspace? (rn it is yes and I think that is a good way to go)