[Workflow Interface]: Refactoring FLSpec and Runtime

[scngupta-dsp]

SUMMARY

One of the main goals of Workflow API is to clearly separate Workflow definition and Runtime infrastructure. While the current implementation is effective, there is some coupling between FLSpec and Runtime and scope for further improvement

This proposal refines the interaction between FLSpec and Runtime classes, such as LocalRuntimeand FederatedRuntime to make the design cleaner and more modular

MOTIVATION

(Current) High Level Design:

In the example shown above, both LocalRuntime and FederatedRuntime are directly assigned to FLSpec Instance illustrating how the runtime is integrated (and coupled) with the flow.

From the above illustration we can see that runtime is an attribute of FLSpec. When flflow.run() is called, the execution flows to the associated runtime, whether it is LocalRuntime or FederatedRuntime

Potential issues with Current Approach:

Any future changes or enhancements to Runtime may need changes to FLSpec

• Currently, flflow.run() is a blocking call, preventing users from querying the experiment status or performing other actions while the experiment is running. While it is possible to modify this behavior by passing an argument to indicate whether sync or async execution is desired, changes are required to FLSpec which should ideally not be the case
• Some more potential Runtime related features where FLSpec may potentially need to be modified are as follows:
  • Async execution of flow shall require providing further APIs in FLSpec to user to query the status, retrieve the results and clean experiment data
  • For long running experiments ability to disconnect from Federation and reconnect maybe needed
  • API to enqueue multiple experiments on the Federated Infrastructure (dependency on Director support also)

PROPOSED APPROACH

(Proposed) High Level Design: To decouple FLSpec from Runtime, we propose:

• Introducing a run method within the runtime
• Instead of binding a runtime instance to the FLSpec, the runtime’s run method will accept the FLSpec instance as an argument

This approach shall achieve a clean separation of Workflow and Runtimes, aligning with the principles of a well-defined Workflow API. With this approach it should be possible to enhance Runtime without impacting FLSpec

TECHNICAL DETAILS

• API Changes:
  • The run() method will be added to LocalRuntime and FederatedRuntime to centralize execution logic within the runtime infrastructure.
  • The run() method will be removed from FLSpec to streamline its responsibilities and improve the separation between Workflow definition and Workflow execution.
  • Deprecation of run_local() and run_federated() in FLSpec: These methods will also be removed, with their functionality now managed by the respective runtime classes.
• Backward Compatibility: This change will is NOT backward compatible. Existing workflows use flflow.run() will need to be updated to use the new run() method in the appropriate runtime classes.
• Dependencies: None

KEY BENEFITS

• Decouple FLSpec from Runtime
• Modular Design
• Enables further enhancements to Runtime without impacting FLSpec

RISKS

1. Adoption Challenges: Users familiar with the existing API might face difficulties transitioning to the new approach.
2. Development Effort for refactoring and updating the tutorials could be significant
3. Testing Effort and chances of regressions being introduced

MITIGATION

1. Documentation and Tutorials shall provide examples to help users become familiar with the new approach
2. Focused reviews and existing test cases can be used to avoid regressions

NEXT STEPS

1. Prepare detailed design for Runtime and message sequence / flow