Workflows: Add first class support for input/output data references

[WhitWaldo]

Describe the feature

Per the parent issue we should identify an means of updating the Workflow to better enable developers to return data in an activity and support automatically persisting this somewhere and returning only a reference to it and vice versa - given a reference, retrieve the data when an activity is started.

I do think that this should rely on the proposed binary store building block so we have a high-performance mechanism to store arbitrary blobs, using the API shape identified in that proposal.

Release Note

RELEASE NOTE:

[olitomlinson]

@WhitWaldo for inspiration, I was proposing this syntax 4 years ago on the Durable Functions project. Take a peek…

Azure/azure-functions-durable-extension#1682 (comment)

[olitomlinson]

Building on my previous Reference<T> example (where T is the large object to be referenced ), I think it may be beneficial to include an additional optional generic (Reference<T,U>) which is not referenced, so that small metadata can be passed back to the Workflow (therefor serialised as normal)

[FunctionName("Orchestration")]
public async Task Orchestration([OrchestrationTrigger] IDurableOrchestrationContext context)
{
    List<Reference<Foo, Bar>> fooRefs = new List<Reference<Foo,Bar>>(); 
       
    foreach(var i in Enumerable.Range(0,100)
    {
        var fooRef = await context.CallActivityAsync<Reference<Foo,Bar>>("GetFoo", i );

        fooRefs.Add(fooRef);

        // use fooRef.Bar here to do metadata stuff, logging etc
    }

    await context.CallActivityAsync("ProcessFoos", fooRefs, LazyLoadReferences : true | false );
}

[FunctionName("GetFoo")]
public async Task<Reference<Foo,Bar>> GetFoo([ActivityTrigger] int id)
{
    var foo = await getLargeFooObject();

    var bar = new Bar("this operation took 30 seconds");

    return new Reference<Foo,Bar>(foo, bar);
}
        
[FunctionName("ProcessFoo")]
public async Task ProcessFoo([ActivityTrigger] List<Reference<Foo>> foos )
{
    foreach(var @ref in foos ) 
    {
         var loadedFoo = await @ref.GetDataAsync();
    }
}

In the example above, Foo is the large object which is automatically referenced. Bar is serialized as normal and is available to the Workflow code as normal.