Complex Aggregation: building sequential workflows

[cfeltz34]

Expected Behavior / New Feature

Currently when using the query aggregators all the calls merge synchronously.
In some cases it may be interesting to ensure that these calls are made synchronously:

• the result of request 1 is retrieved
• then, after a possible business processing, it is added to the parameters of the request 2
  I made the change on a personal fork, it is quite easy to set up (I started from the principle that by default the parameters are pre-filled identically for all requests.
  Here is an example of use:

        override public async Task<DownstreamResponse> Aggregate(List<HttpContext> contexts, RequestDelegate next = null)
        {
            if (!sequential) 
            {
                await FireAll(contexts, next);
                var response1 = GetResponse(contexts[0]);
                var result1 = await GetResult<ExampleApiMessage>(response1);
                var response2 = GetResponse(contexts[1]);
                var result2 = await GetResult<ExampleData>(response2);

                var result = new AggregateData(result1, result2);
                return Return(result);
            }
            else 
            {
                await Fire(contexts[0], next);
                var response1 = GetResponse(contexts[0]);
                var result1 = await GetResult<ExampleApiMessage>(response1);

                SetQueryString(contexts[1], new QueryString("?id=4&parameter=" + result1.message));
                await Fire(contexts[1], next);
                var response2 = GetResponse(contexts[1]);
                var result2 = await GetResult<ExampleData>(response2);

                var result = new AggregateData(result1, result2);
                return Return(result);
            }            
        }

As soon as calls become sequential, it becomes interesting to allow POST-type requests on aggregators. So I also removed this limitation. Here is an example of using with an aggregator with POST and GET:

        override public async Task<DownstreamResponse> Aggregate(List<HttpContext> contexts, RequestDelegate next = null)
        {
            // Get request body
            // dynamic body = await GetBodyAsDynamic(contexts[0]);
            var body = await GetBodyAsDictionary(contexts[0]);

            SetQueryString(contexts[0], new QueryString("?test=" + body["paramForGet"]));
            await Fire(contexts[0], next, "GET");
            var response1 = GetResponse(contexts[0]);
            var result1 = await GetResult<ExampleApiMessage>(response1);

            // Refactor body and refresh request
            body.Remove("paramForGet");
            body.Add("param2", result1.message);
            SetBody(contexts[1], body);

            // Fire post request
            await Fire(contexts[1], next, "POST");
            var response2 = GetResponse(contexts[1]);
            var result2 = await GetResult<DefaultApiMessage>(response2);
            return Return(result2);
        }

My code is not very sexy I admit it because I made sure that there is the least code that differs from the original. I think that ideally there should be more automatic things via the configuration server.

Actual Behavior / Motivation for New Feature

Remove a very strong limitation of Ocelot

[raman-m]

Complex Aggregation docs

Ocelot/docs/features/requestaggregation.rst

Lines 69 to 70 in 58d87c9

	Use Complex Aggregation
	-----------------------

• Read the Docs: Use Complex Aggregation

[cfeltz34]

I made a fork allowing me to manage sequential calls and I can see about integrating it into the official version if you want, but you have to be sure that what I develop corresponds to what you want.
In addition, although I have been a developer for 20 years, I have never participated in an open-source project as important as Ocelot and I would not want to take the slightest risk regarding possible regressions.

In theory, I would keep a default operation as it currently exists. But if we activate an option in the config file (sequential = false) then the calls to the micro-services are no longer done automatically. We must then manually call the micro-services in the aggregator as in the following example:

            await Fire(contexts[0], next);
            var response1 = GetResponse(contexts[0]);
            var result1 = await GetResult<ExampleApiMessage>(response1);

            SetQueryString(contexts[1], new QueryString("?id=4&parameter=" + result1.message));
            await Fire(contexts[1], next);
            var response2 = GetResponse(contexts[1]);
            var result2 = await GetResult<IExampleData>(response2);

            var result = new AggregateData(result1, result2);
            return Return(result);

Remark : with sequential aggregations, it seems useful to me to authorize POST type calls (see FileAggregateRoute.cs, UpstreamHttpMethod). But in your code you write : Only supports GET..are you crazy!! POST, PUT WOULD BE CRAZY!! :)

[raman-m]

to manage sequential calls and I can see about integrating it into the official version if you want, but you have to be sure that what I develop corresponds to what you want.

Indeed, "sequential calls" refer to a sequential workflow where each subsequent step is contingent upon the completion of the previous one.

I see limited benefits in designing and implementing a reusable sequential workflow engine in Ocelot for several reasons:

• Sequential calls are typically implemented on the client side, which is the preferred approach.
• Encapsulating downstream services by designing a sequential workflow may seem logical, but it implies implementing business logic on Ocelot's side, which is not advisable.
• Developers can already implement sequential or hybrid workflows using:
  a) Custom delegating handlers
  b) Custom aggregators
  c) The existing Complex Aggregation feature. However, as a team, we have not further developed this feature, which I presume is due to its basic sequential workflow logic.

Please confirm if the Complex Aggregation feature aligns with your requirements.

[raman-m]

In addition, although I have been a developer for 20 years, I have never participated in an open-source project as important as Ocelot and I would not want to take the slightest risk regarding possible regressions.

I am developer with 40 years of experience, man.
You have zero possible regressions because you haven't forked Ocelot repo.

[raman-m]

In theory, I would keep a default operation as it currently exists. But if we activate an option in the config file (sequential = false) then the calls to the micro-services are no longer done automatically. We must then manually call the micro-services in the aggregator as in the following example:

            await Fire(contexts[0], next);
            var response1 = GetResponse(contexts[0]);
            var result1 = await GetResult<ExampleApiMessage>(response1);

            SetQueryString(contexts[1], new QueryString("?id=4&parameter=" + result1.message));
            await Fire(contexts[1], next);
            var response2 = GetResponse(contexts[1]);
            var result2 = await GetResult<IExampleData>(response2);

            var result = new AggregateData(result1, result2);
            return Return(result);

This is a good code snippet for features:

• Complex Aggregation
• Custom Aggregators

Please use them!

[raman-m]

Remark : with sequential aggregations, it seems useful to me to authorize POST type calls (see FileAggregateRoute.cs, UpstreamHttpMethod).

Dear @cfeltz34,
FYI → #1389 the quality of PR is very low. But I keep this PR open for "other verbs" functionality.

But in your code you write : Only supports GET..are you crazy!! POST, PUT WOULD BE CRAZY!! :)

This is not mine, if you are advanced git Blame user.

[raman-m]

My fork is in a private git from Ocelot v17. I need to start over with the current version if I want to add this feature for the current version.

Seems you don't follow our Dev Process (step 2) and we are not interested in any private repos customizations.

The Quote:

User creates a fork and branches from this

[cfeltz34]

Hello and thank you for this very detailed feedback.
The complex aggregation you mention perfectly meets my needs, but it was not mentioned in the v17 documentation.
It is still not very clear to me on the documentation of the current version but from what I read it is exactly what I needed at the time.
Do you have a complete example (config file + C# code if necessary)? I do not see where and how to use AggregateRouteConfig.

[raman-m]

The complex aggregation you mention perfectly meets my needs, but it was not mentioned in the v17 documentation.
It is still not very clear to me on the documentation of the current version but from what I read it is exactly what I needed at the time.

Indeed, this previously undisclosed feature was revealed and documented in the 23.1.0 release on March 4, 2024. It was introduced in commit 36986d6 and further enhanced by PR #1826, which refined the Multiplexer logic. We have completed a feature that is well-suited for building "sequential workflows" based on aggregation.

[raman-m]

Do you have a complete example (config file + C# code if necessary)? I do not see where and how to use AggregateRouteConfig.

No! There are no ready samples too.
@ggnaegi Could you consult the user plz? You have better understanding of this logic.

Ocelot/docs/features/requestaggregation.rst

Lines 69 to 70 in 58d87c9

	Use Complex Aggregation
	-----------------------

I suggest renaming this section in the document to "Building Sequential Workflows" since "Complex Aggregation" seems too abstract. However, does the concept of complex aggregation extend beyond the scope of sequential workflows?

[raman-m]

@cfeltz34 What's your full name? I prefer not to engage in conversations with anonymous users.

[cfeltz34]

My name is Christophe Feltz :-)

[cfeltz34]

@raman-m FYI → #1389 the quality of PR is very low. But I keep this PR open for "other verbs" functionality.

Allow POST on FileAggregateRoute would be great :-)