Merge branch 'develop' into feature/1658---Global-UpstreamHeaderTrans…

…form-settings-in-GlobalConfiguration-section

docs/features/requestaggregation.rst

@@ -13,12 +13,90 @@ We then specify an Aggregate that composes the two Routes using their keys in th
 Obviously you cannot have duplicate **UpstreamPathTemplates** between **Routes** and **Aggregates**.
 You can use all of Ocelot's normal Route options apart from **RequestIdKey** (explained in `gotchas <#gotchas>`_ below).
 
-Advanced Register Your Own Aggregators
---------------------------------------
+Basic Expecting JSON from Downstream Services
+---------------------------------------------
+
+.. code-block:: json
+  {
+    "Routes": [
+      {
+        "UpstreamHttpMethod": [ "Get" ],
+        "UpstreamPathTemplate": "/laura",
+        "DownstreamPathTemplate": "/",
+        "DownstreamScheme": "http",
+        "DownstreamHostAndPorts": [
+          { "Host": "localhost", "Port": 51881 }
+        ],
+        "Key": "Laura"
+      },
+      {
+        "UpstreamHttpMethod": [ "Get" ],
+        "UpstreamPathTemplate": "/tom",
+        "DownstreamPathTemplate": "/",
+        "DownstreamScheme": "http",
+        "DownstreamHostAndPorts": [
+          { "Host": "localhost", "Port": 51882 }
+        ],
+        "Key": "Tom"
+      }
+    ],
+    "Aggregates": [
+      {
+        "UpstreamPathTemplate": "/",
+        "RouteKeys": [
+          "Tom",
+          "Laura"
+        ]
+      }
+    ]
+  }
+
+You can also set **UpstreamHost** and **RouteIsCaseSensitive** in the Aggregate configuration. These behave the same as any other Routes.
+
+If the Route ``/tom`` returned a body of ``{"Age": 19}`` and ``/laura`` returned ``{"Age": 25}``, the the response after aggregation would be as follows:
+
+.. code-block:: json
+
+    {"Tom":{"Age": 19},"Laura":{"Age": 25}}
+
+At the moment the aggregation is very simple. Ocelot just gets the response from your downstream service and sticks it into a JSON dictionary as above.
+With the Route key being the key of the dictionary and the value the response body from your downstream service.
+You can see that the object is just JSON without any pretty spaces etc.
+
+Note, all headers will be lost from the downstream services response.
+
+Ocelot will always return content type ``application/json`` with an aggregate request.
+
+If you downstream services return a `404 Not Found <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404>`_, the aggregate will just return nothing for that downstream service. 
+It will not change the aggregate response into a ``404`` even if all the downstreams return a ``404``.
+
+Use Complex Aggregation
+-----------------------
+
+Imagine you'd like to use aggregated queries, but you don't know all the parameters of your queries. You first need to call an endpoint to obtain the necessary data, for example a user's id, and then return the user's details.
+
+Let's say we have an endpoint that returns a series of comments with references to various users or threads. The author of the comments is referenced by his Id, but you'd like to return all the details about the author.
+
+Here, you could use aggregation to get 1) all the comments, 2) attach the author details. In fact there are 2 endpoints that are called, but for the 2nd, you dynamically replace the user's Id in the route to obtain the details.
+
+In concrete terms:
+
+1) "/Comments" -> contains the authorId property
+2) "/users/{userId}" with {userId} replaced by authorId to obtain the user's details.
+
+This functionality is still in its early stages, but it does allow you to search for data based on an initial request. To perform the mapping, you need to use **AggregateRouteConfig**.
+
+.. code-block:: csharp
+
+    new AggregateRouteConfig{ RouteKey = "UserDetails", JsonPath = "$[*].authorId", Parameter = "userId" };
+
+**RouteKey** is used as a reference for the route, **JsonPath** indicates where the parameter you are interested in is located in the first request response body and **Parameter** tells us that the value for authorId should be used for the request parameter userId.
+
+Register Your Own Aggregators
+-----------------------------
 
 Ocelot started with just the basic request aggregation and since then we have added a more advanced method that let's the user take in the responses from the 
 downstream services and then aggregate them into a response object.
-
 The **ocelot.json** setup is pretty much the same as the basic aggregation approach apart from you need to add an **Aggregator** property like below:
 
 .. code-block:: json
@@ -98,67 +176,39 @@ In order to make an Aggregator you must implement this interface:
     }
 
 With this feature you can pretty much do whatever you want because the ``HttpContext`` objects contain the results of all the aggregate requests.
-Please note, if the ``HttpClient`` throws an exception when making a request to a Route in the aggregate then you will not get a ``HttpContext`` for it, but you would for any that succeed.
-If it does throw an exception, this will be logged.
 
-Basic Expecting JSON from Downstream Services
----------------------------------------------
+Please note, if the ``HttpClient`` throws an exception when making a request to a Route in the aggregate then you will not get a ``HttpContext`` for it, but you would for any that succeed. If it does throw an exception, this will be logged. 
 
-.. code-block:: json
+Below is an example of an aggregator that you could implement for your solution:
+
+.. code-block:: csharp
 
+  public class FakeDefinedAggregator : IDefinedAggregator
   {
-    "Routes": [
+      public async Task<DownstreamResponse> Aggregate(List<HttpContext> responseHttpContexts)
       {
-        "UpstreamHttpMethod": [ "Get" ],
-        "UpstreamPathTemplate": "/laura",
-        "DownstreamPathTemplate": "/",
-        "DownstreamScheme": "http",
-        "DownstreamHostAndPorts": [
-          { "Host": "localhost", "Port": 51881 }
-        ],
-        "Key": "Laura"
-      },
-      {
-        "UpstreamHttpMethod": [ "Get" ],
-        "UpstreamPathTemplate": "/tom",
-        "DownstreamPathTemplate": "/",
-        "DownstreamScheme": "http",
-        "DownstreamHostAndPorts": [
-          { "Host": "localhost", "Port": 51882 }
-        ],
-        "Key": "Tom"
+          // The aggregator gets a list of downstream responses as parameter.
+          // You can now implement your own logic to aggregate the responses (including bodies and headers) from the downstream services
+          var responses = responseHttpContexts.Select(x => x.Items.DownstreamResponse()).ToArray();
+  
+          // In this example we are concatenating the results,
+          // but you could create a more complex construct, up to you.
+          var contentList = new List<string>();
+          foreach (var response in responses)
+          {
+              var content = await response.Content.ReadAsStringAsync();
+              contentList.Add(content);
+          }
+  
+          // The only constraint here: You must return a DownstreamResponse object.
+          return new DownstreamResponse(
+              new StringContent(JsonConvert.SerializeObject(contentList)),
+              HttpStatusCode.OK,
+              responses.SelectMany(x => x.Headers).ToList(),
+              "reason");
       }
-    ],
-    "Aggregates": [
-      {
-        "UpstreamPathTemplate": "/",
-        "RouteKeys": [
-          "Tom",
-          "Laura"
-        ]
-      }
-    ]
   }
 
-You can also set **UpstreamHost** and **RouteIsCaseSensitive** in the Aggregate configuration. These behave the same as any other Routes.
-
-If the Route ``/tom`` returned a body of ``{"Age": 19}`` and ``/laura`` returned ``{"Age": 25}``, the the response after aggregation would be as follows:
-
-.. code-block:: json
-
-    {"Tom":{"Age": 19},"Laura":{"Age": 25}}
-
-At the moment the aggregation is very simple. Ocelot just gets the response from your downstream service and sticks it into a JSON dictionary as above.
-With the Route key being the key of the dictionary and the value the response body from your downstream service.
-You can see that the object is just JSON without any pretty spaces etc.
-
-Note, all headers will be lost from the downstream services response.
-
-Ocelot will always return content type ``application/json`` with an aggregate request.
-
-If you downstream services return a `404 Not Found <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404>`_, the aggregate will just return nothing for that downstream service. 
-It will not change the aggregate response into a ``404`` even if all the downstreams return a ``404``.
-
 Gotchas
 -------
 

src/Ocelot/Configuration/Creator/AggregatesCreator.cs

@@ -24,16 +24,15 @@ private Route SetUpAggregateRoute(IEnumerable<Route> routes, FileAggregateRoute
         {
             var applicableRoutes = new List<DownstreamRoute>();
             var allRoutes = routes.SelectMany(x => x.DownstreamRoute);
-
-            foreach (var routeKey in aggregateRoute.RouteKeys)
-            {
-                var selec = allRoutes.FirstOrDefault(q => q.Key == routeKey);
-                if (selec == null)
-                {
-                    return null;
-                }
-
-                applicableRoutes.Add(selec);
+            var downstreamRoutes = aggregateRoute.RouteKeys.Select(routeKey => allRoutes.FirstOrDefault(q => q.Key == routeKey));
+            foreach (var downstreamRoute in downstreamRoutes)
+            {
+                if (downstreamRoute == null)
+                {
+                    return null;
+                }
+
+                applicableRoutes.Add(downstreamRoute);
             }
 
             var upstreamTemplatePattern = _creator.Create(aggregateRoute);

src/Ocelot/Configuration/Creator/RouteKeyCreator.cs

@@ -1,17 +1,86 @@
 using Ocelot.Configuration.File;
-using Ocelot.LoadBalancer.LoadBalancers;
+using Ocelot.LoadBalancer.LoadBalancers;
 
-namespace Ocelot.Configuration.Creator
-{
-    public class RouteKeyCreator : IRouteKeyCreator
+namespace Ocelot.Configuration.Creator;
+
+public class RouteKeyCreator : IRouteKeyCreator
+{
+    /// <summary>
+    /// Creates the unique <see langword="string"/> key based on the route properties for load balancing etc.
+    /// </summary>
+    /// <remarks>
+    /// Key template:
+    /// <list type="bullet">
+    /// <item>UpstreamHttpMethod|UpstreamPathTemplate|UpstreamHost|DownstreamHostAndPorts|ServiceNamespace|ServiceName|LoadBalancerType|LoadBalancerKey</item>
+    /// </list>
+    /// </remarks>
+    /// <param name="fileRoute">The route object.</param>
+    /// <returns>A <see langword="string"/> object containing the key.</returns>
+    public string Create(FileRoute fileRoute)
     {
-        public string Create(FileRoute fileRoute) => IsStickySession(fileRoute)
-            ? $"{nameof(CookieStickySessions)}:{fileRoute.LoadBalancerOptions.Key}"
-            : $"{fileRoute.UpstreamPathTemplate}|{string.Join(',', fileRoute.UpstreamHttpMethod)}|{string.Join(',', fileRoute.DownstreamHostAndPorts.Select(x => $"{x.Host}:{x.Port}"))}";
+        var isStickySession = fileRoute.LoadBalancerOptions is
+        {
+            Type: nameof(CookieStickySessions),
+            Key.Length: > 0
+        };
+
+        if (isStickySession)
+        {
+            return $"{nameof(CookieStickySessions)}:{fileRoute.LoadBalancerOptions.Key}";
+        }
+
+        var upstreamHttpMethods = Csv(fileRoute.UpstreamHttpMethod);
+        var downstreamHostAndPorts = Csv(fileRoute.DownstreamHostAndPorts.Select(downstream => $"{downstream.Host}:{downstream.Port}"));
+
+        var keyBuilder = new StringBuilder()
+
+            // UpstreamHttpMethod and UpstreamPathTemplate are required
+            .AppendNext(upstreamHttpMethods)
+            .AppendNext(fileRoute.UpstreamPathTemplate)
+
+            // Other properties are optional, replace undefined values with defaults to aid debugging
+            .AppendNext(Coalesce(fileRoute.UpstreamHost, "no-host"))
+
+            .AppendNext(Coalesce(downstreamHostAndPorts, "no-host-and-port"))
+            .AppendNext(Coalesce(fileRoute.ServiceNamespace, "no-svc-ns"))
+            .AppendNext(Coalesce(fileRoute.ServiceName, "no-svc-name"))
+            .AppendNext(Coalesce(fileRoute.LoadBalancerOptions.Type, "no-lb-type"))
+            .AppendNext(Coalesce(fileRoute.LoadBalancerOptions.Key, "no-lb-key"));
 
-        private static bool IsStickySession(FileRoute fileRoute) =>
-            !string.IsNullOrEmpty(fileRoute.LoadBalancerOptions.Type)
-            && !string.IsNullOrEmpty(fileRoute.LoadBalancerOptions.Key)
-            && fileRoute.LoadBalancerOptions.Type == nameof(CookieStickySessions);
-    }
+        return keyBuilder.ToString();
+    }
+
+    /// <summary>
+    /// Helper function to convert multiple strings into a comma-separated string.
+    /// </summary>
+    /// <param name="values">The collection of strings to join by comma separator.</param>
+    /// <returns>A <see langword="string"/> in the comma-separated format.</returns>
+    private static string Csv(IEnumerable<string> values) => string.Join(',', values);
+
+    /// <summary>
+    /// Helper function to return the first non-null-or-whitespace string.
+    /// </summary>
+    /// <param name="first">The 1st string to check.</param>
+    /// <param name="second">The 2nd string to check.</param>
+    /// <returns>A <see langword="string"/> which is not empty.</returns>
+    private static string Coalesce(string first, string second) => string.IsNullOrWhiteSpace(first) ? second : first;
+}
+
+internal static class RouteKeyCreatorHelpers
+{
+    /// <summary>
+    /// Helper function to append a string to the key builder, separated by a pipe.
+    /// </summary>
+    /// <param name="builder">The builder of the key.</param>
+    /// <param name="next">The next word to add.</param>
+    /// <returns>The reference to the builder.</returns>
+    public static StringBuilder AppendNext(this StringBuilder builder, string next)
+    {
+        if (builder.Length > 0)
+        {
+            builder.Append('|');
+        }
+
+        return builder.Append(next);
+    }
 }

src/Ocelot/Errors/OcelotErrorCode.cs

@@ -43,5 +43,6 @@ public enum OcelotErrorCode
         ConnectionToDownstreamServiceError = 38,
         CouldNotFindLoadBalancerCreator = 39,
         ErrorInvokingLoadBalancerCreator = 40,
+        PayloadTooLargeError = 41,
     }
 }

src/Ocelot/LoadBalancer/LoadBalancers/LoadBalancerHouse.cs

@@ -1,5 +1,5 @@
 using Ocelot.Configuration;
-using Ocelot.Responses;
+using Ocelot.Responses;
 
 namespace Ocelot.LoadBalancer.LoadBalancers
 {
@@ -18,45 +18,40 @@ public Response<ILoadBalancer> Get(DownstreamRoute route, ServiceProviderConfigu
         {
             try
             {
-                Response<ILoadBalancer> result;
-
                 if (_loadBalancers.TryGetValue(route.LoadBalancerKey, out var loadBalancer))
-                {
-                    loadBalancer = _loadBalancers[route.LoadBalancerKey];
-
+                {
+                    // TODO Fix ugly reflection issue of dymanic detection in favor of static type property
                     if (route.LoadBalancerOptions.Type != loadBalancer.GetType().Name)
-                    {
-                        result = _factory.Get(route, config);
-                        if (result.IsError)
-                        {
-                            return new ErrorResponse<ILoadBalancer>(result.Errors);
-                        }
-
-                        loadBalancer = result.Data;
-                        AddLoadBalancer(route.LoadBalancerKey, loadBalancer);
+                    {
+                        return GetResponse(route, config);
                     }
 
                     return new OkResponse<ILoadBalancer>(loadBalancer);
                 }
 
-                result = _factory.Get(route, config);
-
-                if (result.IsError)
-                {
-                    return new ErrorResponse<ILoadBalancer>(result.Errors);
-                }
-
-                loadBalancer = result.Data;
-                AddLoadBalancer(route.LoadBalancerKey, loadBalancer);
-                return new OkResponse<ILoadBalancer>(loadBalancer);
+                return GetResponse(route, config);
             }
             catch (Exception ex)
             {
-                return new ErrorResponse<ILoadBalancer>(new List<Errors.Error>
-                {
-                    new UnableToFindLoadBalancerError($"unabe to find load balancer for {route.LoadBalancerKey} exception is {ex}"),
-                });
+                return new ErrorResponse<ILoadBalancer>(
+                [
+                    new UnableToFindLoadBalancerError($"Unable to find load balancer for '{route.LoadBalancerKey}'. Exception: {ex};"),
+                ]);
             }
+        }
+
+        private Response<ILoadBalancer> GetResponse(DownstreamRoute route, ServiceProviderConfiguration config)
+        {
+            var result = _factory.Get(route, config);
+
+            if (result.IsError)
+            {
+                return new ErrorResponse<ILoadBalancer>(result.Errors);
+            }
+
+            var loadBalancer = result.Data;
+            AddLoadBalancer(route.LoadBalancerKey, loadBalancer);
+            return new OkResponse<ILoadBalancer>(loadBalancer);
         }
 
         private void AddLoadBalancer(string key, ILoadBalancer loadBalancer)