Update benchmark for Kestra

[loicmathieu]

Hi,

Disclaimer, I work for Kestra.

The Kestra flow used in this benchmark uses the default Docker task runner, so for each task execution, a Docker container is started, which takes a few hundred milliseconds.
We default to the Docker task runner as it avoids having to install libraries on the host (python interpreter, DBT, gcloud,...) and provides isolation.

But to be able to compare apples to peaches, it would be more fair to use the Process task runner as none of the other competitors in this benchmark executes the task in an isolated container.

So my proposed Kestra flow is the following:

id: benchmark
namespace: company.team
inputs:
 - id: n
   type: INT
 - id: iters
   type: INT
tasks:
  - id: processIterations
    type: io.kestra.plugin.core.flow.ForEach
    values: '{{ range(0, inputs.iters - 1) }}'
    concurrencyLimit: 1
    tasks:
      - id: python
        type: io.kestra.plugin.scripts.python.Script
        containerImage: python:slim
        taskRunner:
          type: io.kestra.plugin.core.runner.Process
        script: |

          def fibo(n: int):
            if n <= 1:
                return n
            else:
                return fibo(n - 1) + fibo(n - 2)

          print(str(fibo({{ inputs.n }})))

In the documentation page, it is said:

We made some adjustments to it to have a similar setup compared to the other orchestrator.

As I don't know exactly what adjustments you did, I can only give you the numbers I have with the updated flow:

• 10 long running tasks didn't see a big difference, it only goes down to a few tens of milliseconds per task
• 40 lightweights tasks, however have a much bigger difference, in your documentation, it took approximately 600ms per task, using the new flow each task took less than 100ms

I do the reproduction on a m5.large machine which replace m4.large, I only update the 40 lightweights tasks for Kestra as on this machine the 10 long running tasks benchmark only shows minor differences. You may want to re-run the benchmark to validate my results.

There are some missing numbers in the conclusion section, as I don't know how you get those percentage, if you give me the formula I could update my PR.

Please ask me any questions or remarks you may have.

[loicmathieu]

For the record, this is the extracted execution result for the 40 lightweights tasks.
bench_10_40.json

[rubenfiszel]

Thanks @loicmathieu , we will re-run the benchmarks with your updated recommendation and share the numbers here as soon as we have them alongside the methodology we used.

[loicmathieu]

@rubenfiszel great! This would ideed be better! Thanks.