Load Testing

[base0010]

Goals

• Simplify debugging and readability of load tests
• Make load tests more robust for different scenarios
• Implement features/ improvements w/ Amarok in mind

Motivation/Design Pattern

• Separation of Concerns Across the suite
  ie. SDK Agent and Router Agent shouldn't explicitly know about each other
• test atomicity
• Ability to scale different Agents/Test Tasks from config level up
• Lowest common denominator test logic
• Top Down configurable tests across common dimensions of Time, Iterations and Completedness

Deterministic and Trackable build:

• load tests should come from specific docker images/tags
• create a metrics dashboard (graphana/ to be explored) that shows load test results over time, aggregate test iterations into graphical interface
  ex. I like https://www.srsran.com/performance

Nomenclature:

agent - an actor in the system, router or sdk(client), watchtower[future].

environment - the nxtp services (messaging etc) and external dependencies (chains, providers etc) that the tests run in.

task - a test task that defines some behaviors an agents should take. ex. ping↔pong

results - metrics that describe how the task ran across targets

targets - agents and chains that a task will be executed against to create results

Top Level Environment Configuration from envvars through to Agent specific opts.
needs more color

• Top-level Environment configurations

• CI: SECRETS.FUNDED_KEY

• LOCAL: default 1337 sugar daddy

• Configs to Override parts of the code that cant/shouldn't be tested

  • debug_hardcodeTokenPrice //for chains without oracle

• Individual Configs for Agent Type:

  • Router Agent Config : Omit<config.json>

  {mnemonic, swapAssets, chainIds, sdk & rpc provider(optional)}
  

  • SDK Agent Config:

    {menmonic/pk, derriveFromBaseMnemonic: ? }

Pusedo-code:

EnvironmentSanityCheck(env)

//should throw if any of these fail
{
//simple curls etc. to make sure the messaging auth work
confirmMessaging();
//confirm external providers (subgraph, rpc etc.)
confirmProviders()
}

SetupLoadTestEnv():env

{
//spin up environments based on type and targets

//should happen with the dockerAPI for node.js
//override to test docker funkiness
setupEnvironmentContainers(config.overrideDocker);

EnvironmentSanityCheck();

}

AgentSanityCheck(agents){

//check funds, approves etc.
isSdkAgentSane();
//checks for router ping, confirm random auction response etc.
isRouterSane();
}

SetupAgents(agent type, number){
//create the agents
createAgents()

//fund agents etc.
setupOnChainAgent()

agentSanityCheck();

return agents;
}

gets configs for agents and environment
instantiates and confirms them

SetupLoadTest(){

const targets = getTargets();
const env = getEnv();

//setup environment 

const completedEnv = SetupLoadTestEnv(env);

//setup agents routers, sdk clients etc.
const completedAgents = SetupAgents(env, targets);

//should be setup now
returns(completedEnv, completedAgents);

}

metric callback f()

graphanaPrint(res){
//takes all output from test sanitizes and reports
printPrettyGraph(res);

}

test task callback f()

PingPongTask(targets, taskOptions{duration,iterations}) {

//simple transfer from target chain A to target chain B

let res = [];

for(let i = 0, for i≤ iterations; i++){

 res.push(nxtp.transfer(targets));
}

return res;

}

main fn()

StartLoadTest(taskCallback(env, targets, taskOpts), metricsCallback(res)){

//setup

const [env, targets] = SetupLoadTest();

//execute

await res = taskCallback(env, targets, taskOpts);

//collect results

await metrics = metricsCallback(res);

return metrics;

}

Example Test Run:

...
const taskOpts = {iterations: 100}
const test = StartLoadTest(PingPongTask(taskOpts), graphanaPrint());

[rhlsthrm]

This looks good so far. Missing some of the functionality in this ping pong test:

• Listen for transfer event.
• Send funds back.
  This is likely 1 iteration of the test, and I think this test script functionality should be able to be modified/plugged in so that we can easily adapt this to a completely different SDK, like the upcoming hard fork.

[base0010]

Added the idea of Tasks/tests are a type fo LoadTestBehavior, LoadTestController controls that. LoadTestConfig will be high level what params go into the test (ie iterations etc)