feat: add switchboard artillery target #25
Conversation
| @@ -0,0 +1,5 @@ | |||
| SWITCHBOARD_CLIENT_ID="00000000-0000-4000-0000-000000000000" | |||
There was a problem hiding this comment.
Question (not important): Do you intentionally use 4000 in the middle 😆
There was a problem hiding this comment.
Yeah, UUIDv4 always has a 4 at that position
| environments: | ||
| staging: | ||
| target: "https://staging.switchboard.assemble.live" | ||
| phases: |
There was a problem hiding this comment.
Question: Do we want to treat these like unit tests where these tests all need to pass before a production release occurs?
If failing tests are ok, considering you saw good performance up to 500 so far, you could add tests up to 1000? Our Bandwidth rate limit for the registered account is now set to 900, so that feels like a goal to shoot for even if we're not ready to get there yet.
There was a problem hiding this comment.
I like the idea of testing on a kind of log scale beyond our current external rate limits so we know our sending rate is artificially limited by policy rather than software performance.
So like Aashish mentioned I would set phases for like 100, 200, 400, 800, 1600, 3200 to start with, then doing a bit of a binary search to add new phases so we have a good sense of "max possible performance" within 100 req/s or so.
Then we can decide how to do a testing strategy once we know the software perf limits.
There was a problem hiding this comment.
Question: Do we want to treat these like unit tests where these tests all need to pass before a production release occurs?
Treating stress test performance as part of release acceptance seems like a good idea to me but out of scope for this PR.
If failing tests are ok, considering you saw good performance up to 500 so far, you could add tests up to 1000? Our Bandwidth rate limit for the registered account is now set to 900, so that feels like a goal to shoot for even if we're not ready to get there yet.
This is currently blocked by stress test runner limitations. I encountered file descriptor quota exhaustion on Lambda (see 19709a5) and in attempting to fix that ended up down a rabbit hole debugging other AWS runner issues.
| as: "messageCreatedAt" | ||
|
|
||
| # Give process-grey-route-message a chance to work | ||
| - think: 0.2 |
| text: "An SMS from Artillery load test!" | ||
| tag: "v1|{{ messageId }}|{{ messageCreatedAt }}" | ||
|
|
||
| # Receive 'message-delivered' DLR |
There was a problem hiding this comment.
Question: Do we want some think time in btwn here before receiving the 2nd DLR or does it not really make a difference?
There was a problem hiding this comment.
It doesn't make a difference and in practice we have actually seen the second DLR arrive first (that's why there's logic in Spoke for handling DLR looking at previous message status so we don't accidentally go from I can't actually find this logic in Spoke codebase anymore 🤔).DELIVERED to SENT if the sequence gets mixed up
Description
This adds Switchboard as a target along with one Artillery script for the
sendMessageendpoint.Motivation and Context