Replies: 6 comments 7 replies
-
|
It seems like there is an assumption that we need to start the nodes from within the tests - am I interpreting that correctly? If so, what is the basis for that requirement? |
Beta Was this translation helpful? Give feedback.
-
|
For now, we do a bit of both.
I would say there is an advantage to be launch nodes within a Go test such that we can configure them according to the test. There is rarely a configuration-fits-all for each test. To do that we should focus on fast starts, fast teardowns. |
Beta Was this translation helpful? Give feedback.
-
|
Can you elaborate on some of the test-specific configurations that exist and the range of values they make take? |
Beta Was this translation helpful? Give feedback.
-
|
Not yet really, I'm still working through the code, but I'll comment back. Anyway, on top of that we should clear and restart nodes to clear their state from memory and disk. Again, I'm moving plumbing around in the |
Beta Was this translation helpful? Give feedback.
-
|
If we want to start nodes from a specific state can we use a snapshot of the badger db's state for that? And swap out db files for different test/states, or use init db to clear state. Does docker or ansible provide any advantages regarding this? or different ways of handling state? |
Beta Was this translation helpful? Give feedback.
-
|
@qdm12 or @danforbes do you know what kinda overhead we might expect from setting up ansible? From reading this over it seems to me that using docker would be the way to go, but maybe there is more associated with ongoing maintenance (even tho its still go) than there would be if we used ansible? Or would the maintenance be comparable for the two of them? I guess the question im asking myself is "is the overhead of using ansible worth it to make our lives easier in the future?" Would be curious to hear thoughts |
Beta Was this translation helpful? Give feedback.
-
No overhead really, it just parses a yml and executes things really.
Yes and no. The advantage is Ansible is standardized, but on the other hand, I think we'll have to have some custom code with it as well, so we might as well just orchestrate ourselves (like we do now) using the Docker API (or just To be clear, for now we run nodes as sub-processes in each test. The idea behind running them in docker containers is to test the entire docker image, not just the program. There is also the option to run nodes as goroutines, but that tests less stuff than running the whole program. And all this might be out of scope here as well. |
Beta Was this translation helpful? Give feedback.
-
|
I would also like to point out that we could keep it time based as long as we have proper RPC endpoints and retry mechanisms. That's what I'm working on right now. Obviously event-driven would be better, but it would also require more work. |
Beta Was this translation helpful? Give feedback.
-
|
Is the first option running the node via shell? or instantiating the node directly in a goroutine? |
Beta Was this translation helpful? Give feedback.
-
|
What we do now is we run it via |
Beta Was this translation helpful? Give feedback.
-
|
My concern is that the github runner we is unreliable in terms of the amount of dedicated resources we get. I would assume this is the reason why the run time of these stress tests vary and are inherently flaky. We could optimise all we want in terms of timeouts, but block times can vary due to resource constraints. I think we need to run our own github runners to really achieve the quality and throughput of testing we're looking for. |
Beta Was this translation helpful? Give feedback.
-
Unless it's memory and OOM errors, our test code should scale to the runner performance. And I don't think we actually run out of memory really.
As far as my
A periodic RPC call with retry can make the test wait correctly. It won't be as fast as possible (i.e. like parsing logs/event-driven), but it will be fast enough and reliable.
I disagree, again, unless we're memory bound. There are some tests where we run 9 nodes, maybe that can be problematic, but not as far as I know. Also we should rather split the load on multiple runners instead, ideally, and that's most likely feasible since we already communicate over RPC/network. |
Beta Was this translation helpful? Give feedback.
-
|
@qdm12 asked me to comment about Lodestar QA. So far:
Let me know if more context would be useful |
Beta Was this translation helpful? Give feedback.
-
How do we orchestrate multiple nodes for our integration tests/stress tests?
For now we mostly run other nodes in goroutines with time based synchronization. The problem is this doesn't scale with the machine speed and can be unreliable if the wait time is not long enough (how long is enough?).
The minimal criteria for an eligible solution is to:
The following criteria are considered, with a grade of
-1,0or1:Run each node in a goroutine and parse their logs
Score: 3
1we just need to write logs to a bytes buffer and parse it depending on the test.0we might have to dependency inject loggers1no0each goroutine runs independently, but some code may be shared (:eyes: global variables)1it's all Go code, so it's flexible and fast to maintain.Votes: Quentin
Run each node in a goroutine and signal through channels
Score: -2
-1setup channels and careful async logic (to avoid deadlocks etc.) in tests-1whole services refactor to be able to communicate status through channels1no-1each goroutine communicate with channels so there is a tight/risky coupling0it's all Go code, so it's flexible BUT the asynchronous logic with channels will be tedious to get right without running into deadlocks or race conditions.Votes:
Run each node in a container and orchestrate with Docker Go API
We can use the Docker client Go API to launch nodes containers and stream their logs. This also has the advantage to test against the final Docker image and not just the Go code.
Score: 4
0setup logs parsing and launching/stopping of Gossamer containers with the Docker API1niet0Docker running, although most devs have it I guess1we run different containers so they're quite isolated, this also tests the full Docker image0it's Go code so it's quite flexible, and we have all Docker API at our fingertips. However, since we run a container, we are limited to environment variables/flags to pass to Gossamer + Docker image rebuild.Votes: Quentin
Run nodes with Ansible
Use Ansible to get nodes started with the desired configuration. We can use go-ansible to launch it from our Go test code. This can be nice although might require some work to get the right configuration, and it might be a lot of code duplication. We should also use Go 'scripts' instead of bash scripts for it so it's fully cross platform.
Score: 1
-1setup Ansible files (Go scripts, yml files) + calls from Go code using the Go API1niet0Ansible might be needed (to check)1each program can be run either as simple binaries or as containers, and both should be decently isolated.0It should be nicely flexible once setup properly, but it might lead to a lot of code duplication for each test with may make the whole solution not viable.Beta Was this translation helpful? Give feedback.
All reactions