Combine test_framework and add README and guide for integration tests #360

YJDoc2 · 2021-10-04T13:41:55Z

This moves test_framework to the youki_intergation_tests folder.
This also adds a guide for test writers as a help to add integration tests.

… of its workspace

YJDoc2 · 2021-10-04T13:53:14Z

I have also created #361 which is a tracking issue for rest of the test.

utam0k · 2021-10-05T08:07:39Z

youki_integration_test/Cargo.toml

 [dependencies]
 uuid = "0.8"
 rand = "0.8.0"
 tar = "0.4"
 flate2 = "1.0"
-test_framework = { version = "0.1.0", path = "../test_framework"}
+test_framework = { version = "0.1.0", path = "./test_framework"}


Do we need to specify the version of test_framework?

No, I will remove it in the update

utam0k · 2021-10-05T08:08:26Z

youki_integration_test/README.md

@@ -19,6 +19,65 @@ This provides following commandline options :
 - --runtime (-r) : Required. Takes path of runtime executable to be tested. If the path is not valid, the program exits.
 - --tests (-t) : Optional. Takes list of tests to be run, and runs only those tests. Format for it is : `test-grp-1::test-1,test-2 <space> test-grp-2 <space> test-grp-3::test-3 ...`. The test groups with no specific tests specified, (test-grp-2 in the example) , will run all of its tests, and in other cases, only selected tests will be run. Test groups not mentioned will be ignored.

+## For adding tests
+
+To create and tun tests, we use the custom test_framework, which resides in the youki_integration_test/test_framework/ .


Suggested change

To create and tun tests, we use the custom test_framework, which resides in the youki_integration_test/test_framework/ .

To create and run tests, we use the custom test_framework, which resides in the youki_integration_test/test_framework/ .

utam0k · 2021-10-05T08:10:37Z

youki_integration_test/README.md

+This also has some test utils, meant to help doing common functions in tests, which should be used whenever possible, and should be extended when some common function can be extracted. Some notable function provided are :
+
+- generate_uuid : generates a unique id for the container
+- prepare_bundle : creates a temp directory, and sets up the bundle and default config.json. THis folder is automatically deleted when dropped.


Suggested change

- prepare_bundle : creates a temp directory, and sets up the bundle and default config.json. THis folder is automatically deleted when dropped.

- prepare_bundle : creates a temp directory, and sets up the bundle and default config.json. This folder is automatically deleted when dropped.

utam0k · 2021-10-05T08:11:32Z

youki_integration_test/README.md

+5. Create a function which will create the custom/ default tests, Cerate a TestGroup/custom impl TestableGroup out of it, and return that. Make this function public, and `pub use` it from the mod.rs .
+6. In the src/main.rs, import your function, and run it to get instance of your test group, for example see lines 58-60 ish, where lifecycle, create and huge_tlb structs are created.
+7. Add the reference of this to the test_manager in the main function.
+8. Run you tests individually, run your test group individually and the run the whole suite against youki and some other production level runtime, such as runc. As said before, the important thing is to make sure runc passes it as well.


Suggested change

8. Run you tests individually, run your test group individually and the run the whole suite against youki and some other production level runtime, such as runc. As said before, the important thing is to make sure runc passes it as well.

8. Run your tests individually, run your test group individually and run the whole suite against youki and some other production level runtime, such as runc. As said before, the important thing is to make sure runc passes it as well.

utam0k · 2021-10-05T08:12:02Z

youki_integration_test/README.md

+
+This lists some of the things that can be tricky, and can cause issues in running tests. **In case you encounter something, please update this list**.
+
+- The create commond should always have its stdout and stderror as null, and should always be waited by `wait`, and not `wait_with_output` on it after spawning. The reason is, runtime process forks itself to create the container, and then keeps running to start, get state etc for the container. Thus if we try to `wait_with_output` on it, it hangs until that process keeps running. Trying to kill tests by `Ctrl+c` will cause the system to stay in modified state ( /tmp directories, cgroup directories etc). In case you do this, and need to end tests, open a new terminal and send kill signal to the runtime process, that way it will exit, and tests will continue.


Suggested change

- The create commond should always have its stdout and stderror as null, and should always be waited by `wait`, and not `wait_with_output` on it after spawning. The reason is, runtime process forks itself to create the container, and then keeps running to start, get state etc for the container. Thus if we try to `wait_with_output` on it, it hangs until that process keeps running. Trying to kill tests by `Ctrl+c` will cause the system to stay in modified state ( /tmp directories, cgroup directories etc). In case you do this, and need to end tests, open a new terminal and send kill signal to the runtime process, that way it will exit, and tests will continue.

- The create command should always have its stdout and stderror as null, and should always be waited by `wait`, and not `wait_with_output` on it after spawning. The reason is, runtime process forks itself to create the container, and then keeps running to start, get state etc for the container. Thus if we try to `wait_with_output` on it, it hangs until that process keeps running. Trying to kill tests by `Ctrl+c` will cause the system to stay in modified state ( /tmp directories, cgroup directories etc). In case you do this, and need to end tests, open a new terminal and send kill signal to the runtime process, that way it will exit, and tests will continue.

utam0k · 2021-10-05T08:14:34Z

youki_integration_test/test_framework/README.md

@@ -4,6 +4,8 @@ This is a simple test framework which provides various structs to setup and run

 ## Docs

+One important thins to note here, is that all structs provided by default, TestGroup and TestManager run the individual test cases, and test groups , respectively, in parallel. Also the default Test, ConditionalTest and TestGroup structs are meant for stateless tests. For stateful tests, or for tests which need to be run in serial, please implement respective traits on custom structs.


Suggested change

One important thins to note here, is that all structs provided by default, TestGroup and TestManager run the individual test cases, and test groups , respectively, in parallel. Also the default Test, ConditionalTest and TestGroup structs are meant for stateless tests. For stateful tests, or for tests which need to be run in serial, please implement respective traits on custom structs.

One important thing to note here, is that all structs provided by default, TestGroup and TestManager run the individual test cases, and test groups, respectively, in parallel. Also the default Test, ConditionalTest and TestGroup structs are meant for stateless tests. For stateful tests, or for tests which need to be run in serial, please implement respective traits on custom structs.

YJDoc2 · 2021-10-06T06:22:29Z

Hey, so sorry for all the typos 😓 😓 I have updated the PR

utam0k · 2021-10-06T11:50:34Z

Hey, so sorry for all the typos 😓 😓 I have updated the PR

no problem :)

utam0k

Awesome!

YJDoc2 added 2 commits October 4, 2021 18:11

Move test_framework inside youki_integration_test, making it a member…

ea1b1f4

… of its workspace

Add guide to add tests and minor changes in integration tests names

a3df726

YJDoc2 mentioned this pull request Oct 4, 2021

[Tracking Issue] Implement Integraion Tests #361

Open

60 tasks

YJDoc2 requested a review from utam0k October 4, 2021 13:54

utam0k reviewed Oct 5, 2021

View reviewed changes

Fix Typos in readme

b010b16

utam0k approved these changes Oct 6, 2021

View reviewed changes

utam0k merged commit b5089f3 into youki-dev:main Oct 6, 2021

YJDoc2 deleted the add-test-utils branch November 2, 2021 04:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Combine test_framework and add README and guide for integration tests #360

Combine test_framework and add README and guide for integration tests #360

YJDoc2 commented Oct 4, 2021

YJDoc2 commented Oct 4, 2021

utam0k Oct 5, 2021

YJDoc2 Oct 6, 2021

utam0k Oct 5, 2021

utam0k Oct 5, 2021

utam0k Oct 5, 2021

utam0k Oct 5, 2021

utam0k Oct 5, 2021

YJDoc2 commented Oct 6, 2021

utam0k commented Oct 6, 2021

utam0k left a comment

	To create and tun tests, we use the custom test_framework, which resides in the youki_integration_test/test_framework/ .
	To create and run tests, we use the custom test_framework, which resides in the youki_integration_test/test_framework/ .

	- prepare_bundle : creates a temp directory, and sets up the bundle and default config.json. THis folder is automatically deleted when dropped.
	- prepare_bundle : creates a temp directory, and sets up the bundle and default config.json. This folder is automatically deleted when dropped.

	8. Run you tests individually, run your test group individually and the run the whole suite against youki and some other production level runtime, such as runc. As said before, the important thing is to make sure runc passes it as well.
	8. Run your tests individually, run your test group individually and run the whole suite against youki and some other production level runtime, such as runc. As said before, the important thing is to make sure runc passes it as well.


		This lists some of the things that can be tricky, and can cause issues in running tests. In case you encounter something, please update this list.

		- The create commond should always have its stdout and stderror as null, and should always be waited by `wait`, and not `wait_with_output` on it after spawning. The reason is, runtime process forks itself to create the container, and then keeps running to start, get state etc for the container. Thus if we try to `wait_with_output` on it, it hangs until that process keeps running. Trying to kill tests by `Ctrl+c` will cause the system to stay in modified state ( /tmp directories, cgroup directories etc). In case you do this, and need to end tests, open a new terminal and send kill signal to the runtime process, that way it will exit, and tests will continue.

		@@ -4,6 +4,8 @@ This is a simple test framework which provides various structs to setup and run

		## Docs

		One important thins to note here, is that all structs provided by default, TestGroup and TestManager run the individual test cases, and test groups , respectively, in parallel. Also the default Test, ConditionalTest and TestGroup structs are meant for stateless tests. For stateful tests, or for tests which need to be run in serial, please implement respective traits on custom structs.

	One important thins to note here, is that all structs provided by default, TestGroup and TestManager run the individual test cases, and test groups , respectively, in parallel. Also the default Test, ConditionalTest and TestGroup structs are meant for stateless tests. For stateful tests, or for tests which need to be run in serial, please implement respective traits on custom structs.
	One important thing to note here, is that all structs provided by default, TestGroup and TestManager run the individual test cases, and test groups, respectively, in parallel. Also the default Test, ConditionalTest and TestGroup structs are meant for stateless tests. For stateful tests, or for tests which need to be run in serial, please implement respective traits on custom structs.

Combine test_framework and add README and guide for integration tests #360

Combine test_framework and add README and guide for integration tests #360

Conversation

YJDoc2 commented Oct 4, 2021

YJDoc2 commented Oct 4, 2021

utam0k Oct 5, 2021

Choose a reason for hiding this comment

YJDoc2 Oct 6, 2021

Choose a reason for hiding this comment

utam0k Oct 5, 2021

Choose a reason for hiding this comment

utam0k Oct 5, 2021

Choose a reason for hiding this comment

utam0k Oct 5, 2021

Choose a reason for hiding this comment

utam0k Oct 5, 2021

Choose a reason for hiding this comment

utam0k Oct 5, 2021

Choose a reason for hiding this comment

YJDoc2 commented Oct 6, 2021

utam0k commented Oct 6, 2021

utam0k left a comment

Choose a reason for hiding this comment