Merge pull request #39 from woocommerce/24-09/custom-e2e-tests-tweaks

Custom E2E docs tweaks
woocommerce · Sep 16, 2024 · 63ad0ca · 63ad0ca
2 parents b8ee369 + ecf56f1
commit 63ad0ca
Show file tree

Hide file tree

Showing 15 changed files with 125 additions and 191 deletions.
diff --git a/docs/custom-tests/00-introduction.md b/docs/custom-tests/00-introduction.md
@@ -1,165 +1,32 @@
 # Introduction
 
 :::info
-The custom tests feature is available as early-access.
+The custom E2E tests feature is available as early access.
 :::
 
-The custom E2E tests allows you to easily write and share E2E tests with other developers.
+Custom E2E tests allow you to generate, write, and run end-to-end tests for WordPress plugins and themes. They aim to eliminate barriers to writing and running E2E tests by providing:
 
-It provides you with a dockerized development environment, and a ready-to-go example test that you can run with one command.
+- A simple CLI tool that generates test scaffolding.
+- A managed test environment for executing your tests.
+- A platform to share your tests with other developers and to run tests from other plugins and themes.
 
-Customize the test to your needs, then share it with other developers, or run it in your CI/CD pipeline.
+## Getting Started
 
-Or, just use one of our pre-built generic tests to get you started. (Coming soon)
+1. **Install the QIT CLI Tool:** See [Installation](cli/01-installation.md).
+2. **Connect Your Account:** Follow the CLI instructions to connect _(you must have at least one extension listed on WooCommerce.com)._
+3. **List Available Extensions:** Check which extensions you have access to test using `qit extensions`.
+4. **Generate Test Scaffolding:** Run `qit scaffold:e2e my-test` to create a basic E2E test structure.
+5. **Run Your First Test:** Execute `qit run:e2e your-extension-slug my-test` (replace "your-extension-slug" with the slug of an extension you own).
+6. **Generate Custom Tests:** See [Generating Tests](01-generating-tests.md) to create your own tests.
 
-## The power of customization: `run:e2e`
+### Additional Features of Custom E2E Tests
 
-While QIT provides various managed tests, custom E2E tests stand out as the sole category where you, the developer, are the architect of the tests. These are not just any tests; they are comprehensive end-to-end tests designed to ensure your plugin's custom behavior functions flawlessly in real-world scenarios.
-
-## Case study: "QIT the Beaver" tackles plugin challenges
-
-Imagine "QIT the Beaver," our beloved mascot, stepping into a developer's shoes. He launches a plugin on the WooCommerce.com Marketplace, and while it initially receives high praise, soon users report issues related to compatibility and conflicts with other plugins and PHP versions.
-
-> ⭐⭐⭐⭐⭐ Works great
-> 
-> ⭐⭐⭐⭐⭐ Simply wonderful
-> 
-> ⭐☆☆☆☆ Doesn't work on PHP 11!
->
-> ⭐⭐⭐⭐⭐ Very helpful!
-> 
-> ⭐☆☆☆☆ Totally broken on PHP 7.2
-> 
-> ⭐☆☆☆☆ Crashed the page when "Cat Pictures" plugin was enabled!
-> 
-> ⭐☆☆☆☆ Broke when I updated to WordPress or WooCommerce
-
-What does _QIT the Beaver_ do? Even though he tries very hard, he can't possibly keep track of all the possible scenarios that users will come up when using his plugin!
-
-To bring some peace of mind, he finally decides it's time to write some end-to-end tests.
-
-### Generating the tests
-
-So he pulls up the QIT CLI and runs:
-
-```qitbash
-qit scaffold:e2e ./e2e
-```
-
-To generate a basic E2E test structure in the `e2e` directory:
-
-```
-qit run:e2e --codegen
-```
-
-To spin up a local test environment and start a [Playwright Codegen](https://playwright.dev/docs/codegen) session, which generates tests by recording his interactions with the browser.
-
-He records some interactions with his plugin, copies the generated code, and pastes it into his test file.
-
-**It's not 100% copy and paste, but after understanding the workflow with codegen, including what he needs to change in the generated code, it turned out to be much easier than he thought!**
-
-Then he runs the tests and sees them running in a browser:
-
-```qitbash
-qit run:e2e qit-the-beaver ./e2e --ui
-```
-
-### Publishing the tests
-
-Satisfied with the results, Beaver decides it's showtime. He uploads the tests to QIT’s platform, a move that simplifies future test runs:
-
-```qitbash
-qit tag:upload qit-the-beaver ./e2e
-```
-
-### Running published tests
-
-Post-upload, running the tests becomes even easier, without needing to specify the path:
-
-```qitbash
-qit run:e2e qit-the-beaver
-```
-
-To ensure his plugin thrives in diverse environments, QIT experiments with different configurations, replicating conditions reported by users:
-
-```qitbash
-qit run:e2e qit-the-beaver \
-            --php_version 8.3 \
-            --wp nightly \
-            --woo nightly \
-            --plugin cat-pictures \
-            --plugin contact-form-7:activate \
-            --theme storefront
-```
-
-This comprehensive command allows him to test with PHP 8.3, the latest WordPress release candidate, the nightly WooCommerce version, and alongside popular plugins and themes.
-
-### Running tests from different plugins
-
-The maker of `Cat Pictures` also uploaded their tests to QIT, which means that QIT the Beaver can integrate the tests from the "Cat Pictures" plugin in his test runs. This is done by passing the plugin slug to the `--plugin` flag:
-
-```qitbash
-qit run:e2e qit-the-beaver --plugin cat-pictures
-```
-
-This will:
-
-- Run the "bootstrap" phase of all plugins, which takes care of all the mocking and setup needed.
-- And the "test phase" of `qit-the-beaver`
-
-This asserts that his plugin continues to work as expected when Cat Pictures is active and fully configured in a site.
-
-Alternatively, he could also run `cat-pictures` test phase, too:
-
-```qitbash
-qit run:e2e qit-the-beaver --plugin cat-pictures:test
-```
-
-With this, it runs the bootstrap and test phases of all plugins, which asserts that not only his plugin continues to work, but that he is also not breaking any expected behavior from others.
-
-With this approach, QIT the Beaver covers a multitude of scenarios, ensuring his plugin performs seamlessly across various setups. It's a win for his peace-of-mind, and most importantly, for his ever-growing base of satisfied customers!
-
-### Publishing a test tag
-
-After his plugin grows and he matures his release strategy, he decides to create a few test tags. This way, he can run different tests for different versions of his plugin.:
-
-```qitbash
-qit tag:upload qit-the-beaver:nightly ./e2e
-qit tag:upload qit-the-beaver:feature-xyz ./e2e/foo-feature
-qit tag:upload qit-the-beaver:fast ./e2e/fast
-```
-
-### Using test tags
-
-Now QIT can use his tags like this:
-
-```qitbash
-qit run:e2e qit-the-beaver nightly,foo-feature
-```
-
-Or if he wants to use a nightly build from a URL:
-
-```qitbash
-qit run:e2e qit-the-beaver nightly,foo-feature --source https://github.com/woocommerce/qit-the-beaver/releases/tag/nightly.zip
-```
-
-Or in a config file:
-
-```yaml
-plugins:
-  - qit-the-beaver:
-        tests:
-            - nightly
-            - foo-feature
-        source: https://github.com/woocommerce/qit-the-beaver/releases/tag/nightly.zip
-```
-
-To wrap up, he adds tagging as part of his GitHub workflows, which fully automates the process for him.
-
-### Using test tags from other plugins
-
-Other plugins can also leverage the test tags `qit-the-beaver` published:
-
-```qitbash
-qit run:e2e cat-pictures --plugin qit-the-beaver:test:fast,foo-feature
-```
+- **Environment Configuration**: Choose different PHP, WordPress, and WooCommerce versions.
+- **Plugin and Theme Compatibility**: Install plugins and themes to check for compatibility issues.
+- **Visual Test Execution**: Use the `--ui` flag to view your tests running in a browser.
+- **Test Generation with Playwright Codegen**: Refer to [Generating Tests](01-generating-tests.md).
+- **Cross-Plugin Testing**: Run tests from other plugins and themes for compatibility.
+- **Test Publishing**: Publish your tests to QIT and run specific tests from a plugin or theme.
+- **Complex Configurations**: Use a config file to run tests with complex setups.
+- **Detailed Test Reports**: After each test, access a shareable URL with results, including screenshots, videos, and failure traces.
+- **Flexible Execution Environments**: Run tests locally or integrate them into your CI pipeline.
diff --git a/docs/custom-tests/01-generating-tests.md b/docs/custom-tests/01-generating-tests.md
@@ -1,7 +1,7 @@
 # Generating tests
 
 :::info
-The custom tests feature is available as early-access.
+The custom E2E tests feature is available as early-access.
 :::
 
 ## Introduction
@@ -18,7 +18,7 @@ This will create a basic E2E test in the `e2e` directory with essentially a `exa
 bootstrap (Optional)
     bootstrap.sh
     bootstrap.php
-    must-use-plugin.php
+    mu-plugin.php
 example.spec.js
 ```
 
@@ -77,15 +77,15 @@ await page.goto('/wp-admin');
 
 ## Using QIT helpers
 
-We have a set of helpers that you can use in your tests to make your life easier. You can find them in the QIT Helpers documentation _(Coming soon)_.
+We have a set of helpers that you can use in your tests to make your life easier. You can find them in the [QIT Helpers](qit-helpers). documentation.
 
 ```js
 // Add this to the top of a test file.
-const qit = require('qitHelpers');
+import qit from '/qitHelpers';
 
 // Example: Login as an admin and navigate to /wp-admin.
-qit.loginAsAdmin();
+await qit.loginAsAdmin();
 
 // Example: Login as the "customer" user.
-qit.loginAs('customer', 'password');
+await qit.loginAs('customer', 'password');
 ```
diff --git a/docs/custom-tests/02-tagging-tests.md b/docs/custom-tests/02-tagging-tests.md
@@ -1,7 +1,7 @@
 # Tagging tests
 
 :::info
-The Custom Tests feature is available as early-access.
+The custom E2E tests feature is available as early-access.
 :::
 
 ## Introduction

diff --git a/docs/custom-tests/03-running-tests.md b/docs/custom-tests/03-running-tests.md
@@ -4,7 +4,7 @@ import TabItem from '@theme/TabItem';
 # Running tests
 
 :::info
-The custom tests feature is available as early-access.
+The custom E2E tests feature is available as early-access.
 :::
 
 ## Running a basic test

diff --git a/docs/custom-tests/04-running-other-plugins-tests.md b/docs/custom-tests/04-running-other-plugins-tests.md
@@ -1,7 +1,7 @@
 # Running tests from other plugins
 
 :::info
-The custom tests feature is available as early-access.
+The custom E2E tests feature is available as early-access.
 :::
 
 ## Introduction

diff --git a/docs/custom-tests/05-bootstrap-and-test-phases.md b/docs/custom-tests/05-bootstrap-and-test-phases.md
@@ -5,7 +5,7 @@ import TabItem from '@theme/TabItem';
 # Bootstrap and test phases
 
 :::info
-The custom tests feature is available as early-access.
+The custom E2E tests feature is available as early-access.
 :::
 
 ## Introduction
@@ -41,7 +41,7 @@ This is a bash script that, if present, will be called. Here you can use `wp` (W
 
 This is a PHP file that, if present, will be called. WordPress is NOT loaded on the context of this file.
 
-### `must-use-plugin.php`
+### `mu-plugin.php`
 
 This is a file that, if present, will be copied to the `wp-content/mu-plugins` directory and will run on ALL requests on the E2E test. WordPress is loaded on the context of this file.
 
@@ -55,7 +55,7 @@ Place them in the bootstrap directory within your E2E test suite. Here’s how y
 bootstrap             (Optional)
   bootstrap.php       (Optional)
   bootstrap.sh        (Optional)
-  must-use-plugin.php (Optional)
+  mu-plugin.php (Optional)
 entrypoint.qit.js     (Optional)
 example.spec.js
 ```
@@ -69,7 +69,7 @@ example.spec.js
 │        ├── bootstrap               # Bootstrap files for setting up tests
 │        │   ├── bootstrap.php       
 │        │   ├── bootstrap.sh        
-│        │   └── must-use-plugin.php
+│        │   └── mu-plugin.php
 │        ├── entrypoint.qit.js       # The Test Phase entrypoint 
 │        └── example.spec.js         # E2E test file
 ├── src                              # QIT Beaver Source code

diff --git a/docs/custom-tests/06-themes.md b/docs/custom-tests/06-themes.md
@@ -1,12 +1,12 @@
 # Themes
 
 :::info
-The custom tests feature is available as early-access.
+The custom E2E tests feature is available as early-access.
 :::
 
 ## Introduction
 
-You can use the Custom Tests to test both plugins and themes.
+You can use the custom E2E tests to test both plugins and themes.
 
 ## Using a specific theme on your tests
 

diff --git a/docs/custom-tests/07-architecture.md b/docs/custom-tests/07-architecture.md
@@ -0,0 +1,7 @@
+import ArchitectureImageURL from '@site/static/img/architecture.png';
+
+# Architecture
+
+This is a basic overview of the custom E2E tests architecture. It focus on the big building blocks that composes the feature.
+
+<img src={ArchitectureImageURL} alt="QIT Custom E2E Tests Architecture"/>
diff --git a/docs/custom-tests/07-security.md → docs/custom-tests/08-security.md b/docs/custom-tests/07-security.md → docs/custom-tests/08-security.md
@@ -1,7 +1,7 @@
-# The security architecture of the custom tests
+# The security architecture of the custom E2E tests
 
 :::info
-The custom tests feature is available as early-access.
+The custom E2E tests feature is available as early-access.
 :::
 
 ## Containerized test environments

diff --git a/docs/custom-tests/09-qit-helpers.md b/docs/custom-tests/09-qit-helpers.md
@@ -0,0 +1,67 @@
+# QIT Helpers
+
+:::info
+The custom E2E tests feature is available as early-access.
+:::
+
+## Introduction
+
+The QIT Helpers is available in the context of Custom E2E tests and provides a set of helper functions.
+
+```js
+import qit from '/qitHelpers';
+```
+
+## Functions
+
+### loginAsAdmin
+
+Logs in as an admin user. Requires the "page" object.
+
+```js
+await qit.loginAsAdmin(page);
+```
+
+### loginAs
+
+Logs in as a specific user. Requires the "page" object, username, and password.
+
+```js
+await qit.loginAs(page, 'username', 'password');
+```
+
+### wp
+
+Executes a WP-CLI command in the PHP container that is executing the test.
+
+```js
+await qit.wp('plugin list');
+```
+
+### attachScreenshot
+
+Attaches a screenshot to the test context. Requires the "page" object and the "testInfo" object.
+
+```js
+const context = {
+    "Some optional context to embed": ["Some Value"]
+};
+
+await qit.attachScreenshot('example-screenshot', context, page, testInfo);
+```
+
+### getEnv
+
+Gets an environment variable. Requires the key of the environment variable.
+
+```js
+const value = qit.getEnv('MY_ENV_VAR');
+```
+
+### setEnv
+
+Sets an environment variable. Requires the key and value of the environment variable.
+
+```js
+    qit.setEnv('MY_ENV_VAR', 'my-value');
+```
diff --git a/docs/custom-tests/_category_.json b/docs/custom-tests/_category_.json
@@ -1,5 +1,5 @@
 {
-    "label": "Custom tests",
+    "label": "Custom E2E tests",
     "position": 4,
     "link": {
         "type": "generated-index"