Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optic capture in run #2349

Merged
merged 5 commits into from
Sep 26, 2023
Merged

Optic capture in run #2349

merged 5 commits into from
Sep 26, 2023

Conversation

niclim
Copy link
Contributor

@niclim niclim commented Sep 26, 2023

🍗 Description

What does this PR do? Anything folks should know?

Adds optic capture to run. Will run when has capture config, and will skip when not.

Had to log out the capture output rather than providing a summary since capture report data isn't included in the diff report (see below). Could be a lot of data but i'm not sure how else we'd want to do this (without that there isn't a way to be able to understand what failed in CI)

With a capture config

$ optic run --match abe.yml
Optic matched 1 OpenAPI specification file:
abe.yml.

--------------------------------------------------------------------------------------------------

 ┌─────────────────────────────────┐
 │  [1]      Optic Cloud      [2]  │
 └───┬─────────────────────────▲───┘
     │Compare            Update│
 ┌───▼─────────────────────────┴───┐
 │           Local specs           │
 └─────────────────────────────────┘

 [1]: Comparing your local specifications to their last uploaded Optic cloud version for tag `gitbranch:main`.
      Optic compares local specs against the target branch upon PR/MR events, and the current branch upon push events or local runs.

 [2]: Pushing local specs as latest versions in Optic cloud for tag `gitbranch:main`, the current branch tag.

--------------------------------------------------------------------------------------------------

| Untitled service (abe.yml)
| ☑️  No changes ❌ 6 design issues 
| View report: https://app.useoptic.com/organizations/4a1dfbcd-cf7b-4e6d-86f6-e6cc417ffcc0/apis/EMdeSjVDILC4h6j_BYXwc/runs/bghDDg2fa3F_otlVU7D3p
|
| API Test Coverage Report (75.0% coverage)
| GET /authors
|   × 200 response, 404 response
|   [200 response body] schema (/properties/data/items/properties/name/maxLength) with keyword 'maxLength' and parameters {"limit":1} received invalid values "Jane Austen", "F. Scott Fitzgerald", "Harper Lee" and 1 other values
| GET /books/{book}
|   ✓ 404 response
| POST /books/{book}
|   ✓ Request Body, ✓ 404 response
| GET /books
|   ✓ 200 response
| ...and 1 endpoint that did not receive traffic
| 
| 75.0% coverage of your documented operations. All requests matched a documented path (4 total requests)
| 1 diffs detected in documented operations


🤖 Add Optic to your CI flow: https://www.useoptic.com/docs/setup-ci

Exiting with code 1 as errors were found. Disable this behaviour with the `--severity none` option.

Without a capture config

$ optic run --match abe.yml
Optic matched 1 OpenAPI specification file:
abe.yml.
--------------------------------------------------------------------------------------------------

 ┌─────────────────────────────────┐
 │  [1]      Optic Cloud      [2]  │
 └───┬─────────────────────────▲───┘
     │Compare            Update│
 ┌───▼─────────────────────────┴───┐
 │           Local specs           │
 └─────────────────────────────────┘

 [1]: Comparing your local specifications to their last uploaded Optic cloud version for tag `gitbranch:main`.
      Optic compares local specs against the target branch upon PR/MR events, and the current branch upon push events or local runs.

 [2]: Pushing local specs as latest versions in Optic cloud for tag `gitbranch:main`, the current branch tag.

--------------------------------------------------------------------------------------------------

| Untitled service (abe.yml)
| ☑️  No changes ❌ 6 design issues 
| View report: https://app.useoptic.com/organizations/4a1dfbcd-cf7b-4e6d-86f6-e6cc417ffcc0/apis/EMdeSjVDILC4h6j_BYXwc/runs/QwhsNEbVGNEXL0caicKVL
|
| Skipping API test verification (set up by running optic capture init abe.yml)


🤖 Add Optic to your CI flow: https://www.useoptic.com/docs/setup-ci

📚 References

Links to relevant docs (Notion, Twist, GH issues, etc.), if applicable.

👹 QA

How can other humans verify that this PR is correct?

@Alexandre-Herve
Copy link
Contributor

I wonder if logging inline is the right thing to do. If we go in the direction of being a cloud-based CI tool, the logs are really hard to find.

Logging could be a first step, but maybe we should rather upload the capture result somewhere, let people view them in the app and update the comment we post to mention it

@niclim
Copy link
Contributor Author

niclim commented Sep 26, 2023

Yeah I think it could get pretty long, what I was thinking is we could have a verbose mode if you wanted to inspect the logs, but we need to figure out the best way to render capture information as part of a log.

Think we'll use this as a first step and see what we can think of to add a web view for capture

@niclim niclim added this pull request to the merge queue Sep 26, 2023
Merged via the queue into main with commit 454c306 Sep 26, 2023
1 check passed
@niclim niclim deleted the optic-capture-in-run branch September 26, 2023 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants