GSoC: New Bowtie Test Case Widgets

[Julian]

Project title

New Bowtie Test Case Widgets

Brief Description

Bowtie is a JSON Schema tool for comparing implementations with each other, and reporting on when or whether they have behavior which does not match with behavior expected by the JSON Schema specification.

It renders this information on a website at bowtie.report.

A major part of showing this information is communicating:

• a specific JSON Schema
• a specific JSON instance (piece of data)
• the expected result for whether this schema considers the instance valid
• the actual result received from 15 or more implementations which were tested
• a description of this specific test case suitable for human readers

How to present these 5 bits of information (and a few more) can get challenging when showing a large number of test cases.

Bowtie's website currently contains a crude pair of UI widgets for doing this -- specifically it renders a long list of collapsible sections:

and then when expanding one, shows the schema, instance, and results in a table:

But we can do lots better! Help rewrite Bowtie's test UI components so that we can cleanly filter through hundreds or thousands of test cases in a clear and efficient way.

Expected Outcomes

• New UI components for test case lists as well as instance/schema pairs
• Integration of these components with Bowtie's site

Skills Required

• React / TypeScript expertise
• An eye for design
• Basic knowledge of JSON Schema

Mentors

@Julian

Expected Difficulty
Medium

Expected Time Commitment
175 Hour

[SimonDMC]

Hello! I have a few ideas for this project and I'd love to discuss potentially working on it as a part of GSoC, but it seems that the slack link is broken. Here's the error I'm getting:

[Siddharth-Singh-2004]

A few questions from my side to understand this requirement better:

1. Will UI developed by this project be net new completely replacing the existing UI or should we focus only on the parts highlighted in the project description, which is mainly about making it easy to navigate through the test case results?
2. Dialect Reports, for eg., draft2020-12.json: should we expect a change in format of the report as a part of any other project running in parallel with this project ? If yes, where may I access the details of the new format?
3. Would we like the UI to render differently for implementers vs maintainers (org/providers)? Implementers in the dev-test-fix cycle have a different set of requirements as compared to maintainers or decision makers.
4. There is another requirement on bowtie long term trend reporting GSoC: bowtie-trend: Long-Term Reporting With Bowtie #607. Are we looking at persisting data in a datastore (separate table for each type of data) as a part of that requirement or would we be working by searching through files? If yes, then should this project also work off of the latest results from the data store rather than a file, except when the user chooses to work of local file during dev-test-fix cycle?
5. Are we expecting the work from GSoC: bowtie-perf: a Performance Tester for JSON Schema implementations #605 and GSoC: bowtie-trend: Long-Term Reporting With Bowtie #607 getting linked from this report so implementers have one place to look at trends and performance metrics? Are we expecting deeper integration (components to be integrated) into this project's UI in a composable dashboard or will those reports have dashboards which will be linked from this project's UI?
6. Will there be need to support sign-in and role based access control tying it back to point (1) above (different UI based on role)?
7. If we do point (6) above, then support Profile based personalisation, for eg: pick and choose schemas I'm interested in, and not be shown all of the supported schemas.

[Julian]

Will UI developed by this project be net new completely replacing the existing UI or should we focus only on the parts highlighted in the project description, which is mainly about making it easy to navigate through the test case results?

Always focus on what an issue says :)

Dialect Reports, for eg., draft2020-12.json: should we expect a change in format of the report as a part of any other project running in parallel with this project ? If yes, where may I access the details of the new format?

Maybe, but if so the UI parsing code would be updated to handle it.

Would we like the UI to render differently for implementers vs maintainers (org/providers)? Implementers in the dev-test-fix cycle have a different set of requirements as compared to maintainers or decision makers.

Sounds like potentially a nice set of functionality -- but not very related to this issue, which is about "what's the clearest possible way to present the test result information". Unless you think the answer even for that question differs for these two groups (I don't think I do).

Are we looking at persisting data in a datastore (separate table for each type of data) as a part of that requirement or would we be working by searching through files?

Considering we're still using GH Pages, likely not, just files. Perhaps someday.

Are we expecting deeper integration (components to be integrated) into this project's UI in a composable dashboard or will those reports have dashboards which will be linked from this project's UI?

A dashboard and integration certainly sounds nice -- keep in mind though it's not necessarily the case that both projects will go forward, we have a limited number of mentors!

Will there be need to support sign-in and role based access control tying it back to point (1) above (different UI based on role)?

No.

[skushagra9]

I have a good experience with open source
Hey, I am interested to take this issue, should I start contributing?

[AgniveshChaubey]

Hey @skushagra9, since this is a project idea for GSoC and not just a project issue, at present maintainers cannot assign this task specifically to someone. However, you can start contributing to this issue, as early contributions definitely serve as a strong point in your proposal. :)

[skushagra9]

okay, got it

[Swarnendu0123]

Hey @Julian, I'm thinking about an idea. Could we implement a component similar to GitHub-style progress bar to indicate whether tests have passed or not?

We could use green for passed and red for failed. Additionally, users could hover over each box to view the details of the specific tests.

on click also we can design an additional page to show what are the exact points the the specific test is failing/passing.

I am Swarnendu, a student from India, and I have a vast experience in React-TypeScript, and Styling, can I do it as my GSoC project, maybe we can discuss and generate a better idea?

[Julian]

Hi, welcome!
Seems like a nice idea, though that component in GitHub is used to communicate progress over time right? Perhaps that makes that kind of idea a better fit for #607, since otherwise I'm not sure I fully follow what the X and Y axis will represent. But maybe it relates to both!

[Swarnendu0123]

since otherwise I'm not sure I fully follow what the X and Y axis will represent. But maybe it relates to both!

Can we just categories our tests, if we can, then we can solve that issue?

Perhaps that makes that kind of idea a better fit for #607

I have another improved idea for #607: when dealing with large volumes of data, it's beneficial to visualize the information using diagrams such as graphs. We can utilize a graph similar to this:

[AgniveshChaubey]

Adding a graphical representation of tests seems like a great idea to me. How about representing the number of tests on the Y axis and implementation test results on X-axis?

Also grouping the implementation graphs of same the language would make it easier for users to select the best-performing implementation for their use.

If bar chart is used, then maybe we can use the bottom bar to represent the number of passing tests by that implementation. Then we could add bar on top of the base one to show the skipped tests (if any) and the top one could show the number of failing tests. (Maybe we categorize them with colors as well for better visual representation--- green for passing, gray or yellow for skipped and red for failing)

This is the quickest ugliest representation I could come up with for what I said above:

Since there are a lot of implementations registered to Bowie, the graphs could possibly overflow the parent container, so adding a horizontal scroll bar might be a good option.

@Julian what's your comment on this? If Julian agrees then @Swarnendu0123, you might find this as a good start for your further progress.

[Swarnendu0123]

Loved your idea @AgniveshChaubey! And Thanks for the demonstration.

What are your views on it @Julian ?

[Julian]

Now that definitely is the kind of idea this project was about yes! I had that idea in a slightly different way, where we'd show them as stacked horizontal bars, because that makes it easier to sort visually by compliance percentage ("fraction of green"). But yes I agree that's exactly the kind of thing we should think about for this issue!

[benjagm]

Thanks a lot for joining JSON Schema org for this edition of GSoC!!

Qualification tasks will be published as comments in the project ideas by Thursday/Friday of this week. In addition I'd like to invite you to a office hours session this thursday 18:30 UTC where we'll present the ideas and the relevant date to consider at this stage of the program.

Please use this link to join the session:
🌐 Zoom
📅 20124-02-29 18:30 UTC

See you there!