Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE REQUEST] Use the project's WIKI to share results for each Ruby version with latest CI results #206

Open
arielj opened this issue Jul 11, 2022 · 2 comments

Comments

@arielj
Copy link
Collaborator

arielj commented Jul 11, 2022

Currently, the results are in the README file, but all numbers are from one specific ruby version. Trying to add information for more rubies can make the README a mess and super large to follow.

Another source for the benchmark results is the result of CI runs which shows up-to-date and more rubies, but it's not formatted for readability, it's just the output of the rake command which is not ideal.

Maybe another option is to use the Wiki of the project creating one page with the results for each Ruby that's tested, and then use a github action to update the results there so each page is always automatically updated with the latest results from CI.

I've never done this but I think it's possible, there's this action for example that creates a Wiki page given a markdown file https://github.com/Decathlon/wiki-page-creator-action. So I understand it's possible to create/edit Wiki pages if we can output the benchmark results as a .md file using the github API.

This would help finding benchmarks relevant to the Ruby version the user is using, since different ruby versions may add new optimizations to some of the code variants and old Ruby benchmarks would be inaccurate.

@Joshfindit
Copy link

I like the core concepts here, and think there might be an easier option:

Store the results as .md (and json/yaml) files in the filesystem, with the folder as the Ruby version number.

That would give the following benefits:

  1. The wiki could canonically link to the folder
  2. Since it could be stored as results/version/README.md, github would display the rendered markdown when users navigate through
  3. CI would only need to commit the relevant files
  4. Users could create tools to track differences between Ruby versions

@etagwerker
Copy link
Member

etagwerker commented Jul 11, 2022

We might be able to leverage benchmark.fyi which is an OSS Rails app that stores benchmark-ips results.

I've submitted a PR that shares the results to https://ips.fastruby.io (powered by benchmark.fyi) -- you can see it here: #203

One enhancement that I'd like to add to benchmark-ips and benchmark.fyi is more contextual information:

  • Ruby version
  • OS version
  • Other info about the environment that was running the benchmark code?

Not sure how all of this would tie together, but maybe there could be a wiki page that keeps track of the benchmarks we run and share for this repo?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants