Deploying dashboard as a part of an existing gh-pages site

[damithc]

The documentation explains how to set up Travis to auto-deploy the dashboard to a gh-pages site. Can we also give an option to add the dashboard as a sub-folder in an existing gh-pages site so that the dashboard doesn't always have to be a separate site?

[niqiukun]

From my understanding, the deployment of the dashboard to an existing gh-pages site largely depends on how the existing gh-pages site is deployed, whether the site content is updated manually or by auto-deploy scripts, and whether the site is deployed from master or gh-pages branch.

In the case where the existing gh-pages site is deployed based on auto-deploy scripts, the situation is a bit complicated since most likely the maintainer of the repo has to edit the existing auto-deploy script to accomodate the update of the dashboard. The exact way to achieve that may largely depend on how the existing deploy script is written and there may not be a solution that fits all situations.

Nevertheless, publishing a static report onto gh-pages site is quite easy, by uploading the entire folder containing the report to the repo. However, since the report cannot get updated automatically, the use case would be quite limited.

[damithc]

In the case where the existing gh-pages site is deployed based on auto-deploy scripts, the situation is a bit complicated since most likely the maintainer of the repo has to edit the existing auto-deploy script to accomodate the update of the dashboard. The exact way to achieve that may largely depend on how the existing deploy script is written and there may not be a solution that fits all situations.

RepoSense provides the script for this (in https://github.com/reposense/publish-RepoSense). Hence, we just need to update that one script to support this possibility.

[niqiukun]

RepoSense provides the script for this (in https://github.com/reposense/publish-RepoSense). Hence, we just need to update that one script to support this possibility.

I mean before adding RepoSense's deployment script, the existing gh-pages site may have its own script to publish its content (for example, using asciidoctor to convert adoc files to html files and publish them, just as AB3). Running RepoSense's script potentially interferes with this.

[damithc]

I mean before adding RepoSense's deployment script, the existing gh-pages site may have its own script to publish its content (for example, using asciidoctor to convert adoc files to html files and publish them, just as AB3). Running RepoSense's script potentially interferes with this.

You are right. Each scripts needs to be aware of the other and work accordingly.
The deploy script of the host site needs to be able to maintain certain sub folders without touching their content (i.e., sub-folders used by sub sites).
The deploy script of the sub site needs to be able to deploy to a designated folder without touching other folders.

[niqiukun]

Hi Prof, after some experimenting I have figured out a way to do this. But I'm not sure in what way should we teach the users to deploy on an existing gh-pages site.

I have edited some scripts in the publish-RepoSense repo so that they can be used to deploy the dashboard on an existing gh-pages site. The edited scripts are placed into publish-reposense folder. The user can attach the folder to any existing repo (in this case, CS3281-2020):

And edit the following line of .travis.yml:
scirpt: (existing scripts...) && ./publish-reposense/travis-deploy.sh

The user can specify the target page by editing publish-reposense/run.sh:

# Specify target page to deploy RepoSense report
TARGET_PAGE="./reposense"

I'm not very sure if we should package the publish-reposense folder as a zip file or add the files to the publish-RepoSense repo that we have currently. May I have some suggestions? Thanks very much!

[damithc]

Good progress @niqiukun
Is it possible to generalize your approach independent of RepoSense. The general problem:

• Use GitHub pages as the deployment target and travis as the CI to publish a site that gets updated from multiple source repos. One source repo will deploy the outer site (e.g., CS3281 site) while a number of other repos will deploy sub-sites (e.g., reposense dashboard), each onto a sub-folder of the outer site.

[niqiukun]

Good progress @niqiukun
Is it possible to generalize your approach independent of RepoSense. The general problem:

• Use GitHub pages as the deployment target and travis as the CI to publish a site that gets updated from multiple source repos. One source repo will deploy the outer site (e.g., CS3281 site) while a number of other repos will deploy sub-sites (e.g., reposense dashboard), each onto a sub-folder of the outer site.

I see. I will look into this. But just to check, there are still multiple repos but the GitHub page is only deployed to the main repo right? If I understand correctly, only the main repo runs CI, and changes made to other repos would not be able to automatically trigger CI of the main repo (but it won't be a problem if deployment is scheduled by cron job).

[damithc]

I see. I will look into this. But just to check, there are still multiple repos but the GitHub page is only deployed to the main repo right?

Yes. But I should be able to update this main site without wiping out the sub-sites.

If I understand correctly, only the main repo runs CI, and changes made to other repos would not be able to automatically trigger CI of the main repo (but it won't be a problem if deployment is scheduled by cron job).

I was thinking both main and sub sites should support cron jobs or auto-deploy site if the main branch is updated.

Example scenario: CS3281 is the main site, but it has a sub-site named dashboard deployed by a repo named dashboard-repo.

• When I push a commit to the CS3281 repo, it triggers Travis, which deploys the site to the gh-pages branch but does not wipe out the existing dashboard sub-site.
• The dashboard repo has a cron job that deploys the latest dashboard to the sub-folder of the main site every 24 hours.

Not sure if we should also consider MarkBind side of things here, given that the outer site is built by MarkBind. Note that MarkBind has its own deploy command. I guess either it has to support deploying as an outer/inner site, or we have to use an alternative way to deploy a MarkBind site.

We can discuss further on Thursday if the above is not clear.

[tlylt]

Hello,

I think making deployment of the dashboard independent of the outer site that is currently utilizing the GitHub Pages may indeed be troublesome due to interference (being overridden by the outer layer static site generator).

From my investigation, I realize that if we can make use of the existing support of the static site generators to include the dashboard without making further unwanted processing of the built dashboard files, then we can deploy the dashboard nicely as a static page, along with the outer site. Basically, I am treating the dashboard as a static asset that the static generators can simply copy and paste over to the build site, and it will automatically be available on the corresponding URL (just like an image or other media assets).

Since we are treating it as a "static asset", most static site generators have this support built-in. Take MarkBind for example, I have created a demo here. To do it automatically, we need to ensure that the report is built in the CI before the static site generates the new site.

Besides the specific details related to MarkBind, I listed two strategies below to deal with deploying the dashboard, using MarkBind as an example.

Question: Do we want to have a short write-up in the UG about this way of integrating into GitHub Pages, along with a static site generator in the same repo?

(Copied from the demo README)

Strategy 1: Generate locally and add to version control

1. Generate the report locally
   1. e.g. java -jar RepoSense.jar --config ./configs
      1. Assuming you have RepoSense.jar downloaded and configs files available in the configs folder
2. Add the report to your version control
3. Deploy locally via markbind build and markbind deploy or let it trigger in your CI scripts after the files have been committed and synced to your repository

Strategy 2: Generate during continuous integration and deploy

1. Setup a CI script to generate the report
   1. e.g. see .github/workflows/ci.yml
      1. Assuming you have included run.sh and get-reposense.py just like this repository and configs files available in the configs folder
      2. Necessary adjustments can be made your CI scripts to suit your needs
2. Within the CI script, deploy the MarkBind Site after the report has been generated

In the ci.yml:

name: CI

on: 
  push:
    branches:
      - main
  schedule:
    - cron:  '0 0 * * *'

Specifies that the CI script should run every day and also on push to the main branch.

jobs:
  deploy:
    runs-on: ubuntu-latest
    env:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-python@v3
      with:
        python-version: "3.8"
    - name: Install dependencies
      run: pip install requests
      shell: bash
    - name: Install Java
      uses: actions/setup-java@v3
      with:
        java-version: '11'
        distribution: 'temurin'
    - name: Generate RepoSense
      run: bash ./run.sh
      shell: bash

The installation and generation of the RepoSense report.

    - uses: actions/setup-node@v2
      with:
        node-version: '12'
    - run: npm i -g markbind-cli
    - run: markbind build
    - run: markbind deploy --ci

Finally, the deployment of the MarkBind site.

[damithc]

Thanks for looking into this @tlylt
I'm not sure if it will work for the use case of deploying code dashboards as part of CS2103 website. This is because CS2103 website is already heavy and the build is slow, I'd rather not add RepoSense report to it.
I think for now we can put this on hold because the the requested feature (I'm the guilty party) is not very sound, in retrospect. This is because the RepoSense report and the enclosing website very likely will have different update frequencies to begin with, and possibly different repositories/maintainers too. I can't also imagine there is a huge benefit of integrating the report to another website. Once the reader navigates to the report, it will look like a different site anyway.
What do you think?

[tlylt]

Thanks for looking into this @tlylt

I'm not sure if it will work for the use case of deploying code dashboards as part of CS2103 website. This is because CS2103 website is already heavy and the build is slow, I'd rather not add RepoSense report to it.

I think for now we can put this on hold because the the requested feature (I'm the guilty party) is not very sound, in retrospect. This is because the RepoSense report and the enclosing website very likely will have different update frequencies to begin with, and possibly different repositories/maintainers too. I can't also imagine there is a huge benefit of integrating the report to another website. Once the reader navigates to the report, it will look like a different site anyway.

What do you think?

That's very true. My suggested integration can't work well if the generation step takes a long time (might still work in CI to just let it run to completion behind the scene, but for larger sites indeed it might be too slow and doesn't make sense to involve the redeployment of the main site for the sake of the report).

As for the navigating to the report part, I agree that the integration only took care of reducing the effort to independently host the report, but not actually making it easy to embed the report into their main site (styling wise). I think this is something that perhaps can be worked on independent of the integration, maybe something like providing an embed service similar to YouTube videos, where it's going to be iframe, but there are some styling / setup done to make the styles look fine when embedding into another site.