Report Component

[Luolc]

Taken from #2
We will need report component, which runs the configuration(s) and pushes the reports to a server. We should try to re-use the groovy script in contribution repo.

[Luolc]

We should try to re-use the groovy script in contribution repo

I am thinking the way of "re-use". Via intuition, we could clone the contribution repo and run the groovy command line. Or, we could use the groovy source in some way, but it is a problem that how to maintain the groovy file to keep updating with the latest version in contribution repo.

@rnveach What't your suggestion?

[rnveach]

To test the waters, we could just do a clone and see how it works out. Any 'issues' with this way can be made into a new issue so users are aware.

If there are no issues working with the clone, we could relocate the groovy scripts to this repo as we are both aiming for the same goal, regression testing. I see no reason right now why it shouldn't be in our repo instead.

We will face one problem with this component talked about at #5 (comment) when we have multiple instances of the same module to test on. Just keep it in mind when starting this component.

[Luolc]

I am meeting a lot of questions when working on this. I decide to push the changes step by step and confirm the unclear things in the meantime.

1. listOfProjects. The regression needs repositories of a lot of projects. It would be really time-consuming if we clone them every time we run it on Travis. And these repos tend to be consistent. Is it a good idea to add these repos in our regression-tool project directly?(I don't like this actually, but it seems the easiest way to avoid cloning every time) Or using some cache tech on Travis? I am not familiar with it and don't know whether it could work. There is another question that what projects to use for regression, I know the more the better, but time is a factor to be concerned about.
2. diff.groovy. Currently I wrote some code to download it from remote repo. Well, actually it is not so different with putting it in our project directly. If we put it in our project, I think we may need a test to insure that it is the latest version comparing with the one on GitHub. There are many ways to run groovy in Java. By now the most easy-to-use way I found is GroovyShell. Do you have any suggestion on the way of running the groovy? FYI: http://groovy-lang.org/integrating.html
3. configs. Is it possible for us to use different configs for master and PR branch in regression-tool?
4. I think we may need git component to detect change type. Since we may use single mode when the module is new.

[rnveach]

It would be really time-consuming if we clone them every time we run it on Travis.
Or using some cache tech on Travis?

We have no choice on travis. We can't store everything into a cache and we are given new/different machines on each run.
CI will only run one project in each node, so CI shouldn't face a huge time issue. Only local users.

Is it a good idea to add these repos in our regression-tool project directly?
what projects to use for regression, I know the more the better, but time is a factor to be concerned about.

We will not be storing project's .git folders. We will only have metadata on where to find project and what commit to use. These projects will be cached for local users so it is only a one time clone. We almost always require users to use all projects, so we can't get around time issue. Multi-threaded mode might alleviate time issue for us.

Do you have any suggestion on the way of running the groovy?

I am not familar with groovy or how to run it from Java, so whatever you find is fine with me.
As a last resort, you can just run a command line to execute it.

If we put it in our project, I think we may need a test to insure that it is the latest version comparing with the one on GitHub.

Lets go with the idea that we will move it and the property files out of old repo and place it in ours for now. I am confirming with @romani if this is ok. It seems the most logical to me as we are now the authority on regression.
With it in our repo, users can customize projects to run anyway they want, if we will allow this.

Is it possible for us to use different configs for master and PR branch in regression-tool?

It is rare that different configs are needed. If they are how would we connect the results between the 2 as we have multiple module instances in one configuration? When would 2 configurations be different? On a new property that is not in master? Will master config have default module and PR config have module with property on?
We can make this into a new issue as we may need to look into it more. For now, let's assume configs will be the same.

I think we may need git component to detect change type. Since we may use single mode when the module is new.

If module is new, we can only do regression in single mode with PR config.
Properties is issue #5 so we can hold off on that. If there is a new property, than like I discussed above, we could have different configurations.
We can make new properties a separate issue from #5 and only handle existing properties in current issue.