Integrate Gradle with Tacoco

[VijayKrishna]

No description provided.

[VijayKrishna]

@junghuk We might want to create a logical separation between the instrumentation framework and the test-running framework, and the build framework for tacoco. I know that we already have parts of that in place for instrumentation (jacoco) vs. test-runner (tacoco). We might want to create some class(es) to logically identify build-systems as well. Perhaps some documentation or a design spec will clear things up -- feel free to add the design documentation by editing the description of this issue.

[junghuk]

@VijayKrishna I'm thinking about abstract class for TacocoRunner and Implementations like TacocoRunnerForMaven, TacocoRunnerForGradle. For instrumentation, we mention jacoco on runtime and add tacocolistener. I'm not sure for the build framework.

[VijayKrishna]

@junghuk TacocoRunner is really a Junit runner that makes no real assumptions about the build system, or even JaCoCo for that matter. The run_tacoco shell script that we have is really dealing with the build system i.e. Maven, and that is the only point of contact we have with the build system -- where we are simply asking the build system to give us the class paths of all dependencies of the System-Under-Test (SUT) and then we ask for the physical path of the directory of the test classes, which we then feed to TacocoRunner.

I am saying that:

1. We should eliminate the shell script that deals with the build system, and have java code for that which interacts with the build system's apis to do what the shell script is doing now;
2. We should rename TacocoRunner and TacocoListener to JunitTestRunner and JunitListener, like i have done with the Blinky instrumentation framework to see if it makes logical sense: https://github.com/spideruci/blinky/blob/master/src/org/spideruci/analysis/dynamic/JunitRunner.java, https://github.com/spideruci/blinky/blob/master/src/org/spideruci/analysis/dynamic/JunitListener.java
3. Write a TacocoAgent that builds on top of the IAgent interface and Agent class in Jacoco to capture per-test-case coverage that is limited to the source-files that were executed in a test-run and not the entire code-base, thus making the coverage collection for every test-case more efficient (instead of us collecting the coverage data for the entire code-base, for every test-case -- when we know that a test-case would have touched at most one or two source files.)

Doing that will create a clear logical distinction between the parts that are responsible for interacting with the (a) build-system (e.g. Maven, Gradle), (b) the test-system (e.g. JUnit, TestNg) and (c) the instrumentation framework (Jacoco, Blinky).

Please bring up any issues/doubts/concerns that you might have with this proposal. Better yet, if you understand what i am talking about, write up a super brief design spec that details these ideas.

[junghuk]

@VijayKrishna Now I know what is builder. Here's a brief class diagram.

https://repository.genmymodel.com/junghuk/Tacoco

[VijayKrishna]

I would not even call it builder, since it is really not building anything. It is merely using the results of an already successful build, and interfacing with the build system to get the class paths. I would say call it something like AbstractBuildHook or AbstractBuildProbe (i actually like Probe better, because we are "probing" an existing build for test-classes, source-classes and all the class paths), and then make the JunitRunner use the AbstractBuildProbe instead of a AbstracterBuilder using the JunitRunner. Does that make sense? And include the diagram in the description :)

[junghuk]

@VijayKrishna I updated the diagram. Probe is more clear than Builder.
Under this structure, JunitRunner and Listener can be independent from a target's build system.

https://repository.genmymodel.com/junghuk/Tacoco

[VijayKrishna]

Lets call it an AbstractBuildProbe, instead of just AbstractProbe. I hate to suggest these many changes/corrections, but an AbstractProbe could also be misinterpreted as an AbstractInstrumentationProbe, like that used by Jacoco or Blinky. If I am being too obnoxious with these suggestions, you can ignore me as well :D

[junghuk]

Suggestions are always welcome :)

https://repository.genmymodel.com/junghuk/Tacoco

[tariq1890]

@junghuk I am trying to find out how we can extract dependencies from Gradle Project. I am guessing we may need to use the Gradle Custom Model and implement the logic to fetch the project dependencies . Very little documentation on this and the Gradle folks dont seem to reply either. If you have any idea about Custom Model, we can discuss!

[junghuk]

@tariq1890 I already made a small code, dependency extraction from Gradle.
https://github.com/spideruci/tacoco/blob/master/src/main/java/org/spideruci/tacoco/probe/GradleBuildProbe.java#L70

The logic is,

1. inject tacoco task(groovy code) to gradle builde system.
2. run tacoco task then get class path and target dir.

The benefit of this approach is we can get whatever we want using simple groovy coding.
And It is working in my small benchmark. However @tariq1890, you mentioned about GRADLE API to grep dependency. If we can find the API and it supports full functionality, then using API is better solution.

[VijayKrishna]

(ignore this comment if you have already found a working solution with Gradle's API)

Not sure if we have had any breakthroughs yet, but i found the following few blog posts about Gradle's tooling API and classes in the Gradle API quite interesting for this issue.

Blog posts:

• https://docs.gradle.org/current/userguide/embedding.html
• http://gradle.org/feature-spotlight-gradle-tooling-api/
• http://alexruizlog.blogspot.com/2013/06/android-gradle-ide-integration.html

Gradle classes:

• GradleProject: https://docs.gradle.org/current/javadoc/index.html?org/gradle/tooling/model/GradleTask.html
• GradleTask: https://docs.gradle.org/current/javadoc/org/gradle/tooling/model/GradleTask.html
• SourceSet: https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/SourceSet.html

[tariq1890]

Thanks for the update Vijay . I did look at all of these pages during my research and I was unable to use them for achieving tacoco's use case :(.

However I am using existing Project models of Gradle API to extract the dependencies. I will upload a separate repository to demonstrate how it works.

@VijayKrishna @junghuk Let me know if sounds okay to you guys

[tariq1890]

https://github.com/tariq1890/FetchGradleDependencies .

@junghuk @VijayKrishna I have come up with an implementation to fetch the dependencies. Let me know if this logic is okay for implementation in tacoco

[junghuk]

@tariq1890 I think your implementation depends on IDE(eclipse or intellij).

[tariq1890]

Just to be clear, The implementation can be used to fetch dependencies from project which are defined/created in eclpse and IntelliJ iDEA

[junghuk]

@tariq1890 we need IDE independent implementation, because we can not guarantee that a gradle project has IDE dependency. Usually it dose not.

[VijayKrishna]

@tariq1890 can we program a task with the tooling API that does what @junghuk's current solution is doing? Specially, I am wondering if creating a programatic equivalent of the following task is possible:

task tacoco {
    File td = new File('tacoco.testDir')
    td<<sourceSets.test.output.classesDir
    File cp = new File('tacoco.cp')
    sourceSets.main.runtimeClasspath.each { cp<<it<<':'}
    cp<<sourceSets.test.output.classesDir
}

If it is (and I am not asking you to create it), then could we somehow launch that task with BuildLancher and specifically, BuildLauncher#forTasks?

This way we do not have to deal with the API beyond the point of creating and running some arbitrary task against the target build, and at the same time avoid re-writing the build script like what @junghuk's solution is currently doing.

This can be a stop gap measure until we can better understand Gradle's tooling API.

[tariq1890]

@VijayKrishna When I started out, I was trying to achieve the very same thing.(I had even browsed all the pages you mentioned earlier) The programmatic version of the task is the definitely the best approach.

But I have not found any possible way to create this task programmatically.( and also from an outside project that would probe the Gradle project).

Moreover creating a Programmatic version of the task requires the usage of classes like SourceSet, Gradle Task etc which are core Gradle API classes. I have tried importing the gradle core classes through the pom, but I get NoClassDefFound Errors. Turns out that importing Gradle internal jars are discouraged in Non-gradle projects. Should you need to access these classes we would need to add the line compileApi() in build.gradle and that would mean migrating tacoco to gradle.

I have posted multiple questions in the Gradle forums and StackOverflow to get a lead on creating the task programmatically, but I am yet to receive any replies. Which is why I resorted to this approach. Frankly, I am not too happy with this approach either, but at least, it is not intrusive.

Thoughts?

[junghuk]

@tariq1890, did you check this api,
https://docs.gradle.org/current/javadoc/index.html?org/gradle/api/tasks/SourceSet.html

it seems it has almost everything we need.

[tariq1890]

@junghuk I did. SourceSet is part of the Gradle core jar. Importing this jar caused a lot of side-effects for me.
When I checked the Gradle forum posts, It is recommended to use these classes with the compile gradleApi() statement in build.gradle. If we can migrate tacoco from Maven to Gradle, we could explore these possibilities.

[VijayKrishna]

can we get away with a build.grade with just that statement in it, while continuing to use maven for the project's build management?

[tariq1890]

Hmmmm. let me try that :) .

[tariq1890]

@VijayKrishna I tried. No luck with that. The thing is, for our build.gradle to have a tangible effect, the project has to be imported as a gradle project. :(

[tariq1890]

Are you guys okay with my approach to fetch dependencies?

[VijayKrishna]

Not sure what @junghuk has to say, but i am not excited about the IDE aware approach. The whole point of this project is to interface with build and test systems to execute and analyze software systems -- and builds and test-runs are IDE-agnostic. There must some way of building against Gradle's core api, or to run some of Gradles commands from Java code using an embedded version of Gradle like you did with Maven.

[tariq1890]

I agree Vijay. One approach would be to create a separate gradle project that would use the gradle API and create custom models / methods to run Gradle commands programmatically. We can then generate a jar from this project and add it to tacoco.

[VijayKrishna]

That sounds like a reasonable approach to try out. Can you prototype it, then perhaps Junghun can review it?

[junghuk]

@tariq1890 Creating a custom model seems good idea. I'll review that.

If it doesn't work, I think we still use hybrid approach.

1. add tacoco task using groovy code, not tooling api.
2. run tacoco task using tooling API.

[tariq1890]

Unfortunately there is no possible way to programatically create tasks in an external project and execute them on target Gradle projects. Objects like SourceSet,Configurations reside in the JVM of the gradle project and we cannot retrieve it in tacoco.

Also the Gradle team is yet to create an IDE agnostic custom model to fetch dependencies. We would need to contribute to the Gradle Project if we need it ASAP :).

@junghuk I guess the hybrid approach is the best as of now

[tariq1890]

Latest Update!

@VijayKrishna @junghuk . The EclipseModels and IdeaProjectModels are IDE agnostic as far as retrieving basic Gradle build information is concerned. I have confirmed after testing the retrieval of Gradle dependencies using the Eclipse Project model in the following scenarios

1. IntelliJ Idea Projects
2. Eclipse Projects
3. Projects which have not been imported in any IDE.

Also Please check this link out, which confirms the same.

https://discuss.gradle.org/t/custom-gradle-model-which-exposes-a-method-to-retrieve-all-project-dependency-jars/13179

I have also updated my FetchGradleDependencies project (https://github.com/tariq1890/FetchGradleDependencies) to use the EclipseModel only and it is ready for review! Let me know your opinions.

[junghuk]

@tariq1890 Great! It works. We can use tooling API to implement the probe.
(But I'm still not clear why the name "EclipseProject.class" is not rely on Eclipse.)

When you implement the probe, I ask following things.

1. consider multi module gradle project.
2. create sample gradle project(like spiderMath) and
3. make JUnit test cases.

Have a fun @tariq1890.

[tariq1890]

@junghuk Sure . I will take care of those . and yes, I agree with you on the EclipseProject naming. It is very misleading.

What did you mean by spiderMath?

[junghuk]

@tariq1890 spiderMath is currently using maven,
https://github.com/spideruci/tacoco/tree/master/resources/spiderMath_TestNG

you may create one using Gradle.

[tariq1890]

Got it! Will do :)

[VijayKrishna]

While testing you might want to make sure that there are no .classpath or .iml or any other hidden files in the Gradle project that you will use to test this integration.

[tariq1890]

@VijayKrishna I have tested that scenario as well. EclipseProject model even works or a project that has only a buid.gradle file.

[tariq1890]

@junghuk Thanks for accepting the pull request.

Let me know if the implementation is okay. If I may have missed anything, please let me know. Also if there are no concerns, shall we close this issue?