Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix part of #5343: Enable Code Coverage Analysis for many to one test target files #5459

Merged
merged 132 commits into from
Aug 7, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
132 commits
Select commit Hold shift + click to select a range
d1c07cd
Code Coverage M1 edge case, added arg names to specifically pick the …
Rd4dev Jun 28, 2024
b26ac35
Modified script example doc and updated the exception string to be mo…
Rd4dev Jun 28, 2024
b89d9a8
Added tests for edge case checks
Rd4dev Jun 28, 2024
23cba86
Increasing the processTimeout to 10 minutes as coverage analysis take…
Rd4dev Jun 28, 2024
e24578e
Setting processTimeout in the run coverage script to see if this affe…
Rd4dev Jun 28, 2024
96c2b74
Set processTimeout for added test cases
Rd4dev Jun 28, 2024
349be2e
Triggering Build to check TIMEOUT
Rd4dev Jun 29, 2024
300fb90
Reverting the changes just to check if these changes have any effect …
Rd4dev Jun 29, 2024
181e8b2
Reverting the changes just to check if these changes have any effect …
Rd4dev Jun 29, 2024
87926ad
Re-introducing the changes while removing the ignore case check as we…
Rd4dev Jun 29, 2024
718b065
Re-introducing the test changes with the passed ci check with ignore …
Rd4dev Jun 29, 2024
c15d6e7
Trying out other unit of processTimeout value
Rd4dev Jun 29, 2024
f417bda
Do not merge - Updated WAIT_PROCESS_TIMEOUT_MS to x4
Rd4dev Jun 29, 2024
70378d6
Using a smaller / simpler source and test file to check if this still…
Rd4dev Jun 29, 2024
9dbd7be
Commenting out 1 of 2 newly added tests
Rd4dev Jun 29, 2024
f59c768
Using a simpler test and source file for sampleTestsDefaultFormat to …
Rd4dev Jun 29, 2024
f5663d7
Using a simpler test and source file for scriptTestsMarkdownFormat to…
Rd4dev Jun 29, 2024
bab336e
Checking if remvoing the 2nd test case solves the timeout to confirm
Rd4dev Jun 29, 2024
2ee1bba
Reverting back to original changes
Rd4dev Jun 29, 2024
23b15b7
Trying to see if moving the declration to setup could reduce load
Rd4dev Jun 29, 2024
6f30dac
Moving scripts source and test files to setup
Rd4dev Jun 29, 2024
b391b1c
Removed any duplicate coverage file creations as they were already in…
Rd4dev Jun 29, 2024
f684951
Almost making compute affected tests execution time to 1 millisecond …
Rd4dev Jun 29, 2024
100d8f1
The actual compute affected test do reflect processTimeout
Rd4dev Jun 29, 2024
4f2d93f
Removing lazy declaration to see if command Executor works that way
Rd4dev Jun 29, 2024
9886da4
Initialize command Executor reference
Rd4dev Jun 29, 2024
4f5d47c
Import CommandExecutor statement added
Rd4dev Jun 29, 2024
69d4583
Added arg names to script calls
Rd4dev Jun 29, 2024
e82f2fd
Trying to make it die in 1L to see if there are any actual reflection…
Rd4dev Jun 29, 2024
a757fd9
Increasing the processTimeout and commandExecutor to high values
Rd4dev Jun 29, 2024
f215839
Setting default and only option as 100L in RunCoverage
Rd4dev Jun 29, 2024
d3105cd
Attempting with main caLL
Rd4dev Jun 29, 2024
f4b305f
Intentionally causing delay in computeAffectedTestsTest to see if pro…
Rd4dev Jun 30, 2024
d78df95
Bumping time to 550 because 350 worked
Rd4dev Jun 30, 2024
c1bfbe1
Completely cleaned up RunCoverageTest for checking with just one test…
Rd4dev Jun 30, 2024
043fcfc
Making RunCoverage default to 0 while passing processTimeout value
Rd4dev Jun 30, 2024
9e3a16a
Intentionally making RunCoverageTest sleep
Rd4dev Jun 30, 2024
f40793c
Last commit overrided the processTimeout with 10 minutes while defaul…
Rd4dev Jun 30, 2024
8d6a10f
Defaulting processTimeout to 1
Rd4dev Jun 30, 2024
1a69432
Reverting all local changes
Rd4dev Jun 30, 2024
9437d67
Adding test timeout, yet this is for failures just trying out
Rd4dev Jun 30, 2024
3787ea9
Intentionally delaying in MavenDependenciesListCheck to see if same e…
Rd4dev Jun 30, 2024
7ad2668
Importing kotlin coroutines
Rd4dev Jun 30, 2024
876152d
Tried delaying in MavenDependenciesCheckTest and that still passes
Rd4dev Jun 30, 2024
0e99588
Testing timeouts
Rd4dev Jun 30, 2024
67e5fb9
Adding withTimeout to see if it can allow the coroutine to run upto t…
Rd4dev Jun 30, 2024
35e79da
Using longCommandExecutor
Rd4dev Jun 30, 2024
2abf259
Adding --test_timeout flag to bazel test run
Rd4dev Jun 30, 2024
3123047
Reverting to old sample test removing fibonacci test
Rd4dev Jun 30, 2024
c6cf052
Fixed the testRunCoverage_invalidFormat_throwsException by updating t…
Rd4dev Jun 30, 2024
f967850
Bumping up the default processTimeout for RunCoverage to 20 MINUTES
Rd4dev Jun 30, 2024
f1f24f0
Fixed ignoreCaseHTMLArgument test case with proper file creation and …
Rd4dev Jul 1, 2024
935662f
Fixed lint checks for max line length
Rd4dev Jul 1, 2024
eab7f1a
Fixed lint checks for max line length
Rd4dev Jul 1, 2024
4526517
Setting a timeout of 600seconds on both bazel tests and processTimeout
Rd4dev Jul 1, 2024
8ec6b7c
Adding --test_timeout to coverage executor
Rd4dev Jul 1, 2024
fe642a6
Reverting the addition of timeout in bazel execute as that wasn't ref…
Rd4dev Jul 1, 2024
f92e821
Remove redundant code and clean up
Rd4dev Jul 1, 2024
7132b7a
Fixed Lint checks on redundant brances and spaces
Rd4dev Jul 1, 2024
5cffdc4
Fixed testRunCoverage_localTestsMarkdownFormat_returnsCoverageData te…
Rd4dev Jul 1, 2024
e38c940
Fix testRunCoverage_ignoreCaseHTMLArgument_returnsCoverageData test w…
Rd4dev Jul 1, 2024
a136d86
Merge branch 'develop' of https://github.com/Rd4dev/oppia-android int…
Rd4dev Jul 1, 2024
073599d
Added print statment for exemption test file scenario
Rd4dev Jul 5, 2024
fecb7d9
Merge branch 'develop' of https://github.com/Rd4dev/oppia-android int…
Rd4dev Jul 5, 2024
f9d1910
Fix Test file exemption test case and simplified print and return sta…
Rd4dev Jul 5, 2024
2e47173
Fix static check - needless blank line
Rd4dev Jul 5, 2024
a724461
Calculate an aggregated coverage report for many : one test target files
Rd4dev Jul 6, 2024
5f19114
Generating reports with aggregated coverage report instead of list of…
Rd4dev Jul 6, 2024
9f9f9c6
Updated tests for multiple test target and restyles/formatted the htm…
Rd4dev Jul 6, 2024
5f9ae7a
Code cleanup - removed debugging statments, kept the mock multi cover…
Rd4dev Jul 6, 2024
45067f7
Code cleanup and fix lint checks for missing spaces
Rd4dev Jul 6, 2024
b3e4c0f
Updated the visibility of the calculateAggregateCoverageReport function
Rd4dev Jul 6, 2024
2bca7e0
Fix Lint check max line lenght exceeded
Rd4dev Jul 6, 2024
c367957
Triggering CI due to error with repo fetching and insufficient write …
Rd4dev Jul 6, 2024
1929c74
Retriggering CI builds as the test pass locally and suspecting any do…
Rd4dev Jul 6, 2024
c5c11f6
Retriggering CI builds :/ just to see if the issue still persists
Rd4dev Jul 7, 2024
2ac7b04
Fixed the asynchronous flow, which was earlier not actually running a…
Rd4dev Jul 7, 2024
5a57677
Fix Lint check, code clean up and removed the static test targets
Rd4dev Jul 7, 2024
a09057b
Added missed Deferred import
Rd4dev Jul 7, 2024
49b8e0f
Moving the coverage analysis to a else condition removing return
Rd4dev Jul 7, 2024
179e8a7
Replaced the TwoSum sample test class name with AddNums
Rd4dev Jul 7, 2024
f82c35c
Fix Lint check max line length exceeded
Rd4dev Jul 7, 2024
c38f2e8
Updated sha hash value as the sample test class name was changed
Rd4dev Jul 7, 2024
a811947
Retriggering CI builds as latest build crashed or didn't report properly
Rd4dev Jul 7, 2024
e784657
Used kotlin idiomatic way to check for requirements and corrected Cam…
Rd4dev Jul 9, 2024
56809bf
Your commit message here
Rd4dev Jul 9, 2024
48c9d1b
Retriggering ci checks to see if the error on downloading remote jdk …
Rd4dev Jul 10, 2024
4c87573
Specifying jdk version for TestBazelWorkspace build to see if this is…
Rd4dev Jul 10, 2024
3d810b2
Update TestBazelWorkspace tests to include jdk version in the bazelrc…
Rd4dev Jul 10, 2024
595e9df
Fix Lint checks on indentation
Rd4dev Jul 10, 2024
4bc533f
Refactored execute's return value to unit
Rd4dev Jul 11, 2024
138a141
Reverting --test_timeout flag to see other possible fixes
Rd4dev Jul 11, 2024
75e28e2
Add size large to RunCoverageTest to set timeout to 900 seconds
Rd4dev Jul 11, 2024
aaf3caf
Resolve merge conflict after test timeout and fetch error fixes in ed…
Rd4dev Jul 12, 2024
d3eb372
Updated processTimeout to set as standard 5 minutes and moved outCont…
Rd4dev Jul 12, 2024
9b5995e
Merge branch 'code_coverage_fix_edge_cases' of https://github.com/opp…
Rd4dev Jul 12, 2024
6710f66
Retriggering since localTestHtmlFormat pass locally and to confirm if…
Rd4dev Jul 12, 2024
756bed8
Adding shard_count to see if that is making any difference
Rd4dev Jul 12, 2024
ab60123
Addition of dividing tests with shards did help with ci checks, now r…
Rd4dev Jul 12, 2024
d25ea9c
Adding back shards since that significantly lessens the test run time…
Rd4dev Jul 12, 2024
8802b27
Merge branch 'code_coverage_fix_edge_cases' of https://github.com/opp…
Rd4dev Jul 12, 2024
66a387e
Merge branch 'develop' of https://github.com/Rd4dev/oppia-android int…
Rd4dev Jul 13, 2024
50574ae
Merge branch 'code_coverage_fix_edge_cases' of https://github.com/opp…
Rd4dev Jul 13, 2024
f170487
Changed base branch to develop in the repo and added reference comment
Rd4dev Jul 13, 2024
8955775
Merge branch 'develop' of https://github.com/Rd4dev/oppia-android int…
Rd4dev Jul 24, 2024
10d9acd
Updated the sharedAndLocalTest case to sync in with updated multi tar…
Rd4dev Jul 24, 2024
be8ff1c
Removing the mock data for multi target aggregation implementation
Rd4dev Jul 26, 2024
7d76b89
Added test cases for coverage data missing and retrieval failure cases
Rd4dev Jul 26, 2024
c2eddfb
Trigger Build since ExplorationActiveTimeControllerTest seems to fail
Rd4dev Jul 26, 2024
ffe7768
Removed asynchronous flow in both source and test files for CoverageR…
Rd4dev Jul 28, 2024
8e88c35
Renaming the runWithCoverageAsync to runCoverageForTestTarget, update…
Rd4dev Jul 28, 2024
c1f012c
Trying to find the coverage path with regex pattern as they mismatch
Rd4dev Jul 29, 2024
aa0de7f
Save lobby before switching to a side branch to test permission error…
Rd4dev Jul 29, 2024
800ccda
Trying to hit a certain line n number of times in different ways yet …
Rd4dev Jul 29, 2024
ad611b8
Reverted changes with sample test data that were added for testing pu…
Rd4dev Jul 29, 2024
84c5336
Reverting the shard count to 4 which was introducing while testing
Rd4dev Jul 29, 2024
5561b4e
Merge branch 'develop' of https://github.com/Rd4dev/oppia-android int…
Rd4dev Jul 29, 2024
07184ee
Lint fixes for exceeding max line length
Rd4dev Jul 29, 2024
3974ce0
Fixes Missing Coverage Data Test Case
Rd4dev Jul 29, 2024
360dc4b
Fixed the retrieve coverage failure test case :) by reproducing the s…
Rd4dev Jul 30, 2024
565f2f5
Clean up and refactored the test cases
Rd4dev Jul 30, 2024
05a007b
Updating the coverage missing test case while removing the job and re…
Rd4dev Jul 31, 2024
a3a661e
Clean up and changed the test case names to be more specific on what …
Rd4dev Jul 31, 2024
8a12cf4
Added validation to ensure exactly one unique value of filePath and f…
Rd4dev Jul 31, 2024
e72179a
Clean up of aggreagation code and replaced conditional error statment…
Rd4dev Jul 31, 2024
f00ebc6
Removed the mock data used for coverage report with n:1 file target a…
Rd4dev Jul 31, 2024
824c406
Reference point for mock data with new repeated bazel test target pro…
Rd4dev Aug 1, 2024
5ee99c0
Clean up and simplification of usage of single usage entry
Rd4dev Aug 1, 2024
372ae95
Fix lint checks and failing test case with update to the proto strucu…
Rd4dev Aug 1, 2024
6499311
Added missing import for BazelTestTarget
Rd4dev Aug 1, 2024
5e528fe
Merge branch 'develop' of https://github.com/Rd4dev/oppia-android int…
Rd4dev Aug 6, 2024
e72eec3
Added the copy of local and shared test to ensure the multiple test f…
Rd4dev Aug 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ class CoverageRunner(
line.substringAfter(":").split(",")
}
}
println("Coverage Data props: $coverageDataProps")
BenHenning marked this conversation as resolved.
Show resolved Hide resolved

val filePath = coverageDataProps["SF"]?.firstOrNull()?.get(0)
requireNotNull(filePath) { "File path not found" }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ fun main(vararg args: String) {

val reportOutputPath = getReportOutputPath(repoRoot, filePath, reportFormat)

if (!File(repoRoot, filePath).exists()) {
error("File doesn't exist: $filePath.")
check(File(repoRoot, filePath).exists()) {
"File doesn't exist: $filePath."
}

ScriptBackgroundCoroutineDispatcher().use { scriptBgDispatcher ->
Expand Down Expand Up @@ -113,8 +113,8 @@ class RunCoverage(
println("This file is exempted from having a test file; skipping coverage check.")
} else {
val testFilePaths = findTestFiles(repoRoot, filePath)
if (testFilePaths.isEmpty()) {
error("No appropriate test file found for $filePath")
check(testFilePaths.isNotEmpty()) {
"No appropriate test file found for $filePath"
}

val testTargets = bazelClient.retrieveBazelTargets(testFilePaths)
Expand Down Expand Up @@ -150,7 +150,15 @@ private fun calculateAggregateCoverageReport(
return coverages.find { it == Coverage.FULL } ?: Coverage.NONE
}

val allCoveredLines = coverageReports.flatMap { it.coveredLineList }
val groupedCoverageReports = coverageReports.groupBy {
Pair(it.filePath, it.fileSha1Hash)
}

val singleCoverageReport = groupedCoverageReports.entries.single()
val (key, reports) = singleCoverageReport
BenHenning marked this conversation as resolved.
Show resolved Hide resolved
val (filePath, fileSha1Hash) = key

val allCoveredLines = reports.flatMap { it.coveredLineList }
val groupedCoveredLines = allCoveredLines.groupBy { it.lineNumber }
val aggregatedCoveredLines = groupedCoveredLines.map { (lineNumber, coveredLines) ->
CoveredLine.newBuilder()
Expand All @@ -161,13 +169,12 @@ private fun calculateAggregateCoverageReport(

val totalLinesFound = aggregatedCoveredLines.size
val totalLinesHit = aggregatedCoveredLines.count { it.coverage == Coverage.FULL }

val aggregatedTargetList = coverageReports.joinToString(separator = ", ") { it.bazelTestTarget }
val aggregatedTargetList = reports.joinToString(separator = ", ") { it.bazelTestTarget }
BenHenning marked this conversation as resolved.
Show resolved Hide resolved

return CoverageReport.newBuilder()
.setBazelTestTarget(aggregatedTargetList)
.setFilePath(coverageReports.first().filePath)
.setFileSha1Hash(coverageReports.first().fileSha1Hash)
.setFilePath(filePath)
.setFileSha1Hash(fileSha1Hash)
.addAllCoveredLine(aggregatedCoveredLines)
.setLinesFound(totalLinesFound)
.setLinesHit(totalLinesHit)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
package org.oppia.android.scripts.coverage

import com.google.common.truth.Truth.assertThat
import kotlinx.coroutines.delay
import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Before
import org.junit.Rule
Expand Down Expand Up @@ -103,7 +100,7 @@ class CoverageRunnerTest {
}

@Test
fun testRetrieveCoverageDataForTestTarget_coverageRetrievalFailed_throwsException() {
fun testRetrieveCoverageDataForTestTarget_withIncorrectPackageStructure_throwsException() {
testBazelWorkspace.initEmptyWorkspace()
testBazelWorkspace.addSourceAndTestFileWithContent(
filename = "AddNums",
Expand All @@ -124,11 +121,7 @@ class CoverageRunnerTest {
}

@Test
fun testRetrieveCoverageDataForTestTarget_coverageDataMissing_throwsException() {
val pattern = Regex(
".*bazel-out/k8-fastbuild/testlogs/coverage/test/java/com/example/AddNumsTest/coverage.dat"
)

fun testRetrieveCoverageDataForTestTarget_withNoDepsToSourceFile_throwsException() {
BenHenning marked this conversation as resolved.
Show resolved Hide resolved
testBazelWorkspace.initEmptyWorkspace()
testBazelWorkspace.addSourceAndTestFileWithContent(
filename = "AddNums",
Expand All @@ -139,30 +132,47 @@ class CoverageRunnerTest {
testSubpackage = "coverage/test/java/com/example"
)

val exception = assertThrows<IllegalArgumentException>() {
runBlocking {
launch {
coverageRunner.retrieveCoverageDataForTestTarget(
"//coverage/test/java/com/example:AddNumsTest"
)
}

launch {
while (true) {
val dir = File(tempFolder.root.absolutePath.substringBeforeLast('/'))
dir.walkTopDown().firstOrNull { file ->
file.isFile && pattern.matches(file.absolutePath)
}?.apply {
if (exists()) {
delete()
createNewFile()
}
writeText("SF: coverage/test/java/com/example/IncorrectCoverageFile.kt")
}
delay(1)
val subTestFile = tempFolder.newFile("coverage/test/java/com/example/SubNumsTest.kt")
subTestFile.writeText(
"""
package com.example

import org.junit.Assert.assertEquals
import org.junit.Test
import com.example.AddNums

class SubNumsTest {

@Test
fun testSubNumbers() {
assertEquals(AddNums.sumNumbers(0, 1), 1)
assertEquals(AddNums.sumNumbers(3, 4), 7)
assertEquals(AddNums.sumNumbers(0, 0), "Both numbers are zero")
}
}
}
""".trimIndent()
)

val testBuildFile = File(tempFolder.root, "coverage/test/java/com/example/BUILD.bazel")
testBuildFile.appendText(
"""
kt_jvm_test(
name = "SubNumsTest",
srcs = ["SubNumsTest.kt"],
deps = [
"//coverage/main/java/com/example:addnums",
"@maven//:junit_junit",
],
visibility = ["//visibility:public"],
test_class = "com.example.SubNumsTest",
)
""".trimIndent()
)

val exception = assertThrows<IllegalArgumentException>() {
coverageRunner.retrieveCoverageDataForTestTarget(
"//coverage/test/java/com/example:SubNumsTest"
)
}

assertThat(exception).hasMessageThat().contains("Coverage data not found")
Expand Down
Loading