azalea/kotlin-fork
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[BTA tests] Improve README.md

Browse Source 
^KT-61860 In Progress
This commit is contained in:
Alexander.Likhachev
2024-03-06 00:11:15 +01:00
committed by Space Team
parent ea3d5371ef
commit f319ff2c3c
13 changed files with 69 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
compiler/build-tools/kotlin-build-tools-api-tests/README.md
+32 -23
View File
@@ -1,44 +1,53 @@
## Build Tools API integration tests
This module contains integration tests covering the build tools API implementation using the DSL
This module contains integration tests covering the build tools API implementation using the DSL
built on top of the [Build Tools API](../kotlin-build-tools-api/README.md).
#### How to run
To run all tests for all Gradle plugins use `check` task.
To run all tests use the `check` task.
The module defines test matrix using the `jvm-test-suite` plugin to cover different combinations of
the Build Tools API's and its implementations' versions.
The module defines test suites using the `jvm-test-suite` plugin.
#### Test suites:
* `testSnapshotToSnapshot`: runs all tests against the API and implementation of the snapshot version.
* Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testSnapshotToSnapshot` to run them
* `test1.9.20ToSnapshot`: runs all the tests marked with the `CompatibilityTests` annotation against API 1.9.20 and the snapshot implementation
* Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:test1.9.20ToSnapshot` to run them
* `testSnapshotTo1.9.20`: runs all the tests marked with the `CompatibilityTests` annotation against the snapshot API and 1.9.20 implementation
* Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testSnapshotTo1.9.20` to run them
* Compatibility: a special test suit that runs against a set of implementation versions
* Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testCompatibility1.9.20`
to run the tests against BTA implementation 1.9.20
* Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testCompatibilitySnapshot`
to run the tests against the current BTA implementation
* Avoid adding new tests here unless you can articulate their necessity as they will be executed multiple times significantly increasing
the overall test execution time.
* Example: provides examples of the DSL usage. Excluded from the `check` task
* Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testExample` to run them
#### How to work with the tests
Few rules you should follow while writing tests:
- All tests should be written using [JUnit 5 platform](https://junit.org/junit5/docs/current/user-guide/#overview).
- All the compilation test classes should extend [BaseCompilationTest](./src/testCommon/kotlin/compilation/model/BaseCompilationTest.kt)
- Consider using the scenario DSL for the incremental compilation tests, an usage example is located [here](src/testExample/kotlin/ExampleIncrementalScenarioTest.kt)
- Add `@DisplayName(...)` with meaningful description both for test class and methods inside. This will allow developers easier
to understand what test is about.
- Don't create one big test suite (class). Consider splitting tests into smaller suites. All tests are running in parallel (except daemon tests)
and having small tests suites should improve overall tests running time.
- Mark the test with the `DefaultStrategyAgnosticCompilationTest` annotation if the test is expected to perform exactly
the same using the daemon or in-process compiler execution strategy. This way the test will be executed using both strategies.
- If you're writing a test for a specific strategy, consider configuring it manually through `CompilationService.makeCompilerExecutionStrategyConfiguration()`
- Don't create one big test class. Consider splitting tests into smaller classes. All tests can run in parallel, thus having small tests
classes should improve overall tests running time.
- Don't create one big test suit. Consider grouping test classes semantically into test suits. Adding a new test suit is as easy as adding
an entry to `businessLogicTestSuits` in the [build.gradle.kts](./build.gradle.kts)
- All test classes should extend [BaseTest](./src/main/kotlin/BaseTest.kt)
The rules specific to compilation tests:
- All the compilation test classes should extend [BaseCompilationTest](./src/main/kotlin/compilation/BaseCompilationTest.kt)
- Consider using the scenario DSL for the incremental compilation tests, a usage example is
located [here](src/testExample/kotlin/ExampleIncrementalScenarioTest.kt)
- Mark the compilation test with the `DefaultStrategyAgnosticCompilationTest` annotation if the test is expected to perform exactly
the same using the daemon or in-process compiler execution strategy.
- If you're writing a test for a specific strategy, consider configuring it manually
through `CompilationService.makeCompilerExecutionStrategyConfiguration()`
#### The scenario DSL
The incremental compilation tests written using the scenario DSL are subject to some optimizations and automatic checks, allowing you to avoid boilerplate.
The incremental compilation tests written using the scenario DSL are subject to some optimizations and automatic checks, allowing you to
avoid boilerplate.
* Creating a module (e.g. `val module1 = module("jvm-module-1")`), you already have this module compiled non-incrementally
to apply further incremental changes. If this module is reused between different tests, the initial compilation output will be reused
instead of recompiling it again and again.
* Methods for applying modifications allow to perform automatic checks of resulting outputs files after the compilation,
so you don't need to create assertions using `assertOutputs`
Please refer to the example test [class](src/testExample/kotlin/ExampleIncrementalScenarioTest.kt) for more information
compiler/build-tools/kotlin-build-tools-api-tests/build.gradle.kts
+2 -1
View File
@@ -32,7 +32,7 @@ val compatibilityTestsVersions = listOf(
)
class BuildToolsVersion(val version: KotlinToolingVersion, val isCurrent: Boolean = false) {
override fun toString() = version.toString()
override fun toString() = if (isCurrent) "Snapshot" else version.toString()
}
fun Test.ensureExecutedAgainstExpectedBuildToolsImplVersion(version: BuildToolsVersion) {
@@ -64,6 +64,7 @@ fun SourceSet.configureCompatibilitySourceDirectories() {
)
}
// just add a new test suit name here and that's it
val businessLogicTestSuits = setOf(
"testExample",
)
compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/BaseTest.kt → compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/BaseTest.kt
+1 -2
View File
@@ -2,8 +2,7 @@
* Copyright 2010-2024 JetBrains s.r.o. and Kotlin Programming Language contributors.
* Use of this source code is governed by the Apache 2.0 license that can be found in the license/LICENSE.txt file.
*/
package org.jetbrains.kotlin.buildtools.api.tests.compilation.model
package org.jetbrains.kotlin.buildtools.api.tests
import org.jetbrains.kotlin.buildtools.api.CompilationService
import org.jetbrains.kotlin.tooling.core.KotlinToolingVersion
compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/BaseCompilationTest.kt → compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/BaseCompilationTest.kt
+3 -2
View File
@@ -1,11 +1,12 @@
/*
* Copyright 2010-2023 JetBrains s.r.o. and Kotlin Programming Language contributors.
* Copyright 2010-2024 JetBrains s.r.o. and Kotlin Programming Language contributors.
* Use of this source code is governed by the Apache 2.0 license that can be found in the license/LICENSE.txt file.
*/
package org.jetbrains.kotlin.buildtools.api.tests.compilation.model
package org.jetbrains.kotlin.buildtools.api.tests.compilation
import com.intellij.testFramework.TestDataPath
import org.jetbrains.kotlin.buildtools.api.tests.BaseTest
import org.junit.jupiter.api.io.TempDir
import java.nio.file.Path
compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/DefaultStrategyAgnosticCompilationTestArgumentProvider.kt
+1
View File
@@ -5,6 +5,7 @@
package org.jetbrains.kotlin.buildtools.api.tests.compilation.model
import org.jetbrains.kotlin.buildtools.api.tests.BaseTest
import org.junit.jupiter.api.Named.named
import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.params.provider.Arguments
compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/JvmModule.kt
+1
View File
@@ -9,6 +9,7 @@ import org.jetbrains.kotlin.buildtools.api.CompilationResult
import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration
import org.jetbrains.kotlin.buildtools.api.SourcesChanges
import org.jetbrains.kotlin.buildtools.api.jvm.*
import org.jetbrains.kotlin.buildtools.api.tests.BaseTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.Module
import org.jetbrains.kotlin.tooling.core.KotlinToolingVersion
import java.io.File
compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/Project.kt
+2
View File
@@ -7,6 +7,8 @@ package org.jetbrains.kotlin.buildtools.api.tests.compilation.model
import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration
import org.jetbrains.kotlin.buildtools.api.ProjectId
import org.jetbrains.kotlin.buildtools.api.tests.BaseTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.BaseCompilationTest
import java.nio.file.Path
import java.nio.file.Paths
import java.util.*
compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/scenario/scenarioDsl.kt
+2 -3
View File
@@ -10,10 +10,9 @@ import org.jetbrains.kotlin.buildtools.api.SourcesChanges
import org.jetbrains.kotlin.buildtools.api.jvm.IncrementalJvmCompilationConfiguration
import org.jetbrains.kotlin.buildtools.api.jvm.JvmCompilationConfiguration
import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertOutputs
import org.jetbrains.kotlin.buildtools.api.tests.compilation.BaseCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.CompilationOutcome
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.*
import java.nio.file.Files
import java.nio.file.Path
import java.util.*
import kotlin.io.path.*
internal class ScenarioModuleImpl(
compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityCompilationTest.kt
+1 -1
View File
@@ -7,7 +7,7 @@ package org.jetbrains.kotlin.buildtools.api.tests
import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration
import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertOutputs
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.BaseCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.project
import org.jetbrains.kotlin.test.TestMetadata
compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityTest.kt
-1
View File
@@ -5,7 +5,6 @@
package org.jetbrains.kotlin.buildtools.api.tests
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseTest
import org.junit.jupiter.api.Assertions
import org.junit.jupiter.api.Disabled
import org.junit.jupiter.api.DisplayName
compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalCompilationTest.kt
+8 -1
View File
@@ -9,7 +9,6 @@ import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguratio
import org.jetbrains.kotlin.buildtools.api.SourcesChanges
import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertCompiledSources
import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertLogContainsPatterns
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.LogLevel
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.project
@@ -18,6 +17,9 @@ import org.junit.jupiter.api.DisplayName
import kotlin.io.path.readText
import kotlin.io.path.writeText
/**
* Avoid using this DSL unless you face a scenario that can't be solved within the scenario DSL
*/
class ExampleIncrementalCompilationTest : BaseCompilationTest() {
@DisplayName("Sample IC test with a single module")
@DefaultStrategyAgnosticCompilationTest
@@ -26,6 +28,8 @@ class ExampleIncrementalCompilationTest : BaseCompilationTest() {
project(strategyConfig) {
val module1 = module("jvm-module-1")
// this is not the scenario DSL, so the module is not built at this moment
module1.compileIncrementally(SourcesChanges.Unknown)
val fooKt = module1.sourcesDirectory.resolve("foo.kt")
@@ -48,6 +52,9 @@ class ExampleIncrementalCompilationTest : BaseCompilationTest() {
val module1 = module("jvm-module-1")
val module2 = module("jvm-module-2", listOf(module1))
// this is not the scenario DSL, so the modules are not built at this moment
// you should handle the right order of compilation between modules yourself
module1.compileIncrementally(SourcesChanges.Unknown)
module2.compileIncrementally(SourcesChanges.Unknown)
compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalScenarioTest.kt
+8 -3
View File
@@ -9,7 +9,6 @@ import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguratio
import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertCompiledSources
import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertNoCompiledSources
import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.scenario
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.assertAddedOutputs
import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.assertNoOutputSetChanges
@@ -17,6 +16,9 @@ import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.assertRemo
import org.jetbrains.kotlin.test.TestMetadata
import org.junit.jupiter.api.DisplayName
/**
* The preferred way to write incremental compilation tests.
*/
class ExampleIncrementalScenarioTest : BaseCompilationTest() {
@DefaultStrategyAgnosticCompilationTest
@DisplayName("Sample scenario DSL IC test with a single module")
@@ -24,6 +26,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
fun testScenario1(strategyConfig: CompilerExecutionStrategyConfiguration) {
scenario(strategyConfig) {
val module1 = module("jvm-module-1")
// at this moment, the module is already initially built and ready for further incremental compilations
module1.createFile(
"foobar.kt",
@@ -35,7 +38,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
module1.compile {
assertCompiledSources("foobar.kt")
assertAddedOutputs("FoobarKt.class")
assertAddedOutputs("FoobarKt.class") // specify only the difference
}
module1.deleteFile(
@@ -44,7 +47,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
module1.compile {
assertNoCompiledSources()
assertRemovedOutputs("FoobarKt.class")
assertRemovedOutputs("FoobarKt.class") // specify only the difference
}
}
}
@@ -54,6 +57,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
@TestMetadata("jvm-module-1")
fun testScenario2(strategyConfig: CompilerExecutionStrategyConfiguration) {
scenario(strategyConfig) {
// compilation options may be modified
val module1 = module("jvm-module-1", incrementalCompilationOptionsModifier = { it.keepIncrementalCompilationCachesInMemory(false) })
module1.createFile(
@@ -96,6 +100,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
}
)
// you should handle the right order of compilation between modules yourself
module1.compile {
assertCompiledSources("bar.kt")
assertNoOutputSetChanges()
compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleNonIncrementalCompilationTest.kt
+8 -1
View File
@@ -8,7 +8,7 @@ package org.jetbrains.kotlin.buildtools.api.tests.compilation
import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration
import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertLogContainsPatterns
import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertOutputs
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.expectFailWithError
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.LogLevel
import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.project
@@ -23,6 +23,9 @@ class ExampleNonIncrementalCompilationTest : BaseCompilationTest() {
val module1 = module("jvm-module-1")
val module2 = module("jvm-module-2", listOf(module1))
// this is not the scenario DSL, so the modules are not built at this moment
// you should handle the right order of compilation between modules yourself
module1.compile {
assertOutputs("FooKt.class", "Bar.class", "BazKt.class")
}
@@ -43,6 +46,10 @@ class ExampleNonIncrementalCompilationTest : BaseCompilationTest() {
module1.compile {
expectFail()
assertLogContainsPatterns(LogLevel.ERROR, ".*bar\\.kt:1:1 Syntax error: Expecting a top level declaration.*".toRegex())
// equals to
expectFailWithError(".*bar\\.kt:1:1 Syntax error: Expecting a top level declaration.*".toRegex())
}
}
}