diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/README.md b/compiler/build-tools/kotlin-build-tools-api-tests/README.md index 7126961a0c6..c877324c654 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/README.md +++ b/compiler/build-tools/kotlin-build-tools-api-tests/README.md @@ -1,44 +1,53 @@ ## Build Tools API integration tests -This module contains integration tests covering the build tools API implementation using the DSL +This module contains integration tests covering the build tools API implementation using the DSL built on top of the [Build Tools API](../kotlin-build-tools-api/README.md). #### How to run -To run all tests for all Gradle plugins use `check` task. +To run all tests use the `check` task. -The module defines test matrix using the `jvm-test-suite` plugin to cover different combinations of -the Build Tools API's and its implementations' versions. +The module defines test suites using the `jvm-test-suite` plugin. #### Test suites: -* `testSnapshotToSnapshot`: runs all tests against the API and implementation of the snapshot version. - * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testSnapshotToSnapshot` to run them -* `test1.9.20ToSnapshot`: runs all the tests marked with the `CompatibilityTests` annotation against API 1.9.20 and the snapshot implementation - * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:test1.9.20ToSnapshot` to run them -* `testSnapshotTo1.9.20`: runs all the tests marked with the `CompatibilityTests` annotation against the snapshot API and 1.9.20 implementation - * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testSnapshotTo1.9.20` to run them + +* Compatibility: a special test suit that runs against a set of implementation versions + * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testCompatibility1.9.20` + to run the tests against BTA implementation 1.9.20 + * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testCompatibilitySnapshot` + to run the tests against the current BTA implementation + * Avoid adding new tests here unless you can articulate their necessity as they will be executed multiple times significantly increasing + the overall test execution time. +* Example: provides examples of the DSL usage. Excluded from the `check` task + * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testExample` to run them #### How to work with the tests Few rules you should follow while writing tests: + - All tests should be written using [JUnit 5 platform](https://junit.org/junit5/docs/current/user-guide/#overview). -- All the compilation test classes should extend [BaseCompilationTest](./src/testCommon/kotlin/compilation/model/BaseCompilationTest.kt) -- Consider using the scenario DSL for the incremental compilation tests, an usage example is located [here](src/testExample/kotlin/ExampleIncrementalScenarioTest.kt) - Add `@DisplayName(...)` with meaningful description both for test class and methods inside. This will allow developers easier to understand what test is about. -- Don't create one big test suite (class). Consider splitting tests into smaller suites. All tests are running in parallel (except daemon tests) - and having small tests suites should improve overall tests running time. -- Mark the test with the `DefaultStrategyAgnosticCompilationTest` annotation if the test is expected to perform exactly - the same using the daemon or in-process compiler execution strategy. This way the test will be executed using both strategies. -- If you're writing a test for a specific strategy, consider configuring it manually through `CompilationService.makeCompilerExecutionStrategyConfiguration()` +- Don't create one big test class. Consider splitting tests into smaller classes. All tests can run in parallel, thus having small tests + classes should improve overall tests running time. +- Don't create one big test suit. Consider grouping test classes semantically into test suits. Adding a new test suit is as easy as adding + an entry to `businessLogicTestSuits` in the [build.gradle.kts](./build.gradle.kts) +- All test classes should extend [BaseTest](./src/main/kotlin/BaseTest.kt) + +The rules specific to compilation tests: + +- All the compilation test classes should extend [BaseCompilationTest](./src/main/kotlin/compilation/BaseCompilationTest.kt) +- Consider using the scenario DSL for the incremental compilation tests, a usage example is + located [here](src/testExample/kotlin/ExampleIncrementalScenarioTest.kt) +- Mark the compilation test with the `DefaultStrategyAgnosticCompilationTest` annotation if the test is expected to perform exactly + the same using the daemon or in-process compiler execution strategy. +- If you're writing a test for a specific strategy, consider configuring it manually + through `CompilationService.makeCompilerExecutionStrategyConfiguration()` #### The scenario DSL -The incremental compilation tests written using the scenario DSL are subject to some optimizations and automatic checks, allowing you to avoid boilerplate. +The incremental compilation tests written using the scenario DSL are subject to some optimizations and automatic checks, allowing you to +avoid boilerplate. -* Creating a module (e.g. `val module1 = module("jvm-module-1")`), you already have this module compiled non-incrementally -to apply further incremental changes. If this module is reused between different tests, the initial compilation output will be reused -instead of recompiling it again and again. -* Methods for applying modifications allow to perform automatic checks of resulting outputs files after the compilation, -so you don't need to create assertions using `assertOutputs` +Please refer to the example test [class](src/testExample/kotlin/ExampleIncrementalScenarioTest.kt) for more information diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/build.gradle.kts b/compiler/build-tools/kotlin-build-tools-api-tests/build.gradle.kts index d09fce92b08..53523d07f31 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/build.gradle.kts +++ b/compiler/build-tools/kotlin-build-tools-api-tests/build.gradle.kts @@ -32,7 +32,7 @@ val compatibilityTestsVersions = listOf( ) class BuildToolsVersion(val version: KotlinToolingVersion, val isCurrent: Boolean = false) { - override fun toString() = version.toString() + override fun toString() = if (isCurrent) "Snapshot" else version.toString() } fun Test.ensureExecutedAgainstExpectedBuildToolsImplVersion(version: BuildToolsVersion) { @@ -64,6 +64,7 @@ fun SourceSet.configureCompatibilitySourceDirectories() { ) } +// just add a new test suit name here and that's it val businessLogicTestSuits = setOf( "testExample", ) diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/BaseTest.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/BaseTest.kt similarity index 93% rename from compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/BaseTest.kt rename to compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/BaseTest.kt index 7d6267481f8..6d00d5de606 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/BaseTest.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/BaseTest.kt @@ -2,8 +2,7 @@ * Copyright 2010-2024 JetBrains s.r.o. and Kotlin Programming Language contributors. * Use of this source code is governed by the Apache 2.0 license that can be found in the license/LICENSE.txt file. */ - -package org.jetbrains.kotlin.buildtools.api.tests.compilation.model +package org.jetbrains.kotlin.buildtools.api.tests import org.jetbrains.kotlin.buildtools.api.CompilationService import org.jetbrains.kotlin.tooling.core.KotlinToolingVersion diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/BaseCompilationTest.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/BaseCompilationTest.kt similarity index 69% rename from compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/BaseCompilationTest.kt rename to compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/BaseCompilationTest.kt index 81c999b8bbf..9ce18a14f62 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/BaseCompilationTest.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/BaseCompilationTest.kt @@ -1,11 +1,12 @@ /* - * Copyright 2010-2023 JetBrains s.r.o. and Kotlin Programming Language contributors. + * Copyright 2010-2024 JetBrains s.r.o. and Kotlin Programming Language contributors. * Use of this source code is governed by the Apache 2.0 license that can be found in the license/LICENSE.txt file. */ -package org.jetbrains.kotlin.buildtools.api.tests.compilation.model +package org.jetbrains.kotlin.buildtools.api.tests.compilation import com.intellij.testFramework.TestDataPath +import org.jetbrains.kotlin.buildtools.api.tests.BaseTest import org.junit.jupiter.api.io.TempDir import java.nio.file.Path diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/DefaultStrategyAgnosticCompilationTestArgumentProvider.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/DefaultStrategyAgnosticCompilationTestArgumentProvider.kt index 13b5264e4b8..b0ff859aef6 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/DefaultStrategyAgnosticCompilationTestArgumentProvider.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/DefaultStrategyAgnosticCompilationTestArgumentProvider.kt @@ -5,6 +5,7 @@ package org.jetbrains.kotlin.buildtools.api.tests.compilation.model +import org.jetbrains.kotlin.buildtools.api.tests.BaseTest import org.junit.jupiter.api.Named.named import org.junit.jupiter.api.extension.ExtensionContext import org.junit.jupiter.params.provider.Arguments diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/JvmModule.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/JvmModule.kt index c04b55cd21d..f7b5b74e773 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/JvmModule.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/JvmModule.kt @@ -9,6 +9,7 @@ import org.jetbrains.kotlin.buildtools.api.CompilationResult import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration import org.jetbrains.kotlin.buildtools.api.SourcesChanges import org.jetbrains.kotlin.buildtools.api.jvm.* +import org.jetbrains.kotlin.buildtools.api.tests.BaseTest import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.Module import org.jetbrains.kotlin.tooling.core.KotlinToolingVersion import java.io.File diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/Project.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/Project.kt index 6a30378cb93..ab2ca239656 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/Project.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/Project.kt @@ -7,6 +7,8 @@ package org.jetbrains.kotlin.buildtools.api.tests.compilation.model import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration import org.jetbrains.kotlin.buildtools.api.ProjectId +import org.jetbrains.kotlin.buildtools.api.tests.BaseTest +import org.jetbrains.kotlin.buildtools.api.tests.compilation.BaseCompilationTest import java.nio.file.Path import java.nio.file.Paths import java.util.* diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/scenario/scenarioDsl.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/scenario/scenarioDsl.kt index 9ff9dc77811..52408a2f491 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/scenario/scenarioDsl.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/scenario/scenarioDsl.kt @@ -10,10 +10,9 @@ import org.jetbrains.kotlin.buildtools.api.SourcesChanges import org.jetbrains.kotlin.buildtools.api.jvm.IncrementalJvmCompilationConfiguration import org.jetbrains.kotlin.buildtools.api.jvm.JvmCompilationConfiguration import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertOutputs +import org.jetbrains.kotlin.buildtools.api.tests.compilation.BaseCompilationTest +import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.CompilationOutcome import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.* -import java.nio.file.Files -import java.nio.file.Path -import java.util.* import kotlin.io.path.* internal class ScenarioModuleImpl( diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityCompilationTest.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityCompilationTest.kt index f6a67533ea8..687951db4ba 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityCompilationTest.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityCompilationTest.kt @@ -7,7 +7,7 @@ package org.jetbrains.kotlin.buildtools.api.tests import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertOutputs -import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest +import org.jetbrains.kotlin.buildtools.api.tests.compilation.BaseCompilationTest import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.project import org.jetbrains.kotlin.test.TestMetadata diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityTest.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityTest.kt index d9e6d781a57..1f6b37c856e 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityTest.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityTest.kt @@ -5,7 +5,6 @@ package org.jetbrains.kotlin.buildtools.api.tests -import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseTest import org.junit.jupiter.api.Assertions import org.junit.jupiter.api.Disabled import org.junit.jupiter.api.DisplayName diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalCompilationTest.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalCompilationTest.kt index e0e9c9ec821..98b744d81f1 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalCompilationTest.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalCompilationTest.kt @@ -9,7 +9,6 @@ import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguratio import org.jetbrains.kotlin.buildtools.api.SourcesChanges import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertCompiledSources import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertLogContainsPatterns -import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.LogLevel import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.project @@ -18,6 +17,9 @@ import org.junit.jupiter.api.DisplayName import kotlin.io.path.readText import kotlin.io.path.writeText +/** + * Avoid using this DSL unless you face a scenario that can't be solved within the scenario DSL + */ class ExampleIncrementalCompilationTest : BaseCompilationTest() { @DisplayName("Sample IC test with a single module") @DefaultStrategyAgnosticCompilationTest @@ -26,6 +28,8 @@ class ExampleIncrementalCompilationTest : BaseCompilationTest() { project(strategyConfig) { val module1 = module("jvm-module-1") + // this is not the scenario DSL, so the module is not built at this moment + module1.compileIncrementally(SourcesChanges.Unknown) val fooKt = module1.sourcesDirectory.resolve("foo.kt") @@ -48,6 +52,9 @@ class ExampleIncrementalCompilationTest : BaseCompilationTest() { val module1 = module("jvm-module-1") val module2 = module("jvm-module-2", listOf(module1)) + // this is not the scenario DSL, so the modules are not built at this moment + + // you should handle the right order of compilation between modules yourself module1.compileIncrementally(SourcesChanges.Unknown) module2.compileIncrementally(SourcesChanges.Unknown) diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalScenarioTest.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalScenarioTest.kt index a4f7872c3d1..f032ef6f1b4 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalScenarioTest.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalScenarioTest.kt @@ -9,7 +9,6 @@ import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguratio import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertCompiledSources import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertNoCompiledSources import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.scenario -import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.assertAddedOutputs import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.assertNoOutputSetChanges @@ -17,6 +16,9 @@ import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.assertRemo import org.jetbrains.kotlin.test.TestMetadata import org.junit.jupiter.api.DisplayName +/** + * The preferred way to write incremental compilation tests. + */ class ExampleIncrementalScenarioTest : BaseCompilationTest() { @DefaultStrategyAgnosticCompilationTest @DisplayName("Sample scenario DSL IC test with a single module") @@ -24,6 +26,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() { fun testScenario1(strategyConfig: CompilerExecutionStrategyConfiguration) { scenario(strategyConfig) { val module1 = module("jvm-module-1") + // at this moment, the module is already initially built and ready for further incremental compilations module1.createFile( "foobar.kt", @@ -35,7 +38,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() { module1.compile { assertCompiledSources("foobar.kt") - assertAddedOutputs("FoobarKt.class") + assertAddedOutputs("FoobarKt.class") // specify only the difference } module1.deleteFile( @@ -44,7 +47,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() { module1.compile { assertNoCompiledSources() - assertRemovedOutputs("FoobarKt.class") + assertRemovedOutputs("FoobarKt.class") // specify only the difference } } } @@ -54,6 +57,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() { @TestMetadata("jvm-module-1") fun testScenario2(strategyConfig: CompilerExecutionStrategyConfiguration) { scenario(strategyConfig) { + // compilation options may be modified val module1 = module("jvm-module-1", incrementalCompilationOptionsModifier = { it.keepIncrementalCompilationCachesInMemory(false) }) module1.createFile( @@ -96,6 +100,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() { } ) + // you should handle the right order of compilation between modules yourself module1.compile { assertCompiledSources("bar.kt") assertNoOutputSetChanges() diff --git a/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleNonIncrementalCompilationTest.kt b/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleNonIncrementalCompilationTest.kt index 863d75c33b2..01890c91509 100644 --- a/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleNonIncrementalCompilationTest.kt +++ b/compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleNonIncrementalCompilationTest.kt @@ -8,7 +8,7 @@ package org.jetbrains.kotlin.buildtools.api.tests.compilation import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertLogContainsPatterns import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertOutputs -import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest +import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.expectFailWithError import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.LogLevel import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.project @@ -23,6 +23,9 @@ class ExampleNonIncrementalCompilationTest : BaseCompilationTest() { val module1 = module("jvm-module-1") val module2 = module("jvm-module-2", listOf(module1)) + // this is not the scenario DSL, so the modules are not built at this moment + + // you should handle the right order of compilation between modules yourself module1.compile { assertOutputs("FooKt.class", "Bar.class", "BazKt.class") } @@ -43,6 +46,10 @@ class ExampleNonIncrementalCompilationTest : BaseCompilationTest() { module1.compile { expectFail() assertLogContainsPatterns(LogLevel.ERROR, ".*bar\\.kt:1:1 Syntax error: Expecting a top level declaration.*".toRegex()) + + // equals to + + expectFailWithError(".*bar\\.kt:1:1 Syntax error: Expecting a top level declaration.*".toRegex()) } } }