[BTA tests] Improve README.md

^KT-61860 In Progress

compiler/build-tools/kotlin-build-tools-api-tests/README.md

@@ -5,40 +5,49 @@ built on top of the [Build Tools API](../kotlin-build-tools-api/README.md).
 
 #### How to run
 
-To run all tests for all Gradle plugins use `check` task.
+To run all tests use the `check` task.
 
-The module defines test matrix using the `jvm-test-suite` plugin to cover different combinations of 
+The module defines test suites using the `jvm-test-suite` plugin.
-the Build Tools API's and its implementations' versions.
 
 #### Test suites:
-* `testSnapshotToSnapshot`: runs all tests against the API and implementation of the snapshot version. 
+
-  * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testSnapshotToSnapshot` to run them
+* Compatibility: a special test suit that runs against a set of implementation versions
-* `test1.9.20ToSnapshot`: runs all the tests marked with the `CompatibilityTests` annotation against API 1.9.20 and the snapshot implementation
+    * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testCompatibility1.9.20`
-  * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:test1.9.20ToSnapshot` to run them
+      to run the tests against BTA implementation 1.9.20
-* `testSnapshotTo1.9.20`: runs all the tests marked with the `CompatibilityTests` annotation against the snapshot API and 1.9.20 implementation
+    * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testCompatibilitySnapshot`
-    * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testSnapshotTo1.9.20` to run them
+      to run the tests against the current BTA implementation
+    * Avoid adding new tests here unless you can articulate their necessity as they will be executed multiple times significantly increasing
+      the overall test execution time.
+* Example: provides examples of the DSL usage. Excluded from the `check` task
+    * Use `./gradlew :compiler:build-tools:kotlin-build-tools-api-tests:testExample` to run them
 
 #### How to work with the tests
 
 Few rules you should follow while writing tests:
+
 - All tests should be written using [JUnit 5 platform](https://junit.org/junit5/docs/current/user-guide/#overview).
-- All the compilation test classes should extend [BaseCompilationTest](./src/testCommon/kotlin/compilation/model/BaseCompilationTest.kt)
-- Consider using the scenario DSL for the incremental compilation tests, an usage example is located [here](src/testExample/kotlin/ExampleIncrementalScenarioTest.kt)
 - Add `@DisplayName(...)` with meaningful description both for test class and methods inside. This will allow developers easier
   to understand what test is about.
-- Don't create one big test suite (class). Consider splitting tests into smaller suites. All tests are running in parallel (except daemon tests)
+- Don't create one big test class. Consider splitting tests into smaller classes. All tests can run in parallel, thus having small tests
-  and having small tests suites should improve overall tests running time.
+  classes should improve overall tests running time.
-- Mark the test with the `DefaultStrategyAgnosticCompilationTest` annotation if the test is expected to perform exactly 
+- Don't create one big test suit. Consider grouping test classes semantically into test suits. Adding a new test suit is as easy as adding
-  the same using the daemon or in-process compiler execution strategy. This way the test will be executed using both strategies.
+  an entry to `businessLogicTestSuits` in the [build.gradle.kts](./build.gradle.kts)
-- If you're writing a test for a specific strategy, consider configuring it manually through `CompilationService.makeCompilerExecutionStrategyConfiguration()` 
+- All test classes should extend [BaseTest](./src/main/kotlin/BaseTest.kt)
+
+The rules specific to compilation tests:
+
+- All the compilation test classes should extend [BaseCompilationTest](./src/main/kotlin/compilation/BaseCompilationTest.kt)
+- Consider using the scenario DSL for the incremental compilation tests, a usage example is
+  located [here](src/testExample/kotlin/ExampleIncrementalScenarioTest.kt)
+- Mark the compilation test with the `DefaultStrategyAgnosticCompilationTest` annotation if the test is expected to perform exactly
+  the same using the daemon or in-process compiler execution strategy.
+- If you're writing a test for a specific strategy, consider configuring it manually
+  through `CompilationService.makeCompilerExecutionStrategyConfiguration()`
 
 #### The scenario DSL
 
-The incremental compilation tests written using the scenario DSL are subject to some optimizations and automatic checks, allowing you to avoid boilerplate.
+The incremental compilation tests written using the scenario DSL are subject to some optimizations and automatic checks, allowing you to
+avoid boilerplate.
 
-* Creating a module (e.g. `val module1 = module("jvm-module-1")`), you already have this module compiled non-incrementally 
+Please refer to the example test [class](src/testExample/kotlin/ExampleIncrementalScenarioTest.kt) for more information
-to apply further incremental changes. If this module is reused between different tests, the initial compilation output will be reused
-instead of recompiling it again and again.
-* Methods for applying modifications allow to perform automatic checks of resulting outputs files after the compilation, 
-so you don't need to create assertions using `assertOutputs`
 

compiler/build-tools/kotlin-build-tools-api-tests/build.gradle.kts

@@ -32,7 +32,7 @@ val compatibilityTestsVersions = listOf(
 )
 
 class BuildToolsVersion(val version: KotlinToolingVersion, val isCurrent: Boolean = false) {
-    override fun toString() = version.toString()
+    override fun toString() = if (isCurrent) "Snapshot" else version.toString()
 }
 
 fun Test.ensureExecutedAgainstExpectedBuildToolsImplVersion(version: BuildToolsVersion) {
@@ -64,6 +64,7 @@ fun SourceSet.configureCompatibilitySourceDirectories() {
     )
 }
 
+// just add a new test suit name here and that's it
 val businessLogicTestSuits = setOf(
     "testExample",
 )

compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/BaseTest.kt

@@ -2,8 +2,7 @@
  * Copyright 2010-2024 JetBrains s.r.o. and Kotlin Programming Language contributors.
  * Use of this source code is governed by the Apache 2.0 license that can be found in the license/LICENSE.txt file.
  */
-
+package org.jetbrains.kotlin.buildtools.api.tests
-package org.jetbrains.kotlin.buildtools.api.tests.compilation.model
 
 import org.jetbrains.kotlin.buildtools.api.CompilationService
 import org.jetbrains.kotlin.tooling.core.KotlinToolingVersion

compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/BaseCompilationTest.kt

@@ -1,11 +1,12 @@
 /*
- * Copyright 2010-2023 JetBrains s.r.o. and Kotlin Programming Language contributors.
+ * Copyright 2010-2024 JetBrains s.r.o. and Kotlin Programming Language contributors.
  * Use of this source code is governed by the Apache 2.0 license that can be found in the license/LICENSE.txt file.
  */
 
-package org.jetbrains.kotlin.buildtools.api.tests.compilation.model
+package org.jetbrains.kotlin.buildtools.api.tests.compilation
 
 import com.intellij.testFramework.TestDataPath
+import org.jetbrains.kotlin.buildtools.api.tests.BaseTest
 import org.junit.jupiter.api.io.TempDir
 import java.nio.file.Path
 

compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/DefaultStrategyAgnosticCompilationTestArgumentProvider.kt

@@ -5,6 +5,7 @@
 
 package org.jetbrains.kotlin.buildtools.api.tests.compilation.model
 
+import org.jetbrains.kotlin.buildtools.api.tests.BaseTest
 import org.junit.jupiter.api.Named.named
 import org.junit.jupiter.api.extension.ExtensionContext
 import org.junit.jupiter.params.provider.Arguments

compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/JvmModule.kt

@@ -9,6 +9,7 @@ import org.jetbrains.kotlin.buildtools.api.CompilationResult
 import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration
 import org.jetbrains.kotlin.buildtools.api.SourcesChanges
 import org.jetbrains.kotlin.buildtools.api.jvm.*
+import org.jetbrains.kotlin.buildtools.api.tests.BaseTest
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.Module
 import org.jetbrains.kotlin.tooling.core.KotlinToolingVersion
 import java.io.File

compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/model/Project.kt

@@ -7,6 +7,8 @@ package org.jetbrains.kotlin.buildtools.api.tests.compilation.model
 
 import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration
 import org.jetbrains.kotlin.buildtools.api.ProjectId
+import org.jetbrains.kotlin.buildtools.api.tests.BaseTest
+import org.jetbrains.kotlin.buildtools.api.tests.compilation.BaseCompilationTest
 import java.nio.file.Path
 import java.nio.file.Paths
 import java.util.*

compiler/build-tools/kotlin-build-tools-api-tests/src/main/kotlin/compilation/scenario/scenarioDsl.kt

@@ -10,10 +10,9 @@ import org.jetbrains.kotlin.buildtools.api.SourcesChanges
 import org.jetbrains.kotlin.buildtools.api.jvm.IncrementalJvmCompilationConfiguration
 import org.jetbrains.kotlin.buildtools.api.jvm.JvmCompilationConfiguration
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertOutputs
+import org.jetbrains.kotlin.buildtools.api.tests.compilation.BaseCompilationTest
+import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.CompilationOutcome
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.*
-import java.nio.file.Files
-import java.nio.file.Path
-import java.util.*
 import kotlin.io.path.*
 
 internal class ScenarioModuleImpl(

compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityCompilationTest.kt

@@ -7,7 +7,7 @@ package org.jetbrains.kotlin.buildtools.api.tests
 
 import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertOutputs
-import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest
+import org.jetbrains.kotlin.buildtools.api.tests.compilation.BaseCompilationTest
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.project
 import org.jetbrains.kotlin.test.TestMetadata

compiler/build-tools/kotlin-build-tools-api-tests/src/testCompatibility/kotlin/ExampleCompatibilityTest.kt

@@ -5,7 +5,6 @@
 
 package org.jetbrains.kotlin.buildtools.api.tests
 
-import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseTest
 import org.junit.jupiter.api.Assertions
 import org.junit.jupiter.api.Disabled
 import org.junit.jupiter.api.DisplayName

compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalCompilationTest.kt

@@ -9,7 +9,6 @@ import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguratio
 import org.jetbrains.kotlin.buildtools.api.SourcesChanges
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertCompiledSources
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertLogContainsPatterns
-import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.LogLevel
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.project
@@ -18,6 +17,9 @@ import org.junit.jupiter.api.DisplayName
 import kotlin.io.path.readText
 import kotlin.io.path.writeText
 
+/**
+ * Avoid using this DSL unless you face a scenario that can't be solved within the scenario DSL
+ */
 class ExampleIncrementalCompilationTest : BaseCompilationTest() {
     @DisplayName("Sample IC test with a single module")
     @DefaultStrategyAgnosticCompilationTest
@@ -26,6 +28,8 @@ class ExampleIncrementalCompilationTest : BaseCompilationTest() {
         project(strategyConfig) {
             val module1 = module("jvm-module-1")
 
+            // this is not the scenario DSL, so the module is not built at this moment
+
             module1.compileIncrementally(SourcesChanges.Unknown)
 
             val fooKt = module1.sourcesDirectory.resolve("foo.kt")
@@ -48,6 +52,9 @@ class ExampleIncrementalCompilationTest : BaseCompilationTest() {
             val module1 = module("jvm-module-1")
             val module2 = module("jvm-module-2", listOf(module1))
 
+            // this is not the scenario DSL, so the modules are not built at this moment
+
+            // you should handle the right order of compilation between modules yourself
             module1.compileIncrementally(SourcesChanges.Unknown)
             module2.compileIncrementally(SourcesChanges.Unknown)
 

compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleIncrementalScenarioTest.kt

@@ -9,7 +9,6 @@ import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguratio
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertCompiledSources
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertNoCompiledSources
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.scenario
-import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.assertAddedOutputs
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.assertNoOutputSetChanges
@@ -17,6 +16,9 @@ import org.jetbrains.kotlin.buildtools.api.tests.compilation.scenario.assertRemo
 import org.jetbrains.kotlin.test.TestMetadata
 import org.junit.jupiter.api.DisplayName
 
+/**
+ * The preferred way to write incremental compilation tests.
+ */
 class ExampleIncrementalScenarioTest : BaseCompilationTest() {
     @DefaultStrategyAgnosticCompilationTest
     @DisplayName("Sample scenario DSL IC test with a single module")
@@ -24,6 +26,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
     fun testScenario1(strategyConfig: CompilerExecutionStrategyConfiguration) {
         scenario(strategyConfig) {
             val module1 = module("jvm-module-1")
+            // at this moment, the module is already initially built and ready for further incremental compilations
 
             module1.createFile(
                 "foobar.kt",
@@ -35,7 +38,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
 
             module1.compile {
                 assertCompiledSources("foobar.kt")
-                assertAddedOutputs("FoobarKt.class")
+                assertAddedOutputs("FoobarKt.class") // specify only the difference
             }
 
             module1.deleteFile(
@@ -44,7 +47,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
 
             module1.compile {
                 assertNoCompiledSources()
-                assertRemovedOutputs("FoobarKt.class")
+                assertRemovedOutputs("FoobarKt.class") // specify only the difference
             }
         }
     }
@@ -54,6 +57,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
     @TestMetadata("jvm-module-1")
     fun testScenario2(strategyConfig: CompilerExecutionStrategyConfiguration) {
         scenario(strategyConfig) {
+            // compilation options may be modified
             val module1 = module("jvm-module-1", incrementalCompilationOptionsModifier = { it.keepIncrementalCompilationCachesInMemory(false) })
 
             module1.createFile(
@@ -96,6 +100,7 @@ class ExampleIncrementalScenarioTest : BaseCompilationTest() {
                 }
             )
 
+            // you should handle the right order of compilation between modules yourself
             module1.compile {
                 assertCompiledSources("bar.kt")
                 assertNoOutputSetChanges()

compiler/build-tools/kotlin-build-tools-api-tests/src/testExample/kotlin/ExampleNonIncrementalCompilationTest.kt

@@ -8,7 +8,7 @@ package org.jetbrains.kotlin.buildtools.api.tests.compilation
 import org.jetbrains.kotlin.buildtools.api.CompilerExecutionStrategyConfiguration
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertLogContainsPatterns
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.assertOutputs
-import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.BaseCompilationTest
+import org.jetbrains.kotlin.buildtools.api.tests.compilation.assertions.expectFailWithError
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.DefaultStrategyAgnosticCompilationTest
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.LogLevel
 import org.jetbrains.kotlin.buildtools.api.tests.compilation.model.project
@@ -23,6 +23,9 @@ class ExampleNonIncrementalCompilationTest : BaseCompilationTest() {
             val module1 = module("jvm-module-1")
             val module2 = module("jvm-module-2", listOf(module1))
 
+            // this is not the scenario DSL, so the modules are not built at this moment
+
+            // you should handle the right order of compilation between modules yourself
             module1.compile {
                 assertOutputs("FooKt.class", "Bar.class", "BazKt.class")
             }
@@ -43,6 +46,10 @@ class ExampleNonIncrementalCompilationTest : BaseCompilationTest() {
             module1.compile {
                 expectFail()
                 assertLogContainsPatterns(LogLevel.ERROR, ".*bar\\.kt:1:1 Syntax error: Expecting a top level declaration.*".toRegex())
+
+                // equals to
+
+                expectFailWithError(".*bar\\.kt:1:1 Syntax error: Expecting a top level declaration.*".toRegex())
             }
         }
     }