azalea/kotlin-fork
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Minor, fix some typos/errors in test-infrastructure ReadMe

Browse Source
This commit is contained in:
Alexander Udalov
2021-08-16 22:24:02 +02:00
parent 8d4f26cf84
commit 75b8d44be5
1 changed files with 45 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
compiler/test-infrastructure/ReadMe.md
+45 -45
View File
@@ -26,7 +26,7 @@ Each test includes at least one module. Module is a base compilation entity whic
### Steps pipeline
Each test defines multiple number of parametrized steps in specific order. [TestRunner](tests/org/jetbrains/kotlin/test/TestRunner.kt) (main entrypoint to test) takes configuration and module structure and perform next steps:
Each test defines multiple number of parametrized steps in specific order. [TestRunner](tests/org/jetbrains/kotlin/test/TestRunner.kt) (main entrypoint to test) takes configuration and module structure and performs next steps:
1. Parse module structure from testdata
2. For each module in test structure:
1. Introduce `var artifact` which represents artifact produced by last facade
@@ -47,19 +47,19 @@ Each test defines multiple number of parametrized steps in specific order. [Test
Directives are main option for configuring test. With them you can configure files and modules in your test, compiler flags, enable and disable specific handlers etc. Directives are objects of specific class [Directive](tests/org/jetbrains/kotlin/test/directives/model/Directive.kt), and there are three different subclasses for three different types of directives (they all declared in [Directive.kt](tests/org/jetbrains/kotlin/test/directives/model/Directive.kt) file):
- `SimpleDirective` is a directive which can be only enabled or disabled
- `StringDirective` is a directive which may accept one or multiple string arguments
- `ValueDirective<T>` is a directive which may accept on or multiple arguments of type `T`
- `ValueDirective<T>` is a directive which may accept one or multiple arguments of type `T`
All directives should be declared in special containers which are inheritors of [SimpleDirectivesContainer](tests/org/jetbrains/kotlin/test/directives/model/DirectivesContainer.kt). There are multiple utility functions in [SimpleDirectivesContainer](tests/org/jetbrains/kotlin/test/directives/model/DirectivesContainer.kt) which **should** be used for declaring directives:
- function `directive()` declares declares `SimpleDirective`
- function `directive()` declares `SimpleDirective`
- function `stringDirective()` declares `StringDirective`
- function `valueDirective<T>()` takes parser of type `(String) -> T?` and declares `ValueDirective<T>`. Parser function is needed to transform arguments from testdata to real values of type `T`
- function `enumDirective<T>()` is needed to create `ValueDirective<T>` of enum type `T`. It doesn't require `parser` function and parse enum values by their names. Note that you can pass `additionalParser: (String) -> T?` as a fallback parsing option.
All this functions also takes next arguments:
- `description: String` -- required parameter which should include description of this directive
- `applicability: DirectiveApplicability`. With this optional argument you can configure where this directive can be applicable if you test contains multiple files or modules. By default all directives has `Global` applicability which means that directive can be declared at global or module level, but not in test files (about files and modules read in [Module structure](#module-structure))
All these functions also take the following arguments:
- `description: String`: required parameter which should include description of this directive
- `applicability: DirectiveApplicability`: with this optional argument you can configure where this directive can be applicable if you test contains multiple files or modules. By default all directives have `Global` applicability which means that directive can be declared at global or module level, but not in test files (read about files and modules in [Module structure](#module-structure))
Name of directive will be same as name of directive property created by one of those functions. Note that all of them provides a delegate, so you should create directives using `by directive()`, not `= directive()`.
Name of directive will be same as name of directive property created by one of those functions. Note that all of the `*directive()` functions provide a property delegate, so you should create directives using `by directive()`, not `= directive()`.
As an example of directive container you can check [directives for configuring language settings](tests/org/jetbrains/kotlin/test/directives/LanguageSettingsDirectives.kt).
@@ -69,13 +69,13 @@ In testdata file you should declare directives using following syntax:
### Module structure directives
Test framework supports tests which contain different source files or modules in single testdata file. There are two directives which are needed to splitting testdata file:
Test framework supports tests which contain different source files or modules in single testdata file. There are two directives which are needed to split a testdata file:
- Directive `// FILE: fileName.kt` says that all content until next module structure directive belongs to file `fileName.kt`
- Directive `// MODULE: moduleName` says that all files until next `MODULE` directive belongs to module `moduleName`
If there are no `MODULE` directives in testdata file them all files belong to default module with name `main`.
If there are no `MODULE` directives in testdata file, then all files belong to a default module with the name `main`.
If there are no `FILE` directives in module then all content of module belong to default file with name `main.kt`
If there are no `FILE` directives in module, then all content of module belongs to a default file with the name `main.kt`.
#### Module dependencies
@@ -84,17 +84,17 @@ Each module can declare that it depends on some other module with following synt
// MODULE: name[(dep1, dep2)[(friend 1, friend2)][(refined dep 1, refiend dep 2)]]
```
- if module has no friend modules, you can write just `// MODULE: name(dep1, dep2)`
- if module has no dependencies at all you can write only module name: `// MODULE: name`
- if module has no dependencies but has friends then you should declare empty parenthesis of dependencies: `// MODULE: name()(friend1, friend2)`
- if module has not dependencies but refined ones then you should declare empty parenthesis for first two kinds: `// MODULE: name()()(refined dep 1, refiend dep 2)`
- if module has no dependencies at all, you can write only module name: `// MODULE: name`
- if module has no dependencies but has friends, then you should declare empty parentheses of dependencies: `// MODULE: name()(friend1, friend2)`
- if module does not have normal dependencies but has refined ones, then you should declare empty parentheses for first two kinds: `// MODULE: name()()(refined dep 1, refiend dep 2)`
# Implementation details
## Test services
Different parts of test (like facades and handlers) may use some additional components which contain some logic which can be shared between different those parts. For such components there is exists special class [TestServices](tests/org/jetbrains/kotlin/test/services/TestServices.kt) which is a strongly typed container of test services (inheritors of interface `TestService`). All test services are initialized before test started and persist only until test is finished, which means that it's safe to store some caches for specific test in services.
Different parts of test (like facades and handlers) may use some additional components which contain some logic which can be shared between different those parts. For such components there is a special class [TestServices](tests/org/jetbrains/kotlin/test/services/TestServices.kt) which is a strongly typed container of test services (inheritors of interface `TestService`). All test services are initialized before the test is started and persist only until the test is finished, which means that it's safe to store some caches for specific test in services.
To declare your own service you need to do three simple things:
To declare your own service you need to do three things:
1. Declare class of service with one constructor which takes `TestServices` as parameter and inherit it from `TestService` interface
```kotlin
class MySuperService(val testServices: TestServices) : TestService {
@@ -106,7 +106,7 @@ class MySuperService(val testServices: TestServices) : TestService {
val TestServices.mySuperService: MySuperService by TestServices.testServiceAccessor()
```
3. Register service inside test. There are two ways to register service:
- A lot of different test entities (like facades or handlers) are marked with `ServicesAndDirectivesContainer` interface, which has `additionalService` field with list of services this entity uses. During test configuration infrastructure collects all those additional services, creates instances of them and registers inside `TestServices`
- A lot of different test entities (like facades or handlers) are marked with `ServicesAndDirectivesContainer` interface, which has `additionalService` field with list of services this entity uses. During test configuration, infrastructure collects all those additional services, creates instances of them and registers inside `TestServices`
```kotlin
class MyHandler : AnalysisHandler<MyArtifact>() {
override val additionalServices: List<ServiceRegistrationData> = listOf(service(::MySuperService))
@@ -122,20 +122,20 @@ override fun TestConfigurationBuilder.configure() {
### Existing services
There will be described list of existing services which are usefull in wide of different test cases:
- [BackendKindExtrac](tests/org/jetbrains/kotlin/test/services/BackendKindExtractor.kt) transforms `TargetBackend` to `BackendKind`
- [SourceFileProvi](tests/org/jetbrains/kotlin/test/services/SourceFileProvider.kt) retrieves content of test files from test modules
- [KotlinTestI](tests/org/jetbrains/kotlin/test/services/KotlinTestInfo.kt) contains info about test (test name, test class, etc)
- [DependencyProvi](tests/org/jetbrains/kotlin/test/services/DependencyProvider.kt) caches and provides artifacts of modules analyzed by facade steps
- [Assertions](tests/org/jetbrains/kotlin/test/services/Assertions.kt) contains utility assertions methods. This service is needed to abstract tJUnit5Assertiest infrastructure from any existing test framework (most commonly used assertions implementation is [JUnit5Assertions](../tests-common-new/tests/org/jetbrains/kotlin/test/services/JUnit5Assertions.kt))
Here are some existing services which are useful in a wide range of different test cases:
- [BackendKindExtractor](tests/org/jetbrains/kotlin/test/services/BackendKindExtractor.kt) transforms `TargetBackend` to `BackendKind`
- [SourceFileProvider](tests/org/jetbrains/kotlin/test/services/SourceFileProvider.kt) retrieves content of test files from test modules
- [KotlinTestInfo](tests/org/jetbrains/kotlin/test/services/KotlinTestInfo.kt) contains info about test (test name, test class, etc)
- [DependencyProvider](tests/org/jetbrains/kotlin/test/services/DependencyProvider.kt) caches and provides artifacts of modules analyzed by facade steps
- [Assertions](tests/org/jetbrains/kotlin/test/services/Assertions.kt) contains utility assertions methods. This service is needed to abstract assertions infrastructure from any existing test framework (most commonly used assertions implementation is [JUnit5Assertions](../tests-common-new/tests/org/jetbrains/kotlin/test/services/JUnit5Assertions.kt))
- [CompilerConfigurationProvider](../tests-common-new/tests/org/jetbrains/kotlin/test/services/CompilerConfigurationProvider.kt) provider of compiler configuration for different modules (additional info below)
- [TemporaryDirectoryMana](tests/org/jetbrains/kotlin/test/services/TemporaryDirectoryManager.kt) can create temporary directories for test purposes (e.g. directory to write generated .class files)
- [TemporaryDirectoryManager](tests/org/jetbrains/kotlin/test/services/TemporaryDirectoryManager.kt) can create temporary directories for test purposes (e.g. directory to write generated .class files)
There are a lot of other services, you can find them by looking to inheritors of `TestService`
There are many other services, you can find them by looking at inheritors of `TestService`.
#### Compiler configuration provider
[CompilerConfiguration](../config/src/org/jetbrains/kotlin/config/CompilerConfiguration.java) is main class which configures how specific module will be analyzed or compiled and for its setup there is exists a special service named [CompilerConfigurationProvider](../tests-common-new/tests/org/jetbrains/kotlin/test/services/CompilerConfigurationProvider.kt). It creates `CompilerConfiguration` which is based on list of [EnvironmentConfigurators](tests/org/jetbrains/kotlin/test/services/EnvironmentConfigurator.kt) which can be registered in test. So if you want to customize compiler configuration you need to modifiy existing configurator (e.g. [JvmEnvironmentConfigurator](../tests-common-new/tests/org/jetbrains/kotlin/test/services/configuration/JvmEnvironmentConfigurator.kt)) or write your own. Main method of `EnvironmentConfigurator` is a `configureCompilerConfiguration` which takes compiler configuration and test module, so you can configure configuration (sorry for tautology) using directives which are applied to specific module.
[CompilerConfiguration](../config/src/org/jetbrains/kotlin/config/CompilerConfiguration.java) is main class which configures how specific module will be analyzed or compiled and for its setup there is a special service named [CompilerConfigurationProvider](../tests-common-new/tests/org/jetbrains/kotlin/test/services/CompilerConfigurationProvider.kt). It creates `CompilerConfiguration` which is based on list of [EnvironmentConfigurators](tests/org/jetbrains/kotlin/test/services/EnvironmentConfigurator.kt) which can be registered in test. So if you want to customize compiler configuration you need to modify existing configurator (e.g. [JvmEnvironmentConfigurator](../tests-common-new/tests/org/jetbrains/kotlin/test/services/configuration/JvmEnvironmentConfigurator.kt)) or write your own. Main method of `EnvironmentConfigurator` is `configureCompilerConfiguration` which takes compiler configuration and test module, so you can configure configuration (sorry for tautology) using directives which are applied to specific module.
There are also two additional methods which can be used to provide some simple mapping:
1. from some value directive to configuration key of same type
@@ -179,7 +179,7 @@ override fun TestConfigurationBuilder.configure() {
#### Source file providers
Sometimes you may want to add some existing file to multiple test (e.g. pack of helper functions) with some directive. For that you may use [AdditionalSourceProvider](tests/org/jetbrains/kotlin/test/services/AdditionalSourceProvider.kt). This service takes test module and returns list of additional test files, which can be created from regular `File` using `toTestFile` method. If you want to make new `TestFile` manually please ensure that it has flag `isAdditional` set to `true`. This flag removes additional files processing from some handlers.
Sometimes you may want to add some existing file (e.g. pack of helper functions) to multiple test cases with some directive. For that you may use [AdditionalSourceProvider](tests/org/jetbrains/kotlin/test/services/AdditionalSourceProvider.kt). This service takes test module and returns list of additional test files, which can be created from regular `File` using `toTestFile` method. If you want to make new `TestFile` manually please ensure that it has flag `isAdditional` set to `true`. This flag removes additional files processing from some handlers.
# Writing handlers
@@ -201,11 +201,11 @@ abstract class AnalysisHandler<A : ResultingArtifact<A>>(
`processModule` is called for each module with artifact of `artifactKind` if such artifact was produced by facade step. `processAfterAllModules` is called after all modules are analyzed so you can collect some information for each module, combine it to one piece in `processAfterAllModules` and assert something on it.
Boolean flags in constructor defines interaction of specific handler with other handlers and steps:
- if `failureDisablesNextSteps` set to `true` then failure in `processModule` will disable following steps for this module
- if `doNotRunIfThereWerePreviousFailures` set to `true` then this particular handler will be skipped if there were exceptions from handlers which were called before
Boolean flags in the constructor define interaction of specific handler with other handlers and steps:
- if `failureDisablesNextSteps` is set to `true`, then failure in `processModule` will disable following steps for this module
- if `doNotRunIfThereWerePreviousFailures` is set to `true`, then this particular handler will be skipped if there were exceptions from handlers which were called before
Please note that handlers constructor should have shape `(TestServices) -> MyHandler`, so you need to specify flags from `AnalysisHandler` constructor manually
Please note that handler's constructor should have shape `(TestServices) -> MyHandler`, so you need to specify flags from `AnalysisHandler` constructor manually.
### Handler tools
@@ -255,25 +255,25 @@ Header of dump of specific module can be configured in constructor of `MultiModu
Handlers of type 3. (which want to render something inside original test file) can not use simple file dumps because:
1. There can be multiple handlers which want to report something and we need to combine their dumps in single file
2. Before running test someone need to clean all dumps from original testdata, otherwise testfile can be incorrect and test fails
2. Before running test someone needs to clean all dumps from original testdata, otherwise testfile can be incorrect and test fails
To handle this two problems there is an additional infrastructure which uses `CodeMetaInfo` and `GlobalMetadataInfoHandler`
To handle these two problems there is additional infrastructure which uses `CodeMetaInfo` and `GlobalMetadataInfoHandler`.
#### CodeMetaInfo
[CodeMetaInfo](../test-infrastructure-utils/tests/org/jetbrains/kotlin/codeMetaInfo/model/CodeMetaInfo.kt) is a base abstraction for any kind of information you want to render. Basically it contains start and end offsets in original file, `tag` which is main name of meta indo, attributes (additional arguments of meta info) and `renderConfiguration`, which describes how this meta info should be renderered in code. Default syntax for meta info is `<!TAG[attr1, attr2]!>text of original code<!>` (`[attr]` part will be ommitted if attributes are empty).
[CodeMetaInfo](../test-infrastructure-utils/tests/org/jetbrains/kotlin/codeMetaInfo/model/CodeMetaInfo.kt) is a base abstraction for any kind of information you want to render. Basically it contains start and end offsets in original file, `tag` which is main name of meta info, attributes (additional arguments of meta info) and `renderConfiguration`, which describes how this meta info should be rendered in code. Default syntax for meta info is `<!TAG[attr1, attr2]!>text of original code<!>` (`[attr]` part will be omitted if attributes are empty).
#### GlobalMetadataInfoHandler
[GlobalMetadataInfoHandler](../test-infrastructure/tests/org/jetbrains/kotlin/test/services/GlobalMetadataInfoHandler.kt) is a service which is used for working with meta infos from handlers. It serves to two purposes:
1. Parsing meta infos in original testdata, stripping them from it before passing code to steps and provide info about existing meta infos to handlers (`getExistingMetaInfosForFile` method)
[GlobalMetadataInfoHandler](../test-infrastructure/tests/org/jetbrains/kotlin/test/services/GlobalMetadataInfoHandler.kt) is a service which is used for working with meta infos from handlers. It serves two purposes:
1. Parsing meta infos in original testdata, stripping them from it before passing code to steps and providing info about existing meta infos to handlers (`getExistingMetaInfosForFile` method)
2. Collecting meta infos from all handlers, rendering all of them to original test file and comparing it with expected test file on disk
So if your handler wants to report meta infos all it need is just to create meta info instances and pass them to `GlobalMetadataInfoHandler` using `addMetadataInfosForFile` method (`GlobalMetadataInfoHandler` is a test service and is accessible via `testServices.globalMetadataInfoHandler`). Also you need to enable `GlobalMetadataInfoHandler` in test using `enableMetaInfoHandler()` method in test configuration DSL
So if your handler wants to report meta infos, all it needs is to create meta info instances and pass them to `GlobalMetadataInfoHandler` using `addMetadataInfosForFile` method (`GlobalMetadataInfoHandler` is a test service and is accessible via `testServices.globalMetadataInfoHandler`). Also you need to enable `GlobalMetadataInfoHandler` in test using `enableMetaInfoHandler()` method in test configuration DSL.
# Writing your own test. Test configuration DSL
One of main ideas of this test infrastructure is provide ability to define tests in declarative way: describe only _what_ will happen in test (what will be configured, which facades and handlers will be run), not _how_ it will be. To achieve this there was developed special DSL, which is used to description of test. Whole configuration of test is defined in class [TestConfiguration](tests/org/jetbrains/kotlin/test/TestConfiguration.kt), and there is also a [TestConfigurationBuilder](../tests-common-new/tests/org/jetbrains/kotlin/test/builders/TestConfigurationBuilder.kt) class which defines DSL for configuring all parts of test configuration. Here I highlight only most important parts of DSl, full specification you can read in code of `TestConfigurationBuilder`
One of main ideas of this test infrastructure is provide ability to define tests in declarative way: describe only _what_ will happen in test (what will be configured, which facades and handlers will be run), not _how_ it will be. To achieve this, a special DSL was developed, which is used to describe a test. Whole configuration of test is defined in class [TestConfiguration](tests/org/jetbrains/kotlin/test/TestConfiguration.kt), and there is also a [TestConfigurationBuilder](../tests-common-new/tests/org/jetbrains/kotlin/test/builders/TestConfigurationBuilder.kt) class which defines DSL for configuring all parts of test configuration. Here I highlight only most important parts of DSL, you can read the full specification in code of `TestConfigurationBuilder`.
- `defaultDirectives` allows defining directives which will be enabled in tests by default. It supports all kinds of directives:
```kotlin
@@ -291,14 +291,14 @@ defaultDirectives {
- `useAdditionalService` for some custom service
- `facadeStep` to register new facade step
- `handlersStep` and `namedHandlersStep` to register new handlers step
- those methods takes lambda in which you can call `useHandlers` method to register specific handlers
- these methods take lambda in which you can call `useHandlers` method to register specific handlers
- note that all steps will be executed in order of definition
- if you create `namedHandlerStep` you can add additional handlers to it using `configureNamedHandlersStep` latter; it can be useful in case you declare steps in one method (to share this pipeline between different tests) and configure them in specific test runners
- if you create `namedHandlerStep` you can add additional handlers to it using `configureNamedHandlersStep` later; it can be useful in case you declare steps in one method (to share this pipeline between different tests) and configure them in specific test runners
- `enableMetaInfoHandler` for enabling `GlobalMetadataInfoHandler`
- `forTestsMatching` and `forTestsNotMatching` are methods which can be used to apply some configuration only if path to testdata file matches/not matches with regular expression which was passed to this method. Those methods take lambda with `TestConfigurationBuilder` receiver so all methods listed above are accessible in it
- those methods has to overloads: one take `Regex` and second takes `String`, which is converted to `Regex` by simply replacing all `*` symbols with `.*` pattern
- `forTestsMatching` and `forTestsNotMatching` are methods which can be used to apply some configuration only if path to testdata file matches/not matches with regular expression which was passed to this method. These methods take lambda with `TestConfigurationBuilder` receiver so all methods listed above are accessible in it
- these methods have two overloads: first takes `Regex` and second takes `String`, which is converted to `Regex` by simply replacing all `*` symbols with `.*` pattern
Almost all methods of DSL takes `Constructor<SomeService>` as parameter, and `Constructor<T>` is just typealias to `(TestService) -> T`. If your service has constructor of such shape you can just pass callable reference to it (`useHandlers(::MyHandler)`). If your service is parametrized and has some additional parameter you can pass this parameter to service using function `bind`:
Almost all methods of DSL take `Constructor<SomeService>` as parameter, and `Constructor<T>` is just typealias to `(TestService) -> T`. If your service has constructor of such shape you can just pass callable reference to it (`useHandlers(::MyHandler)`). If your service is parametrized and has some additional parameter you can pass this parameter to service using function `bind`:
```kotlin
class MyHandler(testServices: TestServices, val someFlag: Boolean) : AnalysisHandler...
@@ -314,7 +314,7 @@ fun <T, R> ((TestServices, T) -> R).bind(value: T): Constructor<R> {
## AbstractKotlinCompilerTest
[AbstractKotlinCompilerTest](../tests-common-new/tests/org/jetbrains/kotlin/test/runners/AbstractKotlinCompilerTest.kt) is a base class for all kotlin compiler tests. It defines some default configuration and provides simple abstract method to implement `abstract fun TestConfigurationBuilder.configuration()` in inheritors. Whole test configuration should be described in override of this method
[AbstractKotlinCompilerTest](../tests-common-new/tests/org/jetbrains/kotlin/test/runners/AbstractKotlinCompilerTest.kt) is a base class for all Kotlin compiler tests. It defines some default configuration and provides simple abstract method to implement `abstract fun TestConfigurationBuilder.configuration()` in inheritors. Whole test configuration should be described in override of this method
```kotlin
abstract class MyAbstractTestRunner : AbstractKotlinCompilerTest() {
@@ -324,7 +324,7 @@ abstract class MyAbstractTestRunner : AbstractKotlinCompilerTest() {
}
```
If you have hierarchy of test runners then there is no simple way to override `configuration()` method again and call `super.configuration` (because kotlin unfortunately can not call to super member with extension receiver), so for such cases you should use next workaround:
If you have hierarchy of test runners then there is no simple way to override `configuration()` method again and call `super.configuration` (because Kotlin unfortunately [cannot](https://youtrack.jetbrains.com/issue/KT-11488) call to super member with extension receiver), so for such cases you should use the following workaround:
```kotlin
abstract class MyAnotherAbstractTestRunner : MyAbstractTestRunner() {
@@ -341,4 +341,4 @@ abstract class MyAnotherAbstractTestRunner : MyAbstractTestRunner() {
Please keep your abstract test runners as simple as possible. Ideally each abstract test runner should contain **only** test configuration with DSL and nothing else. All services implementations should be declared in separate files.
Also please keep structure of packages. Abstract test runners lays in `runners` package, services in `services`, handlers in `handlers` etc
Also please keep structure of packages. Abstract test runners are located in package `runners`, services in `services`, handlers in `handlers` etc.