gradle-plugin: Use cinterop to compile stubs

The gradle plugin performed interop processing in two steps:
 1. generate stubs using cinterop tool
 2. compile the stubs generated using konanc.

This led to code duplication since the cinterop tool also
can do the same things. This patch removes two tasks used
different tools and replaces them with one task which uses
citerop tool for both stub generation and compilation
This commit is contained in:
Ilya Matveev
2017-09-07 17:20:39 +03:00
committed by ilmat192
parent 4fc0fbe73f
commit 016893991a
10 changed files with 72 additions and 175 deletions
+20 -50
View File
@@ -33,8 +33,8 @@ project property (e.g. in `gradle.properties`). Note: the plugin ignores the `ko
In this case the compiler will not be downloaded by the plugin.
To use the plugin you need to define artifacts you want to build in the `konanArtifacts` block. Here you can specify
source files and compilation parameters (e.g. a target platform) for each artifact (see [**Plugin DSL**](#plugin-dsl) section below for
details). The plugin uses `src/main/kotlin/` as a default directory for sources.
source files and compilation parameters (e.g. a target platform) for each artifact (see [**Plugin DSL**](#plugin-dsl)
section below for details). The plugin uses `src/main/kotlin/` as a default directory for sources.
konanArtifacts {
foo {
@@ -65,28 +65,11 @@ Each element (interop) in this block corresponds to some native library describe
[`INTEROP.md`](INTEROP.md) to read more about def-files). The default path to the def-file of an interop is
`src/main/c_interop/<interop-name>.def`.
Each interop is processed by the plugin in two steps:
Each library is processed by the `cinterop` tool (see [`INTEROP.md`](INTEROP.md) for details). You can specify
additional parameters for this tool in the interop block (e.g. a custom def-file is specified in the example above).
See [**Plugin DSL**](#plugin-dsl) section below for available parameters.
1. **Stub generation.** In this step the plugin uses `cinterop` tool to generate Kotlin and bitcode (*.bc) stubs for the
library. You can specify additional parameters for this tool in the interop block (e.g. a custom def-file is specified
in the example above). See [**Plugin DSL**](#plugin-dsl) section below for available parameters and
[`INTEROP.md`](INTEROP.md) for `cinterop` tool description.
2. **Stub compilation.** In this step the plugin uses Kotlin/Native compiler to compile the stubs generated by the step (1)
into a klib which can be linked with a Kotlin/Native application.
Usually there is no need to specify additional compilation options for this stage. But if for some reason you need to do
it, you can use `compileStubsConfig` property of an interop. This property provides an access to a `KonanCompileConfig`
object containing the same methods as an artifact in the `konanArtifacts` block. E.g. this sample enables time measurement
for compilation of stubs:
```
konanInterop {
stdio {
compileStubsConfig.measureTime true
}
}
```
To build some artifact with an interop add `useInterop` command in the artifact section:
To build an artifact using an interop add `useInterop` command in the artifact section:
konanArtifacts {
CsvParser {
@@ -167,40 +150,32 @@ You may get this task using the `compilationTask` property of an artifact or by
|`enableOptimization`|`boolean` |Is the optimization enabled |
|`enableAssertions `|`boolean` |Is the assertion support enabled |
|`measureTime `|`boolean` |Does the compiler print phase time |
* __gen*InteropName*InteropStubs__. The plugin creates such a task for each an interop defined in a `konanInterop` block.
You may get this task using `generateStubsTask` property of an interop object or by its name:
* __process*InteropName*Interop__. The plugin creates such a task for each an interop defined in a `konanInterop` block.
You may get this task using `interopProcessingTask` property of an interop object or by its name:
```
// The task name is "genFooInteropStubs"
konanInterop['foo'].generateStubsTask
// The task name is "processFooInterop"
konanInterop['foo']. interopProcessingTask
```
Such a task generates stubs for the library defined by the interop and has the following properties accessible
from a build script
Such a task processes the library defined by the interop and creates a *.klib for it. The task has the following
properties accessible from a build script:
|Property |Type |Description |
|-------------- |----------------------------|--------------------------------------------------------|
|`stubsDir `|`File` |An output directory for the Kotlin stubs generated |
|`libsDir `|`File` |An output directory for the compiled stubs |
|`outputDir `|`File` |An output directory for the *.klib built |
|`kLib `|`File` |The *.klib built |
|`defFile `|`File` |Def-file used by the interop |
|`compilerOpts `|`List<String>` |Additional options passed to clang |
|`linkerOpts `|`List<String>` |Additional options passed to a linker |
|`headers `|`Collection<FileCollection>`|Additional headers used for stub generation |
|`linkFiles `|`Collection<FileCollection>`|Additional files linked with the stubs |
|`measureTime `|`boolean` |Does the compiler print phase time for stubs compilation|
* __compile*InteropName*InteropStubs__. The plugin creates such a task for each interop defined in a `konanInterop` block.
You may get this task using `compileStubsTask` property of an interop object or by its name:
```
// The task name is "compileStdioInteropStubs"
konanInterop['stdio'].compileStubsTask
```
This task uses the Kotlin/Native compiler to compile the Kotlin stubs generated by the previous task. The task has the
same properties as __compileKonan*ArtifactName*__.
_Note_: In versions before 0.4 two tasks for each an interop were used: __gen*InteropName*InteropStubs__ and
__compile*InteropName*InteropStubs__. Now actions of both of them are performed by the
__process*InteropName*Interop__ task described above.
* __compileKonan__. This task is dependent on all compilation tasks and allows one to build all the artifacts supported
by the current host. The task has no properties to use by a build script.
@@ -238,11 +213,9 @@ For this project the task graph will be the following:
build
compileKonan
compileKonanFooArtifact
compileFooInteropStubs
genFooInteropStubs
processFooInterop
compileKonanBarArtifact
compileBarInteropStubs
genBarInteropStubs
processBarInterop
clean
## Plugin DSL
@@ -314,9 +287,6 @@ For this project the task graph will be the following:
link <files which will be linked with native stubs> // Additional files to link with native stubs.
dumpParameters true // Print all parameters during the build.
// Print time of stub compilation phases (equivalent of the `--time` command line option).
measureTime true
// Add the `anotherTask` to the stub generation task dependencies.
dependsOn anotherTask
+1 -1
View File
@@ -29,6 +29,6 @@ task downloadTensorflow(type: Exec) {
commandLine './downloadTensorflow.sh'
}
genTensorflowInteropStubs {
processTensorflowInterop {
dependsOn 'downloadTensorflow'
}
@@ -18,7 +18,6 @@ package org.jetbrains.kotlin.gradle.plugin
import org.gradle.api.Named
import org.gradle.api.Project
import org.gradle.api.Task
import org.gradle.api.file.FileCollection
import org.gradle.api.plugins.BasePlugin
import org.gradle.api.tasks.*
@@ -75,7 +74,7 @@ open class KonanCompileTask: KonanTargetableTask() {
val linkerOpts: List<String>
@Input get() = mutableListOf<String>().apply {
addAll(_linkerOpts)
interops.flatMapTo(this) { it.generateStubsTask.linkerOpts }
interops.flatMapTo(this) { it.interopProcessingTask.linkerOpts }
}
@Input var enableDebug = project.properties.containsKey("enableDebug") && project.properties["enableDebug"].toString().toBoolean()
@@ -134,7 +133,7 @@ open class KonanCompileTask: KonanTargetableTask() {
@TaskAction
fun compile() {
project.file(outputDir).mkdirs()
outputDir.mkdirs()
if (dumpParameters) dumpProperties(this@KonanCompileTask)
@@ -163,15 +162,10 @@ open class KonanCompileConfig(
// DSL methods. Interop. --------------------------------------------------
fun useInterop(interop: KonanInteropConfig) = with(compilationTask) {
val generateStubsTask = interop.generateStubsTask
val compileStubsTask = interop.compileStubsTask
val generateStubsTask = interop.interopProcessingTask
dependsOn(compileStubsTask)
dependsOn(generateStubsTask)
library(project.files(compileStubsTask.artifact))
nativeLibrary(project.fileTree(generateStubsTask.libsDir).apply {
include("**/*.bc")
})
library(project.files(generateStubsTask.kLib))
interops.add(interop)
}
@@ -18,7 +18,6 @@ package org.jetbrains.kotlin.gradle.plugin
import org.gradle.api.Named
import org.gradle.api.Project
import org.gradle.api.Task
import org.gradle.api.file.FileCollection
import org.gradle.api.plugins.BasePlugin
import org.gradle.api.tasks.*
@@ -33,17 +32,15 @@ open class KonanInteropTask: KonanTargetableTask() {
dependsOn(project.konanCompilerDownloadTask)
this.libName = libName
this.defFile = project.konanDefaultDefFile(libName)
this.kLib = outputDir.resolve("$libName.klib")
}
// Output directories -----------------------------------------------------
/** Directory with autogenerated interop stubs (*.kt) */
@OutputDirectory
val stubsDir = project.file("${project.konanInteropStubsOutputDir}/$name")
@Internal val outputDir = project.file(project.konanInteropOutputDir)
/** Directory with library bitcodes (*.bc) */
@OutputDirectory
val libsDir = project.file("${project.konanInteropLibsOutputDir}/$name")
lateinit var kLib: File
@OutputFile get
// Interop stub generator parameters -------------------------------------
@@ -56,9 +53,6 @@ open class KonanInteropTask: KonanTargetableTask() {
@Input lateinit var libName: String
internal set
@Optional @Input var manifest: String? = null
internal set
@Input var dumpParameters = false
internal set
@@ -74,17 +68,15 @@ open class KonanInteropTask: KonanTargetableTask() {
@TaskAction
fun exec() {
outputDir.mkdirs()
if (dumpParameters) dumpProperties(this@KonanInteropTask)
KonanInteropRunner(project).run(buildArgs())
}
protected fun buildArgs() = mutableListOf<String>().apply {
addArg("-properties", "${project.konanHome}/konan/konan.properties")
addArg("-flavor", "native")
addArg("-generated", stubsDir.canonicalPath)
addArg("-natives", libsDir.canonicalPath)
manifest ?.let {addArg("-manifest", it)}
addArg("-o", kLib.canonicalPath)
addArgIfNotNull("-target", target)
addArgIfNotNull("-def", defFile.canonicalPath)
@@ -119,82 +111,66 @@ open class KonanInteropConfig(
// Child tasks ------------------------------------------------------------
// Task to process the library and generate stubs
val generateStubsTask: KonanInteropTask = project.tasks.create(
"gen${name.capitalize()}InteropStubs",
val interopProcessingTask: KonanInteropTask = project.tasks.create(
"process${name.capitalize()}Interop",
KonanInteropTask::class.java) {
it.init(name)
it.manifest = "${it.stubsDir.path}/manifest.properties"
it.group = BasePlugin.BUILD_GROUP
it.description = "Generates stubs for the Kotlin/Native interop '$name'"
}
// Config and task to compile *.kt stubs into a library
internal val compileStubsConfig = KonanCompileConfig("${name}InteropStubs", project, "compile").apply {
compilationTask.dependsOn(generateStubsTask)
outputDir("${project.konanInteropCompiledStubsDir}/$name")
produce("library")
inputFiles(project.fileTree(generateStubsTask.stubsDir))
manifest("${generateStubsTask.stubsDir.path}/manifest.properties")
compilationTask.group = BasePlugin.BUILD_GROUP
compilationTask.description = "Compiles stubs for the Kotlin/Native interop '${this@KonanInteropConfig.name}'"
}
val compileStubsTask = compileStubsConfig.compilationTask
// DSL methods ------------------------------------------------------------
fun defFile(file: Any) = with(generateStubsTask) {
fun defFile(file: Any) = with(interopProcessingTask) {
defFile = project.file(file)
}
fun pkg(value: String) = with(generateStubsTask) {
fun pkg(value: String) = with(interopProcessingTask) {
pkg = value
}
fun target(value: String) = with(generateStubsTask) {
generateStubsTask.target = value
compileStubsTask.target = value
fun target(value: String) = with(interopProcessingTask) {
interopProcessingTask.target = value
}
fun compilerOpts(vararg values: String) = with(generateStubsTask) {
fun compilerOpts(vararg values: String) = with(interopProcessingTask) {
compilerOpts.addAll(values)
}
fun header(file: Any) = headers(file)
fun headers(vararg files: Any) = with(generateStubsTask) {
fun headers(vararg files: Any) = with(interopProcessingTask) {
headers.add(project.files(files))
}
fun headers(files: FileCollection) = with(generateStubsTask) {
fun headers(files: FileCollection) = with(interopProcessingTask) {
headers.add(files)
}
fun includeDirs(vararg values: String) = with(generateStubsTask) {
fun includeDirs(vararg values: String) = with(interopProcessingTask) {
compilerOpts.addAll(values.map { "-I$it" })
}
fun linkerOpts(vararg values: String) = linkerOpts(values.toList())
fun linkerOpts(values: List<String>) = with(generateStubsTask) {
fun linkerOpts(values: List<String>) = with(interopProcessingTask) {
linkerOpts.addAll(values)
}
fun link(vararg files: Any) = with(generateStubsTask) {
fun link(vararg files: Any) = with(interopProcessingTask) {
linkFiles.add(project.files(files))
}
fun link(files: FileCollection) = with(generateStubsTask) {
fun link(files: FileCollection) = with(interopProcessingTask) {
linkFiles.add(files)
}
fun dumpParameters(value: Boolean) = with(generateStubsTask) {
fun dumpParameters(value: Boolean) = with(interopProcessingTask) {
dumpParameters = value
compileStubsTask.dumpParameters = value
}
fun measureTime(value: Boolean) = compileStubsConfig.measureTime(value)
fun dependsOn(dependency: Any) = generateStubsTask.dependsOn(dependency)
fun dependsOn(dependency: Any) = interopProcessingTask.dependsOn(dependency)
fun extraOpts(vararg values: Any) = extraOpts(values.asList())
fun extraOpts(values: List<Any>) {
values.mapTo(generateStubsTask.extraOpts) { it.toString() }
values.mapTo(interopProcessingTask.extraOpts) { it.toString() }
}
}
@@ -52,9 +52,7 @@ internal val Project.konanHome: String
internal val Project.konanBuildRoot get() = "${buildDir.canonicalPath}/konan"
internal val Project.konanCompilerOutputDir get() = "${konanBuildRoot}/bin"
internal val Project.konanInteropStubsOutputDir get() = "${konanBuildRoot}/interopStubs"
internal val Project.konanInteropCompiledStubsDir get() = "${konanBuildRoot}/interopCompiledStubs"
internal val Project.konanInteropLibsOutputDir get() = "${konanBuildRoot}/nativelibs"
internal val Project.konanInteropOutputDir get() = "${konanBuildRoot}/c_interop"
internal val Project.konanDefaultSrcFiles get() = fileTree("${projectDir.canonicalPath}/src/main/kotlin")
internal fun Project.konanDefaultDefFile(libName: String)
@@ -171,8 +169,8 @@ internal fun dumpProperties(task: Task) {
is KonanInteropTask -> {
println()
println("Stub generation task: ${task.name}")
println("stubsDir : ${task.stubsDir}")
println("libsDir : ${task.libsDir}")
println("outputDir : ${task.outputDir}")
println("kLib : ${task.kLib}")
println("defFile : ${task.defFile}")
println("target : ${task.target}")
println("pkg : ${task.pkg}")
@@ -1,9 +1,10 @@
package org.jetbrains.kotlin.gradle.plugin
import org.gradle.api.Named
import org.gradle.api.Project
import org.gradle.api.file.FileCollection
internal interface KonanToolRunner {
internal interface KonanToolRunner: Named {
val mainClass: String
val classpath: FileCollection
val jvmArgs: List<String>
@@ -13,8 +14,10 @@ internal interface KonanToolRunner {
fun run(vararg args: String) = run(args.toList())
}
internal abstract class KonanBaseRunner(val project: Project): KonanToolRunner {
abstract val toolName: String
internal abstract class KonanCliRunner(val toolName: String, val fullName: String, val project: Project): KonanToolRunner {
override val mainClass = "org.jetbrains.kotlin.cli.utilities.MainKt"
override fun getName() = fullName
override val classpath: FileCollection =
project.fileTree("${project.konanHome}/konan/lib/")
@@ -26,8 +29,9 @@ internal abstract class KonanBaseRunner(val project: Project): KonanToolRunner {
override val environment = mutableMapOf("LIBCLANG_DISABLE_CRASH_RECOVERY" to "1")
override fun run(args: List<String>) {
project.logger.info("Run tool: $toolName with args: ${args.joinToString(separator = " ")}")
if (classpath.isEmpty) {
throw IllegalStateException("Classpath if the tool is empty: $toolName\n" +
throw IllegalStateException("Classpath of the tool is empty: $toolName\n" +
"Probably the 'konan.home' project property contains an incorrect path.\n" +
"Please change it to the compiler root directory and rerun the build.")
}
@@ -36,22 +40,15 @@ internal abstract class KonanBaseRunner(val project: Project): KonanToolRunner {
it.main = mainClass
it.classpath = classpath
it.jvmArgs(jvmArgs)
it.args(args.apply {
project.logger.info("Run tool: $toolName with args: ${this.joinToString(separator = " ")}")
})
it.args(listOf(toolName) + args)
it.environment(environment)
}
}
}
internal class KonanInteropRunner(project: Project) : KonanBaseRunner(project){
internal companion object {
const val INTEROP_MAIN = "org.jetbrains.kotlin.native.interop.gen.jvm.MainKt"
}
override val toolName get() = "Kotlin/Native cinterop tool"
override val mainClass get() = INTEROP_MAIN
internal class KonanInteropRunner(project: Project)
: KonanCliRunner("cinterop", "Kotlin/Native cinterop tool", project)
{
init {
if (project.host == "mingw") {
environment.put("PATH", "${project.konanHome}\\dependencies" +
@@ -61,11 +58,5 @@ internal class KonanInteropRunner(project: Project) : KonanBaseRunner(project){
}
}
internal class KonanCompilerRunner(project: Project) : KonanBaseRunner(project) {
internal companion object {
const val COMPILER_MAIN = "org.jetbrains.kotlin.cli.bc.K2NativeKt"
}
override val toolName get() = "Kotlin/Native compiler"
override val mainClass get() = COMPILER_MAIN
}
internal class KonanCompilerRunner(project: Project) : KonanCliRunner("konanc", "Kotlin/Native compiler", project)
internal class KonnaKlibRunner(project: Project) : KonanCliRunner("klib", "Klib management tool", project)
@@ -266,7 +266,7 @@ class IncrementalSpecification extends BaseKonanSpecification {
""".stripIndent())
}
def results = buildTwice(project) { KonanInteropProject it ->
it.addInteropSetting("stdio", "generateStubsTask.dependsOn", "konanArtifacts['lib'].compilationTask")
it.addInteropSetting("stdio", "interopProcessingTask.dependsOn", "konanArtifacts['lib'].compilationTask")
it.addInteropSetting("stdio", "link", "files(konanArtifacts['lib'].compilationTask.artifactPath)")
}
@@ -273,7 +273,7 @@ class KonanInteropProject extends KonanProject {
}
""".stripIndent()
)
interopTasks = [defaultStubGenerationTask(), defaultStubCompilationTask()]
interopTasks = [defaultStubGenerationTask()]
compilationTasks = [defaultCompilationTask(), ":compileKonan", ":build"]
return result
}
@@ -312,16 +312,8 @@ class KonanInteropProject extends KonanProject {
return stubGenerationTask(DEFAULT_INTEROP_NAME)
}
String defaultStubCompilationTask() {
return stubCompilationTask(DEFAULT_INTEROP_NAME)
}
String stubGenerationTask(String interopName) {
return ":gen${interopName.capitalize()}InteropStubs"
}
String stubCompilationTask(String interopName) {
return ":compile${interopName.capitalize()}InteropStubs"
return ":process${interopName.capitalize()}Interop"
}
/** Creates a project with default build, source and def files. */
@@ -18,14 +18,8 @@ class PathSpecification extends BaseKonanSpecification {
new File("$konan/bin").listFiles().findAll {
File it -> it.file && it.name.matches('^main\\.[^.]+')
}.size() > 0
def klib = new File("$konan/interopCompiledStubs/stdioInteropStubs/stdioInteropStubs.klib")
def klib = new File("$konan/c_interop/stdio.klib")
klib.exists() && klib.file
def stdioKt = new File("$konan/interopStubs/genStdioInteropStubs/stdio/stdio.kt")
stdioKt.exists() && stdioKt.file
def manifest = new File("$konan/interopStubs/genStdioInteropStubs/manifest.properties")
manifest.exists() && manifest.file
def nativeLib = new File("$konan/nativelibs/genStdioInteropStubs/stdiostubs.bc")
nativeLib.exists() && nativeLib.file
}
def 'Plugin should stop building if the compiler classpath is empty'() {
@@ -44,7 +44,7 @@ class TaskSpecification extends BaseKonanSpecification {
interop { }
}
""".stripIndent())
it.interopTasks = [":interop:genInteropInteropStubs", ":interop:compileInteropInteropStubs"]
it.interopTasks = [":interop:processInteropInterop"]
it.generateDefFile("interop.def")
}
def result = rootProject.createRunner().withArguments("build").build()
@@ -73,40 +73,22 @@ class TaskSpecification extends BaseKonanSpecification {
result.task(":printArgs") != null
result.task(":printArgs").outcome == TaskOutcome.SUCCESS
def expectedKlibPath = project.konanBuildDir.toPath()
.resolve("interopCompiledStubs${File.separator}stdioInteropStubs${File.separator}stdioInteropStubs.klib")
.toFile().canonicalPath
def expectedBcPath = project.konanBuildDir.toPath()
.resolve("nativelibs${File.separator}genStdioInteropStubs${File.separator}stdiostubs.bc")
.resolve("c_interop${File.separator}stdio.klib")
.toFile().canonicalPath
def ls = System.lineSeparator()
result.output.contains("[-lpthread]$ls[$expectedKlibPath]$ls[$expectedBcPath]".stripIndent().trim())
result.output.contains("[-lpthread]$ls[$expectedKlibPath]".stripIndent().trim())
}
def 'Compiler should print time measurements if measureTime flag is set'() {
when:
def project = KonanInteropProject.createEmpty(projectDirectory)
project.generateSrcFile("main.kt")
project.addInteropSetting("measureTime", "true")
project.addCompilationSetting("measureTime", "true")
def result = project.createRunner().withArguments('build').build()
then:
result.output.findAll(~/FRONTEND:\s+\d+\s+msec/).size() == 2
result.output.findAll(~/BACKEND:\s+\d+\s+msec/).size() == 2
result.output.findAll(~/LINK_STAGE:\s+\d+\s+msec/).size() == 1
}
def 'Interop should provide access to stub compilation config'() {
when:
def project = KonanInteropProject.createEmpty(projectDirectory)
project.generateSrcFile("main.kt")
project.addInteropSetting("compileStubsConfig.measureTime", "true")
def result = project.createRunner().withArguments('build').build()
then:
result.output.findAll(~/FRONTEND:\s+\d+\s+msec/).size() == 1
result.output.findAll(~/BACKEND:\s+\d+\s+msec/).size() == 1
result.output.findAll(~/LINK_STAGE:\s+\d+\s+msec/).size() == 0
result.output.findAll(~/LINK_STAGE:\s+\d+\s+msec/).size() == 1
}
}