Currently, we shrink classpath snapshots at 2 steps:
- Classpath diffing: Shrink the current classpath snapshot against
the previous lookup symbols
- Classpath snapshot saving: Shrink the current classpath snapshot
against the current lookup symbols
With this commit, the shrinking at the second step is now incremental.
The shrinking at the first step is still non-incremental.
for performance reasons: (1) the snapshots are too big, and (2) they are
usually updated only at the end of the task execution--in a failed task
run, they are usually unchanged and therefore don't need to be restored.
Also visit a class file with ASM once to extract all information we need
in advance, instead of visiting the class file each time some piece of
info is needed.
as we need access to the lookup tracker to compute classpath changes
more efficiently and reduce the size of the saved classpath snapshot.
The previous commit only changed the files' paths, this
commit actually updates the files' contents.
Note that classpath snapshotting still happens in Gradle artifact
transforms. (However, the previous commit also moved the code for
classpath snapshotting together with the code for classpath diffing as
they are closely related.)
This is faster than the current approach which creates
`JavaClassDescriptor`s, converts them to protos, and then snapshots
these protos.
- Refactor unit tests to faciliate further changes
- moves test data to a directory that matches the tests' package name
- moves expected snapshots to a separate directory
- adds public and private fields/properties to sample class
- Compute changes between ASM-based Java class snapshots
- Don't collect members of an added Java class as changes
as it's enough to report the name of the added Java class as changed (we
also do that for added Kotlin classes and Kotlin/Java removed classes).
- Add unit tests for impact analysis in advance
- Compute impacted symbols of changed symbols
Also do not collect added classes/class members as they don't impact
recompilation.
-Use ClassId when computing Java class changes
It is more precise than JvmClassName, which can be ambiguous around the
`$` character (e.g., ClassId "com/example/A$B.C" and "com/example/A.B$C"
both have the same JvmClassName "com/example/A$B$C").
- Compute impacted set of changed symbols across Kotlin and Java
- Add unit tests for impact analysis across Kotlin and Java
- Compute supertypes of Kotlin classes during snapshotting
- Handle inner classes when computing list of changed symbols.
For the reported symbols, always check all options:
class member, inner class, top level class, top level member.
Test: IncrementalJavaChangeClasspathSnapshotIT.testAddingInnerClass
to avoid unnecessarily loading unchanged ones.
Duplicate classes will make this a bit tricky. This commit outlines the
algorithm to handle them, the full implementation will follow later.
Also handle removed classes when computing classpath changes.
Test: New tests in ClasspathChangesComputerTest
Workers usually execute later than task action and does not raise
exception to the task. Such behaviour skips task outputs restoration to
the pre-execution state.
Kotlin compiler on incremental compilation error leaves 'dirty*' files
in the build output, forcing Gradle to run non-incrementally on the
next build. Fixed it by moving task outputs reset logic into worker as
well.
^KT-46406 In Progress
There are certain classes that we are not yet able to take snapshots of,
either because the class is faulty, or there is a bug in our code.
For these classes, we will use a special snapshot and fall back to the
existing approach to compute classpath changes.
In the short run, we will update this list of cases as they arise. In
the long run, we will:
- Fix all cases that are caused by bugs in our code.
- Decide what to with the faulty jars in general and remove the list.
Test: The following should pass
- Step 1: ./gradlew publish
- Step 2: ./gradlew publish -Pbootstrap.local=true
-Pbootstrap.local.path=/path/to/kotlin/build/repo
-Pkotlin.incremental.useClasspathSnapshot=true
Simplify API of InlineConstTracker
In ConstLowering: Move transformer logic to another class to avoid mutable state. Avoid marking all files in module as module. Support of inner classes.
#KT-46506 Fixed
Create `JavaClassDescriptor`s for Java classes
Ignore anonymous and synthetic classes
as they can't impact recompilation.
Clean up handling of local and anonymous classes
Bug: KT-45777
Test: New JavaClassDescriptorCreatorTest
Now, milestones of major Kotlin releases (e.g. 1.6.0-M1) will NOT
generate prerelease binaries anymore.
The reason for that is it's proven to be quite a complicated process to
turn on IR_PRE_RELEASE after the release is branched, perform double
bootstrap and fix tests, and then do it again in reverse just before
release. With the new release cadence, we don't have that much time to
do it and verify that everything works as intended.
Note that this only removes the "global" prerelease flag. Compiler will
still generate prerelease binaries if a non-stable language version is
used. For example, Kotlin 1.6.0-M1 with `-language-version 1.7` will
generate prerelease binaries.
Reuse the existing IncrementalJvmCache to take snapshots and compute
changes for Kotlin classes.
Java classes will be handled next.
Bug: KT-45777
Test: New ClasspathSnapshotterTest, ClasspathChangesComputerTest
Extract minimal required information into the new `KotlinClassInfo`
class. This info will be part of the classpath snapshot of the
`KotlinCompile` task in an upcoming change.
Bug: KT-45777
Test: Existing tests should pass (this is a refactoring-only change)
This commit wires necessary components only. The actual classpath
changes will be provided later.
Bug: KT-45777
Test: Existing IncrementalCompilationClasspathSnapshotJvmMultiProjectIT
and IncrementalJavaChangeClasspathSnapshotIT
Reduce size of lookup map after rebuild, reduce waiting time by replacing operations of read+write with append, also split remove garbage process into smaller operations in get
#KT-46804 Fixed
Optimizations may remove the `x$delegate` field, in which case there
should be a `getX$delegate` method to reconstruct the value if needed
due to a call to `KProperty{0,1,2}.getDelegate`. We need to know the
signature of this method somehow.
FileSignature, CompositeSignature, LocalSignature
They are needed to make possible reference any non-local declaration via
signature, including private signature, type parameters and so on.
- Support those new signatures in proto and klibs
- Rename `isPublic` -> `isPubliclyVisible` due to changed semantic
- Fix FIR
- clean up code
Now compiler flags with boolean values which are controlled by system
properties could be set in similar way. The following syntax is
appropriate. For setting flag:
'-DflagName,'-DflagName=[y|Y|true|TRUE|yes|YES|on|ON]'
for unsetting flag:
'-DflagName=[n|N|false|FALSE|no|NO|off|OFF]'
Disable Idea JPS build mechanism for marking all sources of common output if one of them is dirty
Add source-to-outputs map for correctly removing Kotlin/JS outputs
#KT-45763 Fixed
#KT-44351 Fixed
Vyacheslav Gerasimov