Sorting invokes descriptor renderer, so the fewer descriptors we are
sorting, the better. This helps to save a few percent of compilation
time in a synthetic project from KT-52326.
- Remove soft references that are too old.
- Compute memory usage based on Runtime.totalMemory() instead of
Runtime.maxMemory() as the latter is not reliable.
- Remove debug logs that haven't proved to be useful.
- Collect metrics on the number of cache hits (in addition to cache
misses).
#KT-52329 In Progress
[New IC] Reduce memory usage of classpath snapshot cache
Use object interning for commonly shared objects. These include:
- supertypes of classes
- package names of classes
One experiment showed that with the above optimization, memory usage was
reduced from 660 MB down to 280 MB (+ 4 MB for the interning pool).
More aggressive object interning didn't reduce memory usage much
further, but would increase interning overhead and code complexity, so
we didn't do this to more objects.
Note that this commit optimizes the size of classpath snapshots in
memory, not their serialized data on disk. (I attempted the latter,
but the size was only reduced from 160 MB down to 130 MB, while the code
complexity became much higher as multiple interning pools would need to
be stored to disk and later loaded from disk, each per classpath entry
snapshot or shrunk classpath snapshot.)
#KT-52329 Fixed
To snapshot a Java class (+ its fields and methods), previously we used
Gson to serialize a class field/method to a string via reflection, and
hash that string.
We now use an ASM ClassWriter to write a placeholder class containing
the field/method of interest and hash the bytecode of that class.
One experiment showed that this new approach is ~10 times faster than
the previous approach (140s down to 16s when snapshotting 600 jars).
Test: Updated expectation files for JavaClassSnapshotterTest unit tests
+ Existing integration tests to prevent regression
^KT-52141 In Progress
Implement an in-memory cache that uses a combination of strong
references and `SoftReference`s so that it adapts to memory
availability.
Cache eviction is currently performed after loading a classpath snapshot
(this can be changed later if necessary).
Evicted cache entries' values will be converted from strong references
into `SoftReference`s so that they can still be used for as long as the
JVM allows them.
There are 2 types of cache eviction:
- Least recently used: Oldest entries will be evicted
- Memory usage limit: If memory is limited, all entries will be
evicted
Test: Added InMemoryCacheWithEvictionTest unit test
^KT-51978 In Progress
Add another one reason for rebuild: if pluginClasspaths was changed or even its jar content, module will be marked for rebuild
Merge-request: KT-MR-5839
Merged-by: Aleksei Cherepanov <aleksei.cherepanov@jetbrains.com>
to reduce the size of the snapshots.
- Track metrics for Gradle classpath snapshot artifact transform
- Format size metrics so it's more readable
^KT-45777 In Progress
1. CLASS_LEVEL: allows tracking whether a .class file has changed
without tracking what specific parts of the .class file (e.g.,
fields or methods) have changed.
2. CLASS_MEMBER_LEVEL: allows tracking not only whether a .class file
has changed but also what specific parts of the .class file (e.g.,
fields or methods) have changed.
The idea is that for better performance we will use CLASS_LEVEL for
classpath entries that are usually unchanged, and CLASS_MEMBER_LEVEL
for classpath entries that are frequently changed. We'll work out the
specifics in a following commit after some measurements.
Support running kotlinc on Windows in ClasspathSnapshotTestCommon
Also add tests for different Kotlin class kinds.
Add unit tests for CLASS_LEVEL snapshotting and diffing
Test: Updated ClasspathSnapshotterTest + ClasspathChangesComputerTest
Add ClasspathChangesComputerTest.testMixedClassSnapshotGranularities
Make sure that:
- Compilation with -Pkotlin.incremental.useClasspathSnapshot=true or
-Dkotlin.incremental.classpath.snapshot.enabled=true is incremental
after cache hit.
- Default compilation (using build history files) is non-incremental
after cache hit.
Also unify the related tests into
BuildCacheRelocationIT.testKotlinIncrementalCompilation*. We don't need
corresponding tests in BuildCacheIT because BuildCacheRelocationIT
already covers it.
- Clean up BuildCacheRelocationIT.testKotlinIncrementalCompilation
- Delete BuildCacheIT.testKotlinCompileIncrementalBuildWithoutRelocation
as it is already covered by the test in BuildCacheRelocationIT
- Add BuildCacheRelocationIT.testKotlinIncrementalCompilation_withClasspathSnapshot
- Updated BuildCacheRelocationIT.testKotlinIncrementalCompilation_withClasspathSnapshot
If LeverVersion is set to greater than 1 of the current compiler version, metadata became null here org.jetbrains.kotlin.load.kotlin.header.ReadKotlinClassHeaderAnnotationVisitor.createHeader, which leads to NPE during incremental analysis. Using the same incompatible data as fallback prevent it
#KTIJ-20954 Fixed
Merge-request: KT-MR-5680
Merged-by: Aleksei Cherepanov <aleksei.cherepanov@jetbrains.com>
to ensure precision (otherwise, rounding errors to milliseconds may
add up and cause unexplainable gaps in the running time).
We can still use milliseconds in the final report after all the precise
sub-build-times have been aggregated.
Previously IC state was stored in System properties. As result parallel
compilation might cause incorrect state of IC, what led to corruption
of kotlin_module files. Now IC state is stored via CompilerArguments
and CompilerConfiguration
#KT-46038 Fixed
to make it easier to add more tests in the next commits.
- Add unit tests for constants and inline functions
Also add tests for different kinds of Kotlin classes: CLASS,
FILE_FACADE, MULTIFILE_CLASS.
-Add unit test for nested classes
Also remove the existing integration test for nested classes to keep the
integration tests focused on the key scenarios while unit tests will
cover the corner cases.
Ignore inline functions that are private
Currently, we shrink classpath snapshots at 2 steps:
- Classpath diffing: Shrink the current classpath snapshot against
the previous lookup symbols
- Classpath snapshot saving: Shrink the current classpath snapshot
against the current lookup symbols
With this commit, the shrinking at the second step is now incremental.
The shrinking at the first step is still non-incremental.
for performance reasons: (1) the snapshots are too big, and (2) they are
usually updated only at the end of the task execution--in a failed task
run, they are usually unchanged and therefore don't need to be restored.
Also visit a class file with ASM once to extract all information we need
in advance, instead of visiting the class file each time some piece of
info is needed.
as we need access to the lookup tracker to compute classpath changes
more efficiently and reduce the size of the saved classpath snapshot.
The previous commit only changed the files' paths, this
commit actually updates the files' contents.
Note that classpath snapshotting still happens in Gradle artifact
transforms. (However, the previous commit also moved the code for
classpath snapshotting together with the code for classpath diffing as
they are closely related.)
This is faster than the current approach which creates
`JavaClassDescriptor`s, converts them to protos, and then snapshots
these protos.
- Refactor unit tests to faciliate further changes
- moves test data to a directory that matches the tests' package name
- moves expected snapshots to a separate directory
- adds public and private fields/properties to sample class
- Compute changes between ASM-based Java class snapshots
- Don't collect members of an added Java class as changes
as it's enough to report the name of the added Java class as changed (we
also do that for added Kotlin classes and Kotlin/Java removed classes).
- Add unit tests for impact analysis in advance
- Compute impacted symbols of changed symbols
Also do not collect added classes/class members as they don't impact
recompilation.
-Use ClassId when computing Java class changes
It is more precise than JvmClassName, which can be ambiguous around the
`$` character (e.g., ClassId "com/example/A$B.C" and "com/example/A.B$C"
both have the same JvmClassName "com/example/A$B$C").
- Compute impacted set of changed symbols across Kotlin and Java
- Add unit tests for impact analysis across Kotlin and Java
- Compute supertypes of Kotlin classes during snapshotting
- Handle inner classes when computing list of changed symbols.
For the reported symbols, always check all options:
class member, inner class, top level class, top level member.
Test: IncrementalJavaChangeClasspathSnapshotIT.testAddingInnerClass
to avoid unnecessarily loading unchanged ones.
Duplicate classes will make this a bit tricky. This commit outlines the
algorithm to handle them, the full implementation will follow later.
Also handle removed classes when computing classpath changes.
Test: New tests in ClasspathChangesComputerTest
Workers usually execute later than task action and does not raise
exception to the task. Such behaviour skips task outputs restoration to
the pre-execution state.
Kotlin compiler on incremental compilation error leaves 'dirty*' files
in the build output, forcing Gradle to run non-incrementally on the
next build. Fixed it by moving task outputs reset logic into worker as
well.
^KT-46406 In Progress
There are certain classes that we are not yet able to take snapshots of,
either because the class is faulty, or there is a bug in our code.
For these classes, we will use a special snapshot and fall back to the
existing approach to compute classpath changes.
In the short run, we will update this list of cases as they arise. In
the long run, we will:
- Fix all cases that are caused by bugs in our code.
- Decide what to with the faulty jars in general and remove the list.
Test: The following should pass
- Step 1: ./gradlew publish
- Step 2: ./gradlew publish -Pbootstrap.local=true
-Pbootstrap.local.path=/path/to/kotlin/build/repo
-Pkotlin.incremental.useClasspathSnapshot=true
Simplify API of InlineConstTracker
In ConstLowering: Move transformer logic to another class to avoid mutable state. Avoid marking all files in module as module. Support of inner classes.
#KT-46506 Fixed
Create `JavaClassDescriptor`s for Java classes
Ignore anonymous and synthetic classes
as they can't impact recompilation.
Clean up handling of local and anonymous classes
Bug: KT-45777
Test: New JavaClassDescriptorCreatorTest
Now, milestones of major Kotlin releases (e.g. 1.6.0-M1) will NOT
generate prerelease binaries anymore.
The reason for that is it's proven to be quite a complicated process to
turn on IR_PRE_RELEASE after the release is branched, perform double
bootstrap and fix tests, and then do it again in reverse just before
release. With the new release cadence, we don't have that much time to
do it and verify that everything works as intended.
Note that this only removes the "global" prerelease flag. Compiler will
still generate prerelease binaries if a non-stable language version is
used. For example, Kotlin 1.6.0-M1 with `-language-version 1.7` will
generate prerelease binaries.
Alexander Udalov