Document coroutines codegen: Returning Inline Classes

compiler/backend/src/org/jetbrains/kotlin/codegen/coroutines/coroutines-codegen.md

@@ -1921,6 +1921,170 @@ However, in this example, we cannot be sure that `generic` returns `Unit`. In th
 generally, the compiler disables tail-call optimization for functions returning `Unit` if the function overrides a function, returning
 non-`Unit` type.
 
+### Returning Inline Classes
+Previously, if a suspend function returns an inline class, the value of the class is boxed. That is undesirable for inline classes
+containing reference types since it leads to additional allocations. Thus, if the compiler can verify that callee returns an inline class,
+it does not generate boxing instructions in the callee and unboxing instructions in the caller. Otherwise, the callee returns a boxed
+value, as in the following example:
+```kotlin
+inline class IC(val a: Any)
+
+interface I {
+    suspend fun overrideMe(): Any
+}
+
+class C : I {
+    override suspend fun overrideMe(): IC = IC("OK")
+}
+
+suspend fun main() {
+    val i = C()
+    println(i.overrideMe())
+}
+```
+Here, the compiler cannot verify that the call-site always expects inline class. Thus, `overrideMe` always boxes the class.
+
+However, the optimization is not as straightforward as it seems. There are two paths of the execution of a suspend call: direct when
+the callee returns to the caller, and resume route when the callee returns to `invokeSuspend` and then to `resumeWith`, which calls
+`completion.resumeWith`, which calls `invokeSuspend,` which calls the caller. In the direct path (the most common case), the class is
+unboxed.
+
+However, in the resume path, we should box the inline class (in this case, we care less about performance).
+`BaseContinuationImpl.resumeWith` calls `invokeSuspend`, and it expects that the return type of `invokeSuspend` is
+"T | COROUTINE_SUSPENDED", where T is a boxed inline class in this case. Breaking this contract leads to throwing the exception in
+the following example:
+```kotlin
+import kotlin.coroutines.*
+
+fun main() {
+    builder {
+        signInFlowStepFirst()
+    }
+    continuation!!.resumeWithException(Exception("BOOYA"))
+}
+
+fun builder(c: suspend () -> Unit) {
+    c.startCoroutine(object : Continuation<Unit> {
+        override val context: CoroutineContext
+            get() = EmptyCoroutineContext
+
+        override fun resumeWith(result: Result<Unit>) {
+            result.getOrThrow()
+        }
+    })
+}
+
+var continuation: Continuation<Unit>? = null
+
+suspend fun suspendMe() = suspendCoroutine<Unit> { continuation = it }
+
+@Suppress("RESULT_CLASS_IN_RETURN_TYPE")
+suspend fun signInFlowStepFirst(): Result<Unit> = try {
+    Result.success(suspendMe())
+} catch (e: Exception) {
+    Result.failure(e)
+}
+```
+The explanation of the bug cause is not that simple:
+1. `signInFlowStepFirst` call `suspendMe` and suspends
+2. We resume the execution with an exception.
+3. Inside `signInFlowStepFirst`, we wrap the exception with Result class, just like in a burrito.
+4. Since it is the resume path (we resumed the execution), the execution returns to `invokeSuspend`, which returns
+`Result$Failure` to `BaseContinuationImpl.resumeWith`.
+5. `BaseContinuationImpl.resumeWith` wraps `Result$Failure` with another `Result`, but since `Result` is an inline class, the result
+(pun not intended) of the operation is the same `Result$Failure`.
+6. `BaseContinuationImpl.resumeWith` calls `completion.resumeWith`, passing the `Result$Failure` as the argument, which is considered
+as `resumeWithException` by the completion.
+
+So. We need to box inline class inside `invokeSuspend` if the function returns inline class, and the compiler has optimized boxing, as
+well as inside the callable reference. That fixes the coroutine contract of `invokeSuspend`.
+
+However, in the direct path, generated code expects an unboxed value. So, in the resume path of the caller, we should unbox it. There
+are a couple of places we can unbox it: `invokeSuspend` and unspilling inside a state-machine. Consider the following snippet:
+```kotlin
+import kotlin.coroutines.*
+
+inline class IC(val a: Any)
+
+fun builder(c: suspend () -> Unit) {
+    c.startCoroutine(Continuation(EmptyCoroutineContext) { it.getOrThrow() })
+}
+
+var c: Continuation<Any>? = null
+
+suspend fun returnsIC() = suspendCoroutine<IC> { c = it as Continuation<Any> }
+suspend fun returnsAny() = suspendCoroutine<Any> { c = it }
+
+suspend fun test() {
+    println(returnsIC())
+    println(returnsAny())
+}
+
+fun main() {
+    builder {
+        test()
+    }
+    c?.resume(IC("OK1"))
+    c?.resume("OK2")
+}
+```
+Here, we resume the `test` function twice, once with the inline class, the other with an ordinary one. However, we need to box the value
+only once: during the first resumption. Meaning that we need to add complex logic to `invokeSuspend` if we want to box the value. It is
+simpler to do the boxing inside the state-machine.
+
+#### Inlining
+Note: this section is about inlining. Nevertheless, it is too specific to be put in the corresponding section.
+
+However, we do not always have a state-machine. Consider the following example:
+```kotlin
+import kotlin.coroutines.*
+
+// Library lib1
+inline class IC(val a: Any)
+
+var c: Continuation<Any>? = null
+
+suspend fun returnsIC() = suspendCoroutine<IC> { c = it as Continuation<Any> }
+
+// Library lib2 depends on lib2
+
+suspend inline fun inlineMe() {
+    println(returnsIC())
+}
+
+// Main module
+
+fun builder(c: suspend () -> Unit) {
+    c.startCoroutine(Continuation(EmptyCoroutineContext) { it.getOrThrow() })
+}
+
+suspend fun test() {
+    inlineMe()
+}
+
+fun main() {
+    builder {
+        test()
+    }
+    c?.resume(IC("OK1"))
+}
+```
+Here, `inlineMe$$forInline` has no state-machine, and thus, the direct path is similar to the resume path. After inlining, the compiler
+has no idea that it should generate unboxing in the resume path. To fix the issue, the compiler can add a marker to show that there should
+be boxing in the resume path. For example, it can generate something like
+```text
+ICONST_1
+INVOKESTATIC kotlin.jvm.internal.InlineMarker.mark(I)V
+// The suspension point
+ICONST_2
+INVOKESTATIC kotlin.jvm.internal.InlineMarker.mark(I)V
+ICONST_8
+INVOKESTATIC kotlin.jvm.internal.InlineMarker.mark(I)V
+// After this marker, there should be a call to box-impl
+INVOKESTATIC IC.box-impl(Ljava/lang/Object;)LIC;
+```
+Generating the marker fixes the issue with inlining.
+
 ## Inline
 
 Inlining a suspend function is a tricky business.