CONCEPT Cited by 1 source

VM deoptimization¶

Deoptimization is the mechanism by which a type-specialized interpreter or JIT bails out of optimised code when a runtime condition invalidates a compile-time assumption, and falls back to a more general, slower interpretation path.

The archetypical flow:

Normal path:   specialized op executes successfully ──▶ next op
Deopt path:    specialized op detects invariant broken ──▶ bail out
                                                          ──▶ fallback interpreter
                                                          ──▶ next op

Why deoptimization is necessary¶

Any optimisation that depends on predicted or derived runtime state must have a fallback for when the prediction fails. This is true at every layer:

JIT compilers emit native code assuming operands are of certain types (e.g. 32-bit ints). A type guard at the start of each region bails out to the bytecode interpreter when a differently-typed value appears.
Statically type-specialized VMs emit opcodes that assume operands are of the types declared by the schema. Value-dependent type promotions (e.g. arithmetic overflow) break the assumption and require a fallback.
Inline caches in dynamic-language VMs cache observed types; cache misses bail out to a slow path.

The canonical SQL deoptimization trigger¶

The Vitess evalengine post gives the textbook example:

"Let's consider this wildly complex SQL expression: -inventory.price. That is, the negation of each of the values in the inventory.price column of our query. We know (thanks to our semantic analysis, and the schema tracker) that the type of the inventory.price column is BIGINT. So what could be the type of -inventory.price? Naive readers without experience in the magical world of SQL may believe the resulting type is BIGINT, but that's not the case in practice!

The vast majority of the time, the negation of a BIGINT yields indeed another BIGINT value. But when the actual value of the BIGINT is -9223372036854775808 (i.e. the smallest value that can be represented in 64 bits), negating it promotes the value into a DECIMAL, instead of silently truncating it, or returning an error."

The compile-time type derivation BIGINT → -BIGINT = BIGINT is wrong for one specific value. Rather than checking every operand at runtime (defeating the point of static specialization), Vitess bails out:

func (c *compiler) emitNeg_i() {
    c.emit(func(vm *VirtualMachine) int {
        arg := vm.stack[env.vm.sp-1].(*evalInt64)
        if arg.i == math.MinInt64 {
            vm.err = errDeoptimize   // ← deoptimize
        } else {
            arg.i = -arg.i
        }
        return 1
    })
}

The VM loop sees the deoptimization sentinel and passes the expression to the AST interpreter, which always type-switches at runtime.

Where the fallback runs¶

Three common fallback targets:

AST interpreter (Vitess) — the original tree-walking interpreter, kept around specifically for this purpose. Always correct, just slower.
Generic bytecode VM (most JITs) — the non-type-specialized VM the JIT sits on top of. V8's Ignition is the fallback for Maglev + TurboFan.
Compile a new specialized version (tracing JITs) — PyPy and LuaJIT sometimes respond to deoptimization by generating a new trace for the new types.

Deoptimization has a maintenance cost¶

The fallback interpreter can never be removed from the codebase. In Vitess's case:

"There is one significant drawback with this approach, however: the code for the AST interpreter can never be removed from Vitess. But this is, overall, not a bad thing. Just like most advanced language runtimes keep their virtual machine interpreter despite having a JIT compiler, having access to our classic AST interpreter gives us versatility. It can be used when we detect that an expression will be evaluated just once (e.g. when we use the evaluation engine to perform constant folding on a SQL expression). In those cases, the overhead of compiling and then executing on the VM trumps a single-pass evaluation on the AST. Lastly, when it comes to accuracy, being able to fuzz both the AST interpreter and the VM against each other has resulted in an invaluable tool for detecting bugs and corner cases."

The two-interpreter architecture becomes a maintenance cost and an operational asset — the fallback doubles as a fuzz oracle for correctness.

Seen in¶

sources/2025-04-05-planetscale-faster-interpreters-in-go-catching-up-with-cpp — canonical wiki instance at the interpreter level. Vitess evalengine bails from specialized VM instructions back to the AST interpreter on value-dependent type promotions (the BIGINT_MIN negation case). First wiki canonicalisation of deoptimization as a static-typing (not just JIT) fallback mechanism.