Implementation Strategy

This document describes how the Turn compiler and runtime are built, detailing the journey from source text to bytecode execution. The compiler, VM, CLI, and LSP are implemented in Rust and shipped through one Turn executable.

Architecture Pipeline

The Turn toolchain follows a traditional compiler pipeline with domain-specific extensions for agentic primitives. Each stage transforms the program into a progressively lower-level representation.

Stage	Component	Output
Lexer	`src/lexer.rs`	A stream of tokens with position information for error reporting.
Parser	`src/parser.rs`	An abstract syntax tree (AST) representing the program's structure.
Semantic Analysis	`src/analysis.rs`	Validated AST with resolved identifiers and collected type information.
Schema Compiler	`src/schema_compiler.rs`	Macro expansion pass that resolves `use schema::openapi("url")` at compile time by fetching the schema, parsing it, and synthesizing native Turn closures in the AST before analysis.
Compiler	`src/compiler.rs`	A flat sequence of bytecode instructions targeting the Turn VM.
VM	`src/vm.rs`	Stack-based bytecode execution with process scheduling and suspension.
Runtime	`src/runtime.rs`	Managed context, key-value memory, struct definitions, confidence telemetry, and pending effects.
Tools	`src/tools.rs`	Kernel trap registry (`__sys_*` prefixed functions) providing host capabilities such as filesystem, network, and environment access. Wrapped by the Turn Standard Library modules. Delegates LLM inference to WASM drivers via `src/wasm_host.rs`.
WASM Host	`src/wasm_host.rs`	Loads and executes WASM provider drivers from `.turn_modules/`. Handles request/response transformation for all LLM providers.
Server	`src/server.rs`	gRPC switchboard for distributed process communication across nodes.
LSP	`src/lsp.rs`	Language Server Protocol implementation for editor integration.

Bytecode Instructions

The compiler emits a flat instruction sequence that the VM executes on a stack machine. Each opcode corresponds to a well-defined semantic operation, ensuring there are no hidden instructions whose behavior depends on runtime configuration.

Opcode	Meaning
`PushConst`	Push a literal value (number, string, boolean, null) onto the operand stack.
`Store` / `Load`	Write a value to or read a value from a named local variable in the current scope.
`Infer`	Trigger an LLM inference call with a struct schema. Suspends the VM until the result arrives.
`Decide`	Trigger a bounded decision-model request. Suspends the VM until structured answers arrive.
`Remember` / `Recall`	Write to or read from the process's persistent key-value memory store.
`Spawn`	Create a new unlinked actor process from a closure. Returns a `Pid` to the parent.
`SpawnLink`	Create a bidirectionally linked actor process. Exit signals propagate to all linked partners.
`SpawnEach`	Scatter each element of a list to a concurrent micro-actor (the scatter/gather primitive).
`Send` / `Receive`	Enqueue a message into another process's mailbox, or block until a message arrives in the current process's mailbox.
`Confidence`	Extract the confidence score from an `Uncertain` value. Used by `if confidence expr < 0.85 { ... }`.
`PushHandler` / `PopHandler` / `Throw`	`try/catch/throw`: push an error handler frame, pop it on normal exit, or unwind to the nearest frame on throw.
`MakeStructSpread`	Create a struct literal by merging a base struct's fields with explicitly named field overrides (`..base` spread syntax).
`Suspend`	Write the full VM state to durable storage and halt execution until an external event resumes it.
`CallTool`	Invoke an external tool by name. This serves as the effect boundary and the single point where non-determinism enters the system.
`CallMethod`	Invoke a named method on a module value returned by `use`. Used for Standard Library and Wasm module method calls.
`LoadModule`	Load and execute a Turn module from a path string, returning the module's exports as a value.
`Index`	Access an element in a list or map by key. Implements `expr[index]` syntax.
`MakeVec`	Construct a vector embedding literal from a fixed number of numeric values on the stack.
`Similarity`	Compute cosine similarity between two vector embeddings. Implements the `~>` operator.
`EnterTurn`	Enter a `turn { }` block, establishing a new execution scope with its own return address.
`GrantIdentity`	Suspends the VM and requests a cryptographic Identity capability from the Rust host. Returns an opaque `Value::Identity` handle.

Concurrency Model

Turn processes are VM-owned values scheduled through a local round-robin queue. Each process carries isolated frames, stack, runtime state, mailbox, parent link, and gas budget.

NOTE

Inter-process communication enqueues serialized Value messages in per-process VecDeque mailboxes. Linked child completion or failure produces an ExitSignal for the parent.

The scheduler executes each process for a bounded instruction slice before yielding to the queue. Effect boundaries return a durable continuation to the runner.

Persistence: File Checkpoints

The runner serializes VmState as JSON in the configured store before external effects and explicit suspension:

The state contains frames, stack, runtime context/memory, mailboxes, scheduler, gas, and the pending effect request.
Runner::run loads a checkpoint by agent ID and replays a pending effect when present.
Successful and uncaught-error completion remove the checkpoint; explicit suspension retains it.
The default FileStore writes one JSON checkpoint per agent ID. It is not an append-only WAL.

Pending effects use at-least-once delivery. Non-idempotent tools must supply idempotency keys.

Why Rust

The choice of Rust is not a mere preference but rather a direct consequence of the design constraints:

One Rust CLI. The Turn compiler, VM, LSP server, and HTTP service ship through one executable. Turn does not require a Python virtual environment or Node.js runtime.
No Global Interpreter Lock. Rust leaves room for parallel schedulers without Python's GIL; the current actor scheduler is local and round-robin.
Memory safety without garbage collection. Long-running agent processes cannot afford GC pauses or memory leaks. Rust's ownership model guarantees memory safety at compile time with zero runtime overhead.
Minimal overhead. Every CPU cycle and byte of memory that the interpreter does not consume is budget that can be spent on LLM tokens. A Rust VM has orders of magnitude less overhead than a Python or JavaScript runtime.

Next Steps

LSP