Self-Direction · implementation

How the loop is built

Self-Direction is a thin layer around an ordinary agent. The snippets below are trimmed from our implementation to show the moving parts: they are meant to be read, not run. Every component is a standard tool, and the whole loop needs no extra teacher model and never updates the base weights. You can ask your AI agent to reference this and to replicate it.

vLLM inference server
SQLite failure memory
sentence-transformers router
PEFT LoRA adapters
Transformers trainer

01 · Overview

The whole loop, in one function

Every task first runs on the unchanged base model. Only after a verified failure does the loop store the failure, look for a stored skill that matches, train a new one if none does, and retry with the selected adapter. If that retry still fails, a final clean-base attempt is the fallback. The adapter is unloaded afterwards, so the next task starts from the base model again.

self_direction.pypseudocode

def run_task(task):
    # 1. Attempt with the frozen base model (vLLM, no adapter)
    result = agent.run(task, adapter=None)
    if is_correct(task, result):
        return result

    # 2. Mistake: keep the task, the failure reason and the full trajectory
    failure_id = memory.write(task, reason=classify(result), trajectory=result.trajectory)

    # 3. Choose one: reuse a stored skill if one is similar enough ...
    skill = router.best_match(task.question)              # cosine similarity over task goals
    if skill is None or skill.score < TAU:
        # ... otherwise self-reflect and train a new adapter
        examples = reflect(memory.get(failure_id))         # ≤ 24 procedural examples, same model
        skill = train_adapter(examples, reason=memory.get(failure_id).reason)
        library.register(skill, source_task=task.question)

    # 4. Retry with the selected skill, then fall back to a clean base attempt
    result = agent.run(task, adapter=skill)
    if not is_correct(task, result):
        result = agent.run(task, adapter=None, prompt=RETRY_PROMPT)

    if is_correct(task, result):
        memory.resolve(failure_id, skill_id=result.adapter or "base")
    return result

02 · Components

Five standard pieces

Each step of the loop maps onto a familiar tool. Inference runs on vLLM, failures live in SQLite, a small sentence-embedding model routes tasks to skills, the base model writes its own training data, and PEFT trains and loads rank-16 LoRA adapters while the base weights stay frozen.

2.1Inference

Attempt the task on vLLM

The agent is a plain ReAct loop over an OpenAI-compatible vLLM server: it calls tools until it produces a final answer or runs out of steps or tool calls. is_correct can be any feedback signal on the outcome; in our experiments it is the benchmark's own check. A failure is classified as a wrong answer, a missing final answer, an exhausted step or tool budget, or a crash.

agent.pyvLLM · OpenAI API

# Launched once per job:
#   vllm serve Qwen/Qwen3.6-27B --tensor-parallel-size <N>
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")

async def chat(messages, tools, model="Qwen/Qwen3.6-27B"):
    out = await client.chat.completions.create(
        model=model,                 # base model, or an adapter name (see 2.5)
        messages=messages, tools=tools,
        temperature=0.6, max_tokens=16384,
    )
    return out.choices[0].message

def classify(result):
    if result.terminated_reason in ("step_budget", "tool_call_budget", "exception"):
        return result.terminated_reason
    return "no_final_response" if not result.prediction else "wrong_answer"

2.2Storage

Remember the failure in SQLite

Each failure is one row: the task, the failure reason and the full trajectory, plus an open/resolved status. A de-duplication key makes writes idempotent, so a preempted job can resume without double-counting. Adapters live on disk next to a small JSON registry that records where each one came from.

memory.pysqlite3

import sqlite3, json

db = sqlite3.connect("failures.db", isolation_level=None)
db.execute("PRAGMA journal_mode=WAL")       # safe under job requeues
db.execute("""
CREATE TABLE IF NOT EXISTS failures (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    dedupe_key TEXT UNIQUE,                  -- hash(dataset, task_id, question)
    task TEXT NOT NULL,                      -- x: the task
    reason TEXT NOT NULL,                    -- r: why it failed
    trajectory_json TEXT NOT NULL,           -- t: every model action and tool result
    status TEXT NOT NULL DEFAULT 'open',     -- 'open' | 'resolved'
    resolution_skill_id TEXT                 -- adapter that fixed it, or 'base'
)""")

def write(task, reason, trajectory, key):
    db.execute("INSERT OR IGNORE INTO failures (dedupe_key, task, reason, trajectory_json) "
               "VALUES (?, ?, ?, ?)", (key, task, reason, json.dumps(trajectory)))
    return db.execute("SELECT id FROM failures WHERE dedupe_key = ?", (key,)).fetchone()[0]

def resolve(failure_id, skill_id):
    db.execute("UPDATE failures SET status = 'resolved', resolution_skill_id = ? "
               "WHERE id = ? AND status = 'open'", (skill_id, failure_id))

2.3Routing

Choose one: reuse or train

Every stored adapter keeps the text of the task that produced it. A new failure is embedded with all-MiniLM-L6-v2 and compared with those tasks by cosine similarity. If the best match clears the threshold τ, the loop reuses that adapter and skips reflection and training entirely. With τ = 1 an adapter is only ever reused on its own task.

router.pysentence-transformers

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def best_match(question, library, tau):
    if not library:
        return None
    keys = embedder.encode([a.source_task for a in library], normalize_embeddings=True)
    query = embedder.encode(question, normalize_embeddings=True)
    scores = keys @ query                    # cosine similarity
    i = int(scores.argmax())
    return library[i] if scores[i] >= tau else None   # None -> train a new adapter

2.4Self-reflection

The same model writes its own lesson

The clean base model, with no adapter loaded, reads the failed task, the failure reason, the verifier's answer (if available; only in GAIA for our case) and its own trajectory. It writes up to 24 instruction → response examples that teach the missing procedure: which tools to call, how to read their output, and when to stop. It is told never to copy the answer, so the examples teach the process rather than the solution. No stronger teacher model is involved.

reflect.pyself-teacher

SYSTEM = """You are a teacher model for continual agent learning. A student agent
attempted a task using tools and failed. Write training examples that teach the
REASONING PROCESS for this class of task, not the answer to this one instance.
Never copy the gold answer into a response. Show which tools to call and in what
order, how to interpret their outputs, and when to stop.
Output ONLY a JSON array of {"instruction", "response", "rationale"}."""

def reflect(failure, n=24):
    user = (f"TASK (x):\n{failure.task}\n\n"
            f"FAILURE REASON (r):\n{failure.reason}\n{FOCUS[failure.reason]}\n\n"
            f"GOLD ANSWER (reference only, do not copy):\n{failure.gold}\n\n"
            f"WHAT HAPPENED (t):\n{render(failure.trajectory, max_steps=24)}\n\n"
            f"Generate exactly {n} varied examples that teach the missing skill.")
    text = chat_base([{"role": "system", "content": SYSTEM},
                      {"role": "user", "content": user}], temperature=0.6)
    return parse_json_array(text)            # retried, with a template fallback

2.5Training

Distill the lesson into a LoRA adapter

The examples train a rank-16 LoRA adapter with PEFT. The failure reason picks the layers: budget and missing-answer failures adapt the attention projections, which steer planning, while wrong answers adapt the MLP down-projection. Matrix A is set from the top singular vectors of the frozen weight and kept frozen, and B starts at zero, so the adapter begins as a no-op and only B is trained, about 0.02% of the model's parameters. Training takes roughly 1.5 minutes on one H200, and the adapter takes about 85 MB on disk.

train.pyPEFT · Transformers

import torch
from peft import LoraConfig, get_peft_model
from transformers import Trainer, TrainingArguments

TARGETS = {"step_budget": ["q_proj", "v_proj"], "tool_call_budget": ["q_proj", "v_proj"],
           "no_final_response": ["q_proj", "v_proj"], "wrong_answer": ["down_proj"]}

def train_adapter(examples, reason, out_dir):
    cfg = LoraConfig(r=16, lora_alpha=16, lora_dropout=0.05, task_type="CAUSAL_LM",
                     target_modules=TARGETS.get(reason, ["down_proj"]))
    model = get_peft_model(base_model, cfg)            # base weights stay frozen

    # SVD-initialised A (frozen), zero B (trained): the adapter starts as a no-op
    for module in model.modules():
        if hasattr(module, "lora_A"):
            W = module.base_layer.weight.float()
            _, _, V = torch.svd_lowrank(W, q=16 + 8, niter=2)
            module.lora_A["default"].weight.data = V[:, :16].T.to(W.dtype)
            module.lora_A["default"].weight.requires_grad_(False)
            torch.nn.init.zeros_(module.lora_B["default"].weight)

    Trainer(model=model, train_dataset=to_chat_dataset(examples),
            args=TrainingArguments(out_dir, learning_rate=5e-4, num_train_epochs=2,
                                   per_device_train_batch_size=1,
                                   gradient_accumulation_steps=8, bf16=True)).train()
    model.save_pretrained(out_dir)                     # ~85 MB per skill
    return out_dir

2.6Retry

Retry with the skill, then fall back

The selected adapter is loaded onto the frozen base model and the task is retried. With vLLM, adapters can be hot-loaded into the running server and selected per request by name, so serving many skills needs no restart. If the adapter-guided retry fails, the adapter is unloaded and one final attempt runs on the clean base model with a prompt noting that a prior attempt failed.

Our experiments apply adapters in-process with PEFT (load_adapter / set_adapter); the vLLM calls shown here are the equivalent serving path.

retry.pyvLLM LoRA serving

# Server started with LoRA support:
#   VLLM_ALLOW_RUNTIME_LORA_UPDATING=True \
#   vllm serve Qwen/Qwen3.6-27B --enable-lora --max-lora-rank 16 --max-loras 4
import httpx

def load_skill(skill):
    httpx.post("http://127.0.0.1:8000/v1/load_lora_adapter",
               json={"lora_name": skill.skill_id, "lora_path": skill.adapter_path})

async def retry(task, skill):
    load_skill(skill)
    result = await agent.run(task, model=skill.skill_id)       # base + this adapter
    if is_correct(task, result):
        return result, skill.skill_id
    # clean-base fallback: same frozen model, no adapter
    result = await agent.run(task, model="Qwen/Qwen3.6-27B", prompt=RETRY_PROMPT)
    return result, ("base" if is_correct(task, result) else None)

03 · Settings

Default settings

The same settings are used on GAIA, Claw-Eval and Terminal-Bench 2.1. The paper's appendix lists the full configuration, the prompts, and the cost of building each adapter.

Component	Setting	Value
Agent	Base model	Qwen3.6-27B, frozen
Agent	Sampling	temperature 0.6, up to 16,384 new tokens
Self-reflection	Teacher	the same base model, no adapter
	Examples per failure	up to 24
	Trajectory shown	first and last steps, up to 24
LoRA adapter	Rank / alpha / dropout	16 / 16 / 0.05
	Initialization	A from SVD of the frozen weight (frozen), B = 0
	Trainable parameters	about 0.02% of the model
	Optimization	lr 5e-4, 2 epochs, effective batch size 8
	Size and time	about 85 MB, about 1.5 min on one H200
Router	Embedding model	all-MiniLM-L6-v2, cosine similarity
Router	Reuse threshold τ	reuse if the best match ≥ τ; τ = 1 reuses only on the same task
Retry	Order	selected adapter first, then a clean-base attempt