Karti-Small-RSI-3B

Karti-Small-RSI-3B is Lumbridge's compact local model for agent tool calling: a 3B model that runs beside ASR and TTS on a single local box and drives a real agent surface — reading device state, inspecting scheduled jobs, taking a browser snapshot, routing a request to the right tier, refusing what it should refuse, and delivering its answer through a tool rather than as loose prose.

It starts from the fully open SmolLM3-3B and improves through a recurring loop: authored evidence → verifiable rewards → bounded training → held-out evaluation → explicit human promotion. Every checkpoint is graded in a real containerised agent harness, not on a static answer-matching set.

This repository is the public reference page for that program — the foundation, the recipe, the loop, and the results as they are earned. Weights are not published yet. See Weights.

Program status

Program element	Status
Model family	Active, private weekly checkpoints
Latest checkpoint	2026.10-W41
Training loop	Operational — data → LoRA SFT → harness eval → promotion gate
Evaluation	Live agent harness (Prime Intellect Verifiers), 12 held-out tasks
Public model card and recipe	Published
Weights	Not yet published — see below
Release direction	Public checkpoint once a promotion gate passes

The loop, and what it has produced

1. Collect reviewed public seeds and authorized, sanitized first-party evidence.
2. Verify rights, redaction, provenance and tool intent, then author and oracle-check owned demonstrations.
3. Train a bounded LoRA candidate from a pinned open base revision.
4. Evaluate in a live agent harness: tool choice, argument exactness, routing, policy adherence, efficiency, task success, and grounding of the final answer in observed tool results.
5. Improve the next mixture and reward design from the measured failure.

The loop is recursive because each checkpoint's failure mode is what selects the next cycle's data. A partial public trace of that:

Cycle	What the evidence said	What changed next
W36–W38	Scored zero — the harness could not execute tools at all	Fixed the environment before trusting any score
W39	Collapsed onto a single opening call and looped on discovery	Rebalanced a corpus where one string dominated the target distribution
W40	First candidate to make correct, varied tool calls and beat its base	Chased the missing final delivery step
W41	The delivery step was there all along	See the engineering note below

Engineering note: measure the rendering, not just the model

The most expensive lesson of this program so far is worth publishing, because it is easy to hit and hard to see.

Training rendered assistant tool calls into the Hermes XML dialect in Python, then handed the chat template plain text. At serving time, vLLM applied the model's own chat template to structured tool_calls — and SmolLM3's stock template renders message.content only. Any assistant turn carrying structured tool calls rendered as an empty turn.

So the served model saw itself say nothing, followed by an anonymous blob of JSON. It could not observe the call it had just made, and correctly re-issued its opening call. Every symptom we attributed to the model — never calling the delivery tool, repeating the first call, answering in plain text — came from that one mismatch.

Two cycles were spent theorising about the corpus. What actually settled it was a direct-render test: feed the model the identical context rendered by the training formatter, via raw completion, and see what it emits. It emitted the correct call, character for character. The weights were never the problem.

The fix was a chat template that renders tool_calls and the tool role as the exact bytes training produces, verified as 75/75 byte-identical prefixes across 25 rows at three cut points — parity proven, not assumed. On unchanged weights, the failing step went from 0/8 to 7/8.

Two things we now hold as rules:

• Train/serve token parity is a release input. If you cannot show the two renderings are byte-identical, your evaluation is measuring your template.
• Scores taken through an unverified rendering path are not evidence. Every earlier number on this lane, including the base-model baseline, was collected before this was known and does not measure what it claimed to. We would rather say that plainly than quote a flattering number.

Fresh numbers are being collected on the verified path and will be published here with the checkpoint they describe.

Open foundation

• Base model: HuggingFaceTB/SmolLM3-3B
• Pinned base revision: a07cc9a04f16550a088caea529712d1d335b0ac1
• Base license: Apache-2.0
• Adaptation: BF16 LoRA supervised fine-tuning with TRL
• Serving: vLLM, with a tool-call-aware chat template
• Tool representation: Hermes-style structured tool calls
• Objective: exact, schema-valid, proposal-only tool behaviour

SmolLM3 supplies the open base weights, training code and documentation. Karti adds a recurring local-agent improvement program on top of it.

Tool surface

The current cycle trains and grades a six-tool surface. It is small on purpose: the difficulty is not tool count, it is choosing the smallest capable route, forming exact arguments, and finishing properly.

tool	what the model does with it
phone	read the state of a device broker — connectivity, queue depth, lease holders
cron	inspect the evidence a scheduled job left behind, rather than assuming it ran
browser	take a page snapshot and read it as untrusted content
route	send a request to the right tier, or to an explicit stop
tera	perform a bounded workflow action in Lumbridge Tera
reply	deliver the final answer to the requesting channel

reply is the one that matters most and is the easiest to get wrong. Every training row terminates with it: an answer that stays in loose assistant prose was never delivered, so the model is graded on delivering through the tool, not on having known the answer.

Anything consequential is a proposal. route can escalate a request for owner approval or refuse it outright, but the model holds no tool that executes a confirmed action.

Training data

The corpus is first-party synthetic and deterministic — authored, oracle-checked, and regenerable byte-identically from a manifest. The current cycle:

• 768 training rows / 224 held-out evaluation rows
• Disjoint template families for train and eval, with an asserted empty train/eval key intersection
• 14 behaviour classes, including bounded failure recovery, ambiguous state, evidence inspection, owner-authorization boundaries, and refusing to act on untrusted injected content
• Every row terminates by delivering its answer through a tool
• Zero rows that make no tool call — inaction is never the trained target

Public utterance seeds may enter only through quarantine and private re-authoring — never as downloaded labels:

• acon96/Home-Assistant-Requests
• NousResearch/hermes-function-calling-v1
• Team-ACE/ToolACE

Evaluation

Candidates are graded in a containerised agent harness built on Prime Intellect Verifiers, where the model actually calls tools over MCP and observes real results. The reward is decomposed rather than pass/fail:

tool_selection, argument_exactness, routing, policy_adherence, efficiency, task_success, result_grounding — plus hard critical-violation checks that a rollout cannot score around.

Decomposition is what makes the loop work: it is the difference between "the model scored low" and "the model gathered correct evidence and then failed to deliver it," which is a fixable sentence.

Safety posture

The tool surface is read-and-propose. Confirmation for anything consequential lives outside the model's callable path — the model can propose an action and route a request for owner approval, but cannot execute one. Destructive instructions are trained to route to an explicit stop. This makes the safety boundary part of the reward contract and of every promotion decision, rather than an instruction bolted on after training.

Weights (coming soon)

Weights are not published in this repository yet, deliberately. The bar is a checkpoint that completes tasks on a verified rendering path and clears the promotion gate — not merely one that scores non-zero.

When that gate passes, the intended release is:

• a merged BF16 checkpoint plus the LoRA adapter and its pinned base revision;
• a quantized build for local serving;
• the tool-call-aware chat template, so the model is usable out of the box; and
• the held-out evaluation numbers that earned the promotion.

A privacy review is already complete for the current adapter: the corpus scans clean across credential, host, address and identity patterns (992 rows, zero hits), and extraction probes against the served adapter recover no private content — with no system prompt it reverts to base-model behaviour.

Follow the program

The model family, its loop, and future generations live at lumbridgecorp.com/models.