Train Your Own Mini-Model: From nanochat to a Fine-Tuned Daily Driver

For about $100 and four hours on a rented 8×H100 node, you can pretrain a ChatGPT-shaped model from scratch — tokenizer, chat UI, and all. That is nanochat, Andrej Karpathy’s full-stack speedrun, and it made “train your own LLM” a weekend project instead of a lab budget. The number that matters more, though, is smaller: for roughly the price of lunch in rented GPU time, you can fine-tune an open 8B model on your own transcripts and get a specialist that does one job in your house style, every time, on your hardware.

This playbook is the complete route. Three lanes, ordered by effort: a LoRA/QLoRA fine-tune with Unsloth or Axolotl, distillation from a teacher model onto a small student, and the from-scratch nanochat run. Around them, the parts most guides skip: turning your own session archive into a training set without leaking secrets into weights, evaluating before and after so you know whether anything improved, serving the result with Ollama or vLLM, and wiring it into your agent harness as a cheap lane.

One expectation set now, because everything downstream depends on it: a mini-model is a specialist, not a frontier peer. Nothing you train this quarter will out-reason Claude Fable 5 or GPT-5.6. Plenty you train this quarter can out-format, out-price, and out-private them on one narrow job. That is the prize.

Why train your own LLM when tokens cost $0.14 a million?

The objection writes itself. DeepSeek V4 Flash sells at $0.14 per million input tokens and $0.28 out — the credible price floor for agentic work, as we covered in the DeepSeek effect on agent economics. You will not beat that floor on raw intelligence-per-dollar with a home-trained model. So don’t try. The reasons to train your own are the ones a hosted API structurally cannot sell you:

Form. A fine-tuned model produces your output dialect — your commit-message format, your JSON schemas, your review-comment tone — from the weights, without the 3,000-token system prompt you currently paste into every call. Prompted format compliance plateaus; trained format compliance is what the model does by default.

Privacy. Some corpora cannot leave the building. A model trained and served on your own box, on your own redacted transcripts, is the only architecture where that sentence stays true end to end — and it pairs naturally with the local AI workstation you may already be speccing.

Marginal cost and latency. A local specialist answers in milliseconds of queue time with no per-token meter, which changes what you’re willing to automate. Commit messages, log triage, PR summaries — jobs too trivial to spend frontier tokens on become free.

Understanding. The third lane exists mostly for this. Training a tiny model end to end teaches you what tokenizers, data mix, and eval design actually do — knowledge that transfers directly to how you drive the big models.

If none of those four apply to you, close this tab and keep buying tokens; that is the correct call for most workloads. If at least one does, pick a lane.

The three lanes, ranked by effort

Each lane trades more effort for more ownership of the result. Most readers want Lane 1. Lane 2 is Lane 1 with a synthetic-data step in front. Lane 3 is an education with a model-shaped diploma.

	Lane 1: fine-tune	Lane 2: distill	Lane 3: from scratch
You start from	An open-weight base (Qwen3, Llama, Gemma class)	A teacher model + a small student base	Random weights and a text corpus
Data	500–50k examples from your real work	5–20k synthetic pairs the teacher writes	~10B+ tokens of public web text
Compute	One 12–24GB GPU, hours	Teacher tokens + one GPU, days	Rented 8×H100 node, ~4 hours+
Cash cost	~$0–30	~$30–600, teacher-dependent	~$100–1,000 by tier
What you get	Your style and task dialect in a capable base	A cheap copy of one teacher skill	A toy peer of 2019’s GPT-2 — and the education
Skill floor	Can read a YAML file	Comfortable with data pipelines	Comfortable with training runs

Effort buys ownership: adapt an open base, copy a teacher, or build the whole thing.

The lanes also compose. The strongest pattern we see in practice is 2-into-1: use a big model to synthesize training data for exactly one job, then QLoRA it into a small base. You get teacher-grade behavior on the narrow task at student-grade prices.

Data first: your session archive is a training set

Every lane except nanochat runs on your data, and here is the underrated fact of 2026: you already have a corpus. If you run coding agents daily, your CLIs have been writing transcripts for years — real tasks, real repo context, real fixes, and (crucially) real examples of what “done right” looks like in your codebase. That archive beats any public instruction dataset for teaching a model to be useful to you.

It is also radioactive in raw form. Transcripts contain API keys that passed through tool arguments, internal hostnames, customer identifiers, and the occasional pasted .env file. Language models memorize training data, and a fine-tuned checkpoint can be prompted into regurgitating it — so a secret that enters your training set has a real path back out of your model’s mouth, and the weights file itself becomes a leak vector the moment you share it. Redaction is not a nice-to-have; it is the step that makes the rest of the pipeline safe to run.

The pipeline, in order:

1. Collect. Export sessions from every harness you run — most CLIs write JSON or JSONL session files to a dot-directory.
2. Filter to wins. Keep sessions that ended well: merged diff, passing tests, an answer you actually used. Failed meanders teach failure.
3. Redact. Scrub credentials, tokens, internal URLs, personal data. Automated pattern passes first, then spot-check by hand. Do this before anything else touches disk twice.
4. Dedupe. Exact-match first, then near-duplicate removal; repeated boilerplate teaches the model to parrot it.
5. Format. Convert to chat-format JSONL matching your target model’s template — system, user, assistant turns.
6. Split and hold out. Reserve 10–20% of sessions as eval material that the training run never sees.

A finished example looks like this:

{
  "messages": [
    {
      "role": "system",
      "content": "You write conventional-commit messages for the acme monorepo."
    },
    {
      "role": "user",
      "content": "Diff summary: 3 files in services/auth; adds refresh-token rotation with a 24h grace window."
    },
    {
      "role": "assistant",
      "content": "feat(auth): rotate refresh tokens on exchange\n\nAdds a 24h grace window for in-flight clients. Closes #4182."
    }
  ]
}

On volume: style-and-format adapters train convincingly on 500–5,000 examples; task competence wants 5,000–50,000. Quality dominates quantity at every scale — a thousand curated wins beat twenty thousand raw scrapes, every time.

The pipeline that turns transcript exhaust into a training corpus — with a hard gate where the secrets die.

Product note: The collect-and-redact steps are already solved if your sessions live in one place. Automater Lite consolidates transcripts from 10+ CLIs — Claude Code, Codex, Qwen Code, OpenCode, and friends — into a local-first, full-text-searchable archive, and its Vault redaction scrubs keys and other sensitive strings from stored sessions before they go anywhere near a training run. The archive never leaves your machine, which is exactly the property you want your corpus to inherit. Free on automater.ai.

Lane 1: fine-tune an open model with LoRA or QLoRA

LoRA (low-rank adaptation) is a fine-tuning method that freezes the base model’s weights and trains small low-rank matrices injected alongside them, so you update well under 1% of parameters. The result is an adapter file measured in megabytes that steers a multi-gigabyte model — cheap to train, cheap to store, easy to swap. The technique comes from a 2021 Microsoft paper and remains the default way individuals fine-tune.

QLoRA pushes the hardware floor down further: the frozen base is quantized to 4-bit while the LoRA adapters train on top in higher precision. The QLoRA paper demonstrated fine-tuning a 65B model on a single 48GB card; in 2026 practice it means an 8B fine-tune fits a 12GB consumer GPU and a 24GB card handles far more than you’d guess.

Hardware realism, assuming QLoRA with a modern stack:

Base model	VRAM needed	Fits on	Typical run (10k examples, 2 epochs)
4B	~6–8GB	Nearly any current GPU	Under an hour
7–8B	~10–12GB	RTX 3060 12GB and up	1–3 hours
14B	~16–20GB	One 24GB card	2–5 hours
30B-A3B MoE	~20–24GB	RTX 3090 / 4090	3–6 hours
70B dense	~46–48GB	48GB card, 2×24GB, or rent	Rent it

Two tools own this lane. Unsloth (unsloth.ai) is the speed play: an open-source library claiming roughly 2× faster training with substantially less VRAM than the vanilla Hugging Face stack, notebook-friendly, with direct GGUF export for serving. Axolotl (axolotl.ai) is the config play: your whole run is a YAML file, it scales from one GPU to multi-node, and the config doubles as documentation of what you did. Start with Unsloth for a first run; graduate to Axolotl when you want reproducibility.

An Unsloth run, sketched (a real run adds checkpointing and logging):

from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Qwen3-8B-bnb-4bit",      # 4-bit base; fits a 12GB card
    max_seq_length=8192, load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=32, lora_dropout=0,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
)
ds = load_dataset("json", data_files="corpus/train.jsonl", split="train")
SFTTrainer(model=model, tokenizer=tokenizer, train_dataset=ds,
    args=SFTConfig(per_device_train_batch_size=2, gradient_accumulation_steps=8,
                   learning_rate=2e-4, num_train_epochs=2, bf16=True,
                   output_dir="out/repo-scribe")).train()
model.save_pretrained_gguf("out/repo-scribe-gguf", tokenizer,
                           quantization_method="q4_k_m")

The same shape in Axolotl is a YAML you can diff and review:

base_model: Qwen/Qwen3-8B
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
datasets:
  - path: corpus/train.jsonl
    type: chat_template
sequence_len: 8192
sample_packing: true
micro_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 2e-4
num_epochs: 2
output_dir: ./out/repo-scribe

Run it with axolotl train qlora.yml and go make coffee. The hyperparameters that matter: lora_r (adapter capacity — 16 for style, 32–64 for task skills), learning rate (2e-4 is the QLoRA-era default; halve it if loss spikes), and epochs (2–3; more mostly memorizes). Base-model choice matters more than any of them — pick from the current open-weight field, which we map in open-weight models that can drive a harness, and prefer a base whose license you can live with downstream.

When a fine-tune beats prompting. The tell is repetition: if every call to a task ships the same enormous system prompt — format spec, taxonomy, tone rules, few-shot examples — you are paying tokens and latency to re-teach a fixed lesson, and compliance still wobbles. Fine-tuning moves that lesson into the weights: output format, house style, a tool-call dialect, a labeling taxonomy. Prompting (plus retrieval) stays the right tool for anything that changes weekly, and no fine-tune adds knowledge reliably — that is what context is for. The rule of thumb: tune for form, retrieve for facts, prompt until repetition hurts.

Where Lane 1 fails. Four traps take most casualties. Overfitting a tiny corpus — training loss near zero, model parrots training examples verbatim. Catastrophic forgetting — a style adapter trained too hard stops being able to code; keep r modest and epochs low, or mix ~10–20% general instruction data back in. Chat-template mismatch — training with one template and serving with another silently lobotomizes the model; it is the most common “my fine-tune got worse” diagnosis. And eval contamination — if held-out sessions leaked into training, your before/after numbers are fiction.

Lane 2: distillation — teacher, student, and the licensing fine print

Distillation trains a small student model to imitate a larger teacher. The classic form matched output distributions (Hinton et al., 2015); the form practitioners use in 2026 is simpler: have the teacher generate high-quality demonstrations for your task, filter them, and fine-tune the student on the survivors. Same Lane 1 mechanics, but the teacher writes the corpus.

Distill when your archive is too thin for the task you want — you have 400 real examples and need 15,000 — or when the skill you want lives in a model too big to serve. The recipe:

1. Pick one narrow job with a checkable output. “Write our migration scripts” distills; “be good at engineering” does not.
2. Harvest real prompts. Pull 5–20k genuine inputs from your archive using the data pipeline above — real prompts, synthetic answers is the quality trick. Fully synthetic prompts drift toward what the teacher finds easy.
3. Generate. Teacher answers every prompt, one to three samples each, at low temperature.
4. Verify and filter. This is where distillation is won. Run generated code, check schemas against a validator, lint formats — keep only passes. Rejection sampling turns a mediocre synthetic set into a sharp one.
5. Dedupe and decontaminate against your eval set, again.
6. Train the student with the Lane 1 QLoRA recipe on a 4–8B base.
7. Score retention. Run student and teacher on the same held-out eval; report the student’s pass rate as a fraction of the teacher’s. A good narrow-task distill retains most of the teacher’s success rate at a double-digit fraction of the price — and if it doesn’t, you learned that for under a hundred dollars.

The teacher bill is smaller than intuition says. Twenty thousand examples at ~1,500 output tokens each is 30M tokens: about $8 from DeepSeek V4 Flash at $0.28/M output, low tens of dollars from the bigger open-weight flagships, and a few hundred from a US frontier model.

Which brings us to the fine print, and it deserves plain language. As of August 2026, the major US labs’ terms restrict using their outputs to train models that compete with theirs. Whether your internal commit-message specialist “competes” is a question for your reading of the current terms and your risk tolerance — read them yourself before pointing a frontier teacher at a student, and involve counsel if the result ships commercially.

The clean path avoids the question: use an open-weight teacher whose license permits it. The mid-2026 open-model field makes this easy — DeepSeek V4 and GLM-5.2 are MIT-licensed, Qwen3-Coder-Next is Apache-2.0, and Kimi K3 ships under a modified MIT license . We keep the full cards in the open-weight scorecard. Two residual checks even on the clean path: your student inherits its base model’s license (another vote for Apache-2.0 and MIT bases), and some open licenses carry attribution or naming clauses that apply to derivatives. Five minutes with the actual license text is the whole diligence.

Lane 3: nanochat — the $100 model you train from scratch

The third lane trains the whole thing: tokenizer, pretraining, the works. Until recently that sentence ended with “so don’t.” Then Karpathy published nanochat, billed as “the best ChatGPT that $100 can buy” — a single, dependency-light codebase of roughly 8,000 lines that runs the entire pipeline: train a tokenizer, pretrain a transformer on web text, midtrain on conversations and tool use, apply supervised fine-tuning, optionally run RL, then serve the result through an inference engine with a ChatGPT-style web UI.

The headline speedrun runs on one rented 8×H100 node for about four hours — roughly $100 of cloud GPU time — and produces a small model in the half-billion-parameter class. Spend more, get more: the repo documents larger targets, with a ~$300 tier that lands in GPT-2’s neighborhood on the CORE benchmark and a ~$1,000 tier around 1.9B parameters that starts posting non-trivial scores on reasoning and math suites.

What you get is a party guest, not a colleague. The $100 model chats, rhymes, answers simple questions, and states wrong facts with total confidence; coverage of the release rightly framed it as education and experimentation, not a production tool. It will not drive your harness, and it will not replace even the free tier of anything. Do not build your cheap lane here — that is what Lanes 1 and 2 are for.

What you learn is the actual product, and it is substantial. Every mechanism you otherwise know as a vendor abstraction becomes code you ran and can modify: why tokenizer choices show up in model behavior, what the data mix does, how SFT turns a text-completer into an assistant, why inference needs a KV cache, how evals get gamed. The codebase is deliberately readable and forkable — designed as a course capstone — and “I have trained a model end to end” measurably changes how you debug, prompt, and budget the big ones. Fork it, change one variable per run, and keep notes.

Lane 3 makes sense if you have $100–$1,000 of curiosity budget, a free weekend, and the itch to know what is actually happening under the APIs you rent. It is the best value in AI education right now. It is also, and we say this with affection, the worst value in AI capability.

Eval before, eval after

Training without evaluation is astrology with GPUs. The discipline is one afternoon of work: build a private benchmark from the 10–20% of sessions you held out in data prep, and run it against the base model before you train. That pre-run number is the whole justification for the project — sometimes the base model with a good prompt already clears your bar, and you just saved a week.

Keep the bench small and real: 30–50 tasks drawn from actual work, each with a checkable outcome. Score three things per task: format validity (does the output parse — schema check, conventional-commit regex, whatever your dialect demands), task success (does the diff apply, do the tests pass, is the label right), and quality preference (a strong model judging tuned vs. base output blind, positions swapped to kill order bias). Our full guide to evals for AI agents covers the machinery; for a mini-model you need maybe 150 lines of harness script.

A before/after sheet for, say, a commit-message specialist has exactly this shape:

Metric (n=40 held-out tasks)	Base 8B + prompt	Fine-tuned 8B	Frontier + prompt
Format-valid rate	—	—	—
Task success	—	—	—
Judge preference vs. base	—	—	—
Median latency / cost per call	—	—	—

Fill it with your numbers, not anyone’s marketing. The frontier column keeps you honest about what you’re trading away; the latency/cost row usually explains why the trade is worth it anyway. Add one regression probe if your model serves broader duty — a handful of general tasks to catch catastrophic forgetting — and re-run the sheet any time you retrain. If tuned doesn’t beat base on your bench, do not ship it to your harness on vibes.

Serving your mini-model and wiring it into a harness

A model you can’t call is a science-fair project. Two servers cover the field, and both speak the OpenAI-compatible dialect that nearly every CLI harness in the mid-2026 field map accepts.

Ollama is the five-minute path for a single user. Unsloth exports GGUF directly, so serving is a Modelfile away:

# Modelfile
FROM ./repo-scribe-q4_k_m.gguf
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
SYSTEM You write commit messages and PR summaries in acme house style.

ollama create repo-scribe -f Modelfile
ollama run repo-scribe          # sanity check
# OpenAI-compatible endpoint now lives at http://localhost:11434/v1

vLLM is the answer when concurrency arrives — several harness lanes or teammates hitting the model at once. It batches continuously, and it serves LoRA adapters on top of a shared base without merging, which means one base model can host several of your specialists:

vllm serve Qwen/Qwen3-8B \
  --enable-lora \
  --lora-modules repo-scribe=./out/repo-scribe triage=./out/triage \
  --max-model-len 16384

Wiring it into a harness is a base-URL override. Aider, OpenCode, and most of the open-source field take an OpenAI-compatible endpoint straight from the environment:

export OPENAI_API_BASE="http://localhost:11434/v1"
export OPENAI_API_KEY="local"
aider --model openai/repo-scribe

Then run it as your cheap lane, not your only lane. The pattern that works: route the high-volume, low-stakes, format-heavy jobs to the mini-model — commit messages, changelog entries, PR summaries, log and test-failure triage, first-pass code-review comments — and keep architecture, debugging, and multi-file changes on the frontier lane. The escalation rule stays dumb on purpose: anything the mini-model fails or flags goes up a tier. If you already run several assistants side by side, this is just one more lane in the fleet — the routing discipline is the same one we describe in running multiple AI coding agents without the chaos, except this lane’s meter reads zero.

Honest expectations, and the bill

What each lane actually buys, stated without romance:

Lane	What you’ll have	What it will never be
Fine-tune	A specialist that nails your format and tone on one job, runs free and fast on your box	Smarter than its base; a source of new knowledge; a frontier replacement
Distill	Teacher-grade behavior on one narrow task at student prices	A general copy of the teacher; legally carefree unless the teacher’s license is
nanochat	A working toy model, and a working understanding of the entire stack	Useful in a harness, at any tier you can afford

And the money, all-in orders of magnitude:

Lane	Compute	Data cost	Calendar time
QLoRA fine-tune, 8B	$0 on a 24GB card you own; ~$5–30 rented	$0 — your archive	A weekend
Distillation, 8B student	~$20–100 GPU time	~$10–50 open-weight teacher; ~$300–600 frontier teacher, terms permitting	A week of evenings
nanochat speedrun	~$100 (8×H100, ~4h)	$0 — public corpora	A day, plus babysitting
nanochat, largest documented tier	~$1,000	$0	~2 days

The pattern worth noticing: the entire table costs less than one month of a heavy frontier-token habit. That is the real story of small language model training in 2026 — not that your model will beat theirs, but that “form, privacy, marginal cost, and understanding” now price in hobby range. Train the specialist. Keep the frontier lane. Let each do the job the other can’t.

FAQ: training your own LLM

How much does it cost to train your own LLM in 2026?

A QLoRA fine-tune of an 8B open model costs $0–30 in GPU time and a weekend. Distilling a teacher into a small student runs $30–600 depending on the teacher. Pretraining from scratch via nanochat starts around $100 for the speedrun tier. Data from your own archives is free.

Is fine-tuning better than prompting or RAG?

They solve different problems. Fine-tune for form — output format, style, a fixed dialect you currently re-teach in every prompt. Retrieve for facts, because fine-tuning adds knowledge unreliably. Prompt for everything that changes often. The tell that you should fine-tune is a huge, static system prompt shipped with every call.

What can the $100 nanochat model actually do?

It holds a conversation, writes stories and poems, answers simple questions, and confidently gets things wrong — a half-billion-parameter model in roughly GPT-2’s league, behind a ChatGPT-style UI. Its real output is education: you run and can modify every stage, from tokenizer to serving. It cannot drive a coding harness.

Is it legal to train on another model’s outputs?

Open-weight teachers with permissive licenses — MIT-licensed DeepSeek V4 and GLM-5.2, Apache-2.0 Qwen — are the clean path for distillation. Major US labs’ terms restrict training competing models on their outputs, so read the current terms before using a frontier teacher, and check your student base’s license too.

What hardware do I need to fine-tune an LLM at home?

A 12GB GPU handles QLoRA on 7–8B models; a 24GB card (used RTX 3090 class) covers 14B dense and 30B-A3B MoE bases. Anything larger, rent by the hour. Training needs more memory than inference, so size for the fine-tune and inference comes free.

Sources

• nanochat — karpathy/nanochat on GitHub
• Train your own language model for $100 with nanochat — dig.watch
• LoRA: Low-Rank Adaptation of Large Language Models — arXiv
• QLoRA: Efficient Finetuning of Quantized LLMs — arXiv
• Distilling the Knowledge in a Neural Network — arXiv
• Unsloth — unsloth.ai
• Axolotl — axolotl.ai
• Ollama — ollama.com
• vLLM documentation — docs.vllm.ai
• Best open-source coding models, 2026 — morphllm.com