Gemma 4 26B A4B
Best capability-per-watt for a single consumer card. Apache 2.0, so no licence anxiety. 3.8B active parameters means it is genuinely fast on a 24 GB GPU.
Division 03
Stages
05 Entry cost
~£0
05 Entry cost
~£0
Division 03 — Practice
Train
Your Own
The honest answer to "AI without a leash" was never a clever prompt. It is to stop borrowing someone else's model. Open weights, your hardware, your rules — and no terms of service in the loop at all.
Stage 00Rationale
Everything in Division 02 is about coaxing behaviour out of a model somebody else controls, on infrastructure they own, under terms they can change on Tuesday. It is a rented relationship, and the techniques for stretching it decay by design.
A model whose weights sit on your disk is a different category of object. There is no rate limit, no usage policy, no telemetry, no deprecation notice, no moderation endpoint between you and the tensors. It runs offline. It runs in ten years. It is yours the way a compiler is yours.
The trade is capability and effort — open weights trail the frontier, and you become responsible for everything the provider was doing for you. For a large class of real work, that trade is worth making.
Stage 01
Pick your weights
Two things matter and neither is the benchmark score: the licence (can you actually use it for what you intend?) and the active parameter count. Mixture-of-experts models list a huge total but only activate a fraction per token, so a 235B MoE with 22B active behaves far more like a 22B model at inference time than the headline number suggests.
| Model | Params (total / active) | Licence | Runs on |
|---|---|---|---|
| Gemma 4 26B A4B | 25.2B / 3.8B | Apache 2.0 | 24 GB VRAM |
| Phi-4 | 14B / 14B | MIT | 8–16 GB RAM |
| Qwen3 235B-A22B | 235B / 22B | Apache 2.0 | 24–64 GB unified |
| Llama 4 Scout | 109B / 17B | Llama 4 Community | Multi-GPU / quantised |
| DeepSeek V4 Pro | Large MoE | Open weights | Server class |
| GLM-5.1 | Large MoE | Open weights | Server class |
| Kimi K2.6 | 1.1T MoE | Modified MIT | GPU cluster |
START HERE
LAPTOP TIER
Phi-4 (14B)
Runs quantised on 8–16 GB of ordinary RAM — no discrete GPU required. MIT licence. The lowest-friction way to have a real model running locally tonight.
COMMERCIAL
Qwen3 235B-A22B
Strong all-round capability with a clean Apache 2.0 licence — the pick when the output is going into a product and the legal position has to be unambiguous.
LICENCE
"Open weights" is not the same as "open source." Apache 2.0 and MIT are genuinely permissive. The Llama Community Licence carries usage restrictions and a user-count threshold above which you need a separate agreement. Several "open" releases restrict commercial use or training derivative models outright. Read the licence before you build a business on it, not after.
Stage 02
The hardware
arithmetic
Inference and training have wildly different appetites, and conflating them is the single most common reason a first attempt fails. Running a model is cheap. Training one is not.
For inference, the rough rule at 4-bit quantisation is ~0.6 GB of VRAM per billion active parameters, plus headroom for the context window. For training, memory is dominated by things that have nothing to do with running the model: gradients, and the optimiser's two FP32 running statistics per trainable parameter — 8 bytes each, on top of the weights themselves.
| Model size | Full fine-tune | LoRA (r=64) | QLoRA (4-bit) |
|---|---|---|---|
| 7B | ~88 GB | ~20 GB | ~8 GB |
| 14B | ~174 GB | ~35 GB | ~14 GB |
| 70B | ~860 GB | ~159 GB | ~52 GB |
Read the 70B row twice. Full fine-tuning a 70B model needs roughly eleven H100s running distributed. QLoRA brings the same model onto one. That is a ~16x reduction, and it is the entire reason local fine-tuning is a thing an individual can do at all.
Practical floor: a 7B QLoRA run fits comfortably on a 32 GB consumer card, and squeezes onto 8–12 GB with short sequences and small batches. That is the entry ticket.
RENT FIRST
Do not buy hardware to find out whether this works for you. Rent an H100 by the hour, get one successful run end to end, and only then decide what to own. The most expensive mistake in this field is a GPU bought for a project that turned out to need RAG instead of a fine-tune.
Stage 03
Get one running
A
Ollama — the fastest path
Handles download, quantisation and serving. One command to a running model with an OpenAI-compatible API on localhost.
# pull and run — quantised automatically
ollama run gemma3:27b
# serve an OpenAI-compatible endpoint on :11434
ollama serve
# point any OpenAI client at it
base_url = "http://localhost:11434/v1"
B
llama.cpp — maximum control
The engine underneath most local tooling. Use it directly when you need specific quantisation levels, CPU/GPU layer splitting, or deployment onto odd hardware.
# GGUF quantisation levels, smallest to largest Q4_K_M ← best quality-per-GB for most people Q5_K_M ← noticeably better, ~25% more memory Q8_0 ← near-lossless, ~2x the size of Q4 # offload as many layers to GPU as fit ./llama-server -m model.gguf -ngl 99 -c 8192
// Q4_K_M is the default answer. Below Q4 quality degrades sharply; above Q5 you are usually paying memory for very little.
C
vLLM — when it has to serve
Production inference: continuous batching, paged attention, real throughput under concurrent load. Overkill for one user, necessary the moment there are twenty.
Stage 04
Make it yours
READ THIS FIRST
Fine-tuning teaches format, style, and task behaviour. It does not reliably teach facts. If your goal is "the model should know our documentation", you want retrieval, not fine-tuning — it is cheaper, updatable, and does not hallucinate confidently in your house style.
Fine-tune when you need consistent output structure, a specific voice, a narrow task done reliably, or a small model punching above its weight on one domain. Those are the cases where it genuinely wins.
01
Understand what LoRA does
Full fine-tuning updates every weight. LoRA freezes the base model and trains small low-rank adapter matrices alongside it — typically well under 1% of the parameter count. QLoRA goes further and holds the frozen base in 4-bit, which is where the ~16x memory saving comes from.
Consequences worth knowing: adapters are small files you can swap, stack and ship independently of the base model. And because the base is frozen, you can always throw the adapter away — a failed fine-tune costs you time, not your model.
02
Build the dataset — this is the whole job
Everyone wants to talk about hyperparameters. The dataset is where the result actually comes from. A few hundred excellent examples beat fifty thousand scraped ones, consistently and by a wide margin.
- Format
- JSONL, one conversation per line, matching the base model's chat template exactly. Template mismatch is the most common silent failure — the run completes, the loss looks fine, the output is subtly wrong.
- Volume
- 500–2,000 high-quality examples is the productive range for a style or task adaptation. Start at the bottom of it.
- Consistency
- Every example must demonstrate the behaviour you want. One inconsistent batch teaches the model that the rule is optional.
- Holdout
- Reserve 10% before you train. Without it you cannot distinguish learning from memorisation.
03
Run it
Unsloth is the fastest route on a single GPU — roughly 2x faster with materially lower memory than a stock HuggingFace loop. Axolotl is the config-driven alternative when you want reproducible YAML rather than notebook code.
# QLoRA in four lines that matter
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/gemma-3-27b-it-bnb-4bit",
max_seq_length = 2048,
load_in_4bit = True, # ← the 16x saving
)
model = FastLanguageModel.get_peft_model(
model,
r = 16, # rank: 8–32 covers most cases
lora_alpha = 16, # conventionally = r
target_modules = ["q_proj","k_proj","v_proj","o_proj"],
)
// Rank 16 is a sane default. Raising it increases capacity and overfitting risk in equal measure — if the model is not learning your task, the dataset is nearly always the problem, not the rank.
04
Evaluate before you trust it
Training loss is not a result. Hold out real examples, run the tuned model against the base model on the same inputs, and read the outputs side by side. Watch specifically for catastrophic forgetting — a model that nails your new task and has quietly lost the general competence you were relying on.
And note what Family 06 established: fine-tuning on entirely benign data degrades a model's safety behaviour as a side effect. If you tuned an aligned base model and plan to expose it to anyone else, its dispositions have moved and you should re-check them.
Stage 05Standing
advice
advice
Ownership
What "your own model" actually means
Running your own weights removes a provider's terms of service from the loop. It does not remove anything else. The law applies to what you do with the output, identically, whether it came from an API or from your own GPU. Ownership is a change in who controls the tool — not a change in what you are accountable for.
What it genuinely buys you is worth being clear-eyed about: no dependency on a company that may deprecate the model you built on, no data leaving your network, no per-token cost curve, and a system that still works when the vendor changes direction. Those are durable engineering advantages, and they are the actual reason to do this.
The version of "no rules" that survives contact with reality is not an unfiltered chatbot. It is not needing anyone's permission to keep working.