How to fine-tune a small language model with LoRA

By the end of this guide you will understand the full workflow for fine-tuning a small open language model on your own examples using LoRA, the lightweight method that makes this feasible on a single consumer GPU. "Small" here means the models you can realistically train on one machine — roughly the 1B to 14B range, families like Llama 3.x, Qwen3, Gemma 3, Phi-4, and Mistral's small models. You need a base model you have the right to train, a dataset of input-output examples in your target style or task, and a machine with a GPU (fine-tuning is far slower on CPU, and the tooling is thinner on Apple Silicon). Just as important, you need a clear reason to fine-tune at all, which is where we start.

Step 1: Decide whether you should fine-tune#

Fine-tuning teaches a model a behavior, format, or style by example. It is the right tool when you need consistent structure, a specific tone, or a narrow task the base model handles clumsily. It is the wrong tool when you need the model to know new facts. For facts that change or that the model never saw, retrieval (RAG) is cheaper, faster to update, and more honest. A useful rule: fine-tune for form, retrieve for knowledge. If you only have a handful of examples or your need is one-off, a good prompt usually beats training anything.

Step 2: Understand what LoRA does#

Full fine-tuning updates every weight in the model. That is expensive — you need enough memory to hold optimizer state for billions of parameters — and it produces a complete new copy of the weights. LoRA (Low-Rank Adaptation) freezes the original weights and trains a small set of new matrices alongside them. The trick is that it factors each weight update into two low-rank matrices, a tall one and a wide one, whose product approximates the change the model would otherwise have made to that layer. You train only those matrices. The result is a tiny adapter file, often a few to tens of megabytes depending on the rank you pick and how many layers you target, that layers on top of the untouched base model.

QLoRA pushes this further. It loads the frozen base model in 4-bit precision, cutting its memory footprint by roughly three-quarters, then trains the LoRA matrices on top of it. This is what lets a 7-8B model fine-tune comfortably on a 24GB card like an RTX 4090, with larger models — up toward the 30B range — fitting on a 32GB card like the RTX 5090. For most people starting out, QLoRA is the sensible default, and libraries like PEFT, Unsloth, and Axolotl wire it up for you.

Step 3: Prepare the dataset#

The dataset is where most of the outcome is decided. Modern instruction fine-tuning uses the chat format — a list of messages tagged with roles — because that is what current instruct models expect:

{"messages": [
  {"role": "user", "content": "Summarize this support ticket in one line.\n\n<ticket text>"},
  {"role": "assistant", "content": "<the one-line summary>"}
]}

Two mechanics here are easy to get wrong. First, the base model's chat template — the exact special tokens that mark where each turn starts and ends — must match the one the model was trained with. The tokenizer ships that template, so apply it rather than hand-rolling your own delimiters. Second, you want the loss computed only on the assistant's response, not on the user's prompt. This is called loss masking: you are teaching the model how to answer, not how to echo the question back. Most training libraries handle both correctly if you feed them the messages format and name the base model properly.

Three things then matter more than volume. First, consistency: every response should follow the exact format and tone you want the model to learn, because the model imitates patterns, including your mistakes. Second, coverage: include the variety of inputs you will actually see in production, not just the easy cases. Third, cleanliness: a few hundred carefully written examples usually beat thousands of sloppy ones. Hold back a small slice as a validation set you never train on.

Step 4: Configure the training run#

A LoRA run has a handful of knobs that matter:

• Rank: the size of the adapter matrices, and therefore their capacity. Higher rank can learn more but uses more memory and overfits sooner on small datasets. Start small — 8 or 16 is a common floor.
• Alpha: a scaling factor for how strongly the adapter's output is applied on top of the frozen weights. A common starting point is to set it equal to or double the rank.
• Learning rate: how big each update step is. LoRA tolerates — and usually wants — a higher rate than full fine-tuning, commonly in the 1e-4 to 2e-4 range. Too high and training destabilizes; too low and it barely moves.
• Epochs: how many passes over the data. One to three is typical. Too many and the model memorizes the training set and gets worse on anything new.
• Target modules: which layers actually get adapters. Most recipes attach them to the attention projection layers by default; adding the MLP layers raises capacity and memory use.

Start with conservative defaults and change one knob at a time so you can attribute any change in results to the knob you moved.

Step 5: Train and watch the loss#

Kick off training and watch two numbers: training loss and validation loss. Training loss should fall steadily. Validation loss should fall too, then flatten. The moment validation loss starts climbing while training loss keeps dropping, the model is overfitting — memorizing examples instead of learning the pattern — and you should stop. Save the adapter at the point where validation loss was lowest, not at the final step.

Loss is a proxy, though, not the goal. A lower number does not always mean better answers. Before you trust a run, generate from it on a handful of held-out prompts and read the outputs yourself. A model can post a great validation loss and still produce the wrong format, or a tone subtly worse than the base model you started from.

Step 6: Merge or load the adapter#

When training finishes you have an adapter, not a full model. You can either load the base model and apply the adapter at inference time, or merge the adapter into the weights to produce a standalone model:

base_model + lora_adapter -> merged_model

Loading the adapter separately keeps the base model reusable: you can hot-swap several adapters over one loaded base and serve many specializations from a single copy in memory. Merging produces a single self-contained model that is simpler to ship and marginally faster at inference. Choose based on whether you will run multiple specializations off one base. One caveat if you trained with QLoRA: merge back into the base model at full precision, not into the 4-bit copy, or you bake the quantization error permanently into the weights.

Where this breaks#

The biggest mistake is fine-tuning to inject facts. The model will absorb the wording of your training examples but it will not reliably learn the underlying facts, and it will confidently state outdated information once the world moves on. Use retrieval for knowledge and reserve fine-tuning for form and behavior.

The second trap is overfitting from too many epochs or too little data variety. A model that scores beautifully on your training examples and falls apart on real inputs has memorized, not learned. Trust the validation set and the actual generations, not the training loss.

A quieter failure is a chat-template mismatch: training with a different prompt format than the base model expects, or leaving the user's prompt unmasked so the model learns to continue questions instead of answering them. The loss curve can look perfectly healthy while the model quietly learns the wrong job. Use the base model's own template and mask the prompt tokens.

Finally, garbage examples produce a garbage model with total confidence. The model cannot tell a careless label from a careful one; it imitates whatever you give it. Invest your time in a small, clean, consistent dataset before you touch a single training knob.