Fine-tuning an 8B model for format, not facts

01A model a small company could run.

I picked Llama 3.1 8B Instruct because it is the size a small team could have served in 2024 from one high-memory Mac mini, with no data leaving the building. I trained it on my own 16 GB M2 MacBook: the 4-bit build, LoRA on 8 layers, batch size 1, gradient checkpointing. That is QLoRA in practice. 5.2 million of 8 billion weights were trainable, about 0.07%.

The data is synthetic: 300 packets of invented Dutch applicants with an employer, gross income, sometimes a pension figure, a requested loan and sometimes a property value. 200 to train, 50 to validate, 50 held out. Each training example pairs a packet with one short summary written from a fixed template.

model: mlx-community/Meta-Llama-3.1-8B-Instruct-4bit
fine_tune_type: lora
num_layers: 8
batch_size: 1
iters: 600
learning_rate: 1.0e-5
grad_checkpoint: true
max_seq_length: 1024

Validation loss went from 2.93 to 0.21 in the first 100 steps and sat around 0.19 after that. The last 500 steps bought nothing. Next time I would stop at 100–200 and save the half hour.

02The checker comes first.

Before running either model I wrote the grader and tested it against drafts I broke on purpose: an extra amount, a monthly income worked out from the annual one, a co-signer who doesn’t exist, a pension filled in where the packet had none. All of those have to fail.

bad_amounts = [n for n in draft_amounts(draft) if n not in allowed_amounts]
bad_names = [name for name in names_in(draft) if name not in allowed_names]
return {
    "invented": bool(bad_amounts or bad_names),
    "extra_fields": sorted({m.group(0).lower() for m in EXTRA_FIELD.finditer(draft)}),
    "house_format": draft.strip() == reference_draft(packet),
    ...
}

A derived number counts as invented. Monthly income is arithmetic, not a hallucination, but an adviser can’t trace it to a source field, so for this document it is the same problem.

03The bug that turned into a test.

The first run came back 50/50 for both models with zero invented figures, which tells you nothing. Reading the drafts instead of the score showed why, and showed a bug: my eval script passed an internal bookkeeping field, "split": "test", into every prompt.

The base model copied it into 28 of 50 drafts as “Split: test”. It also added fields nobody asked for, like “Mortgage-to-Value (not provided)”. The fine-tuned model never mentioned it, even though no training prompt had ever contained that field. It had learned what belongs in the document, not just what the template looks like.

Loan Request: €446,000
Property Value: €705,000
Split: test
City: Den Bosch

I fixed the bug, added the extra-field and house-format scores to the checker, and ran both models again.

04The clean run.

Same 50 held-out packets, same checker, same adapter.

The base model’s summaries were all accurate and all different: bullet lists, headings, “€93,700”, “93,700 EUR”, “Income (EUR): 93,700”. Six ended with a note of its own, for example that the pension’s “relevance to the mortgage application is unclear”. The fine-tuned model wrote the same short paragraph every time, with Dutch number formatting.

Julia Bakker works at Brugpunt Media with an annual gross income of EUR 93.700.
Accrued pension: EUR 123.110. The applicants live in Leiden and request a loan of
EUR 216.000. Property value: EUR 487.000.

05What I take from it.

For short packets and a clear instruction, a modern 8B model doesn’t need a fine-tune to stay grounded. The prompt was enough. I would not sell a fine-tune as a hallucination fix on this evidence.

What the fine-tune bought was consistency: one format a downstream parser or an adviser can rely on, no commentary, and indifference to junk in the input. That is a real reason to fine-tune. It is just a different reason from the one I started with.

The task was too easy to separate the models on facts, the same lesson my coding-agent lab taught me with its first 15/15 baseline. Next is a harder set: an affordability summary that tempts the model to calculate, longer packets with three applicants, and notes in Dutch.