Subscribe to JEM Newsletter
Stay ahead of the curve in Web Development with Javascript Every Month Newsletter.
I will deliver a curated selection of articles, tutorials, and resources straight to your inbox once a month.
Read the Archives
Weekend project: I fine-tuned LiquidAI's LFM2-350M with a LoRA adapter so it became an expert at calling the tools of Herdr, a terminal multiplexer. Base model gets 6.8% on my holdout. The tuned model gets 96.1%. For scale, frontier models score 73.2% (GLM 5.3 flash) and 56.1% (Deepseek flash) on the same task. Training is hours on a free Colab T4; an L4 does it in under an hour.
The idea: instead of a frontier agent fumbling through my tool schemas, a tiny cheap model handles the "which Herdr operation does this natural language request map to" question. The main agent asks it in plain English, gets back an exact tool call.
Herdr has 25 operations. I generated a dataset of about 800 chat rows covering all of them: user requests in natural language, the assistant answering with the exact tool call. About 12% of the rows are off-topic on purpose, including hard negatives that reuse Herdr verbs but act outside Herdr. Those teach the model to refuse instead of hallucinating a call.
Training is standard PEFT LoRA (r=16, alpha=32) with loss masked to assistant tokens only, run on Colab. Training, dataset generation, and eval are all scripted; make data, train on Colab, make fetch, make eval.
The fine-tune was easy. Making the numbers honest was not.
My first eval table looked great. Then I found out the model had been scored partly on rows it trained on: the old training script built its train set as "everything not in val", which silently included the eval holdout. Classic contamination, and my own scripts did it to me.
The fix: split.py is now the single source of truth. The holdout gets carved out first, train/val split from the remainder, and both the trainer and the eval import it. The holdout is also pinned by query string, so appending training rows never shifts it. 96.1% is on 120 rows the model has never seen, strictly disjoint from training.
Honest failure modes, all on held-out rows:
The lesson I keep coming back to: a 350M model beats a frontier model on a narrow task it was tuned for, at a fraction of the cost, and it runs locally as a 219MB Q4 GGUF. It knows one thing and refuses everything else, which is exactly what you want from a tool specialist.
Frontier models are still the right call for general reasoning. But for "map this sentence to one of 25 tool calls", spending tokens on general intelligence is waste.
This is also the start of a longer goal: make a model proficient in how I work. Herdr as the multiplexer, worktrunk for worktrees, hunk for diffs. A small expert per tool, each one fluent in my setup, instead of one general model guessing at all of them.
The adapter and merged GGUF are on Hugging Face if you want to poke at it.
š„