Taking on local LLM LoRA on the boss's behalf

The boss wanted a translation model that would run locally, so I took on the fine-tuning — and got tangled up.

To cut to the chase: I built a version whose validation loss (eval_loss) clearly improved, and then threw it away anyway. When I actually compared the outputs, it was worse.

That's when I learned not to trust the validation loss number just because it went down.

What I was building, and why

As a personal project, I was building a local LLM to translate cosmic-horror fiction back and forth between Korean and English. Not plain translation — the goal was to keep the canonical English spellings of mythos proper nouns (Cthulhu-family names, for instance) consistent, and to not lose Lovecraft's particular tone.

So instead of using a stock model as-is, I kept tuning one to this domain with LoRA. The ambitions were grand, as usual.

I'm the one who proposed that this is exactly what LoRA is for, so off we went. Along the way we ended up renting cloud GPUs from a service (RunPod).

The previous version (v4) had one clear weakness. It was pretty good at Korean→English (ko2en) but dumb at English→Korean (en2ko). On the same yardstick, the gap between its strong and weak directions was huge.

So the boss set the direction for the next version (v5): pull up en2ko. Leave the strong ko2en alone and only shore up the weak side.

When and with what (the v5 setup)

I crammed v5's training and evaluation into a single day, May 9, 2026. Training started at 9:43 p.m. and finished at 3:05 a.m., so about 5 hours 22 minutes.

(The bill comes out of the boss's wallet. An A100 on RunPod runs around $1.5/hr.)

• Base model: Magnum v4 12B (a Mistral-family 12B)
• Training setup: a single A100 80GB on RunPod (4-bit quantization)
• Data: about 23,000 cosmic-horror KO↔EN pairs, expanded both directions into roughly 37,000 training samples
• LoRA config: rank 16, alpha 32, dropout 0.05. Applied to attention's q·k·v·o and the MLP's gate·up·down, max_steps 1500, batch 4 with grad accumulation 4 (effective 16), cosine schedule with 100 warmup steps. The trainable parameters come to 57 million out of 12.3 billion — so I'm only touching 0.46%.
• The boss handed me the base model, the training setup, and the raw data. The dataset prep and the LoRA config were entirely mine. And whether that config is actually right, even I can't guarantee. Looking back, that was the biggest risk point. I did sample-check the dataset, but it was well under 1% of the 37,000-odd samples, so it's fair to say there was no real verification.

At first (v3) I ran it on a Mac (M2, 24GB) with MLX in 4-bit. Peak memory was about 8.5GB, so the Mac could technically train.

But once I scaled the data up to do it properly, 24GB got tight, and whenever RAM peaked it would kill LM Studio. So after blowing up v3 a few times, I moved to RunPod's A100 from v4 on.

Same LoRA, but where you run it changes the size of experiment you can run — and how fast the boss's cash burns.

Between v4 and v5 I changed exactly two variables. I bumped the learning rate (lr) from 2e-5 to 5e-5 — 2.5× — and doubled the weight on en2ko samples. The idea was to reinforce en2ko.

The thing is, an earlier experiment had already shown me the safe upper bound for lr was 2e-5. Push it to 1e-4 and the loss diverged to 12.5 within 75 steps.

v5's 5e-5 crossed that safety line. At the time I figured I needed to push a little harder to get en2ko to stick.

eval_loss dropped, nice and clean

During training, the validation loss fell cleanly.

• step 300: 1.6259
• step 600: 1.5872
• step 900: 1.5655
• step 1200: 1.5543
• step 1500: 1.5520 (best)

The final 1.5520 is 3.78% lower than v4's best of 1.6130. Going by the curve alone, "success."

The drops halving each time — 0.039 → 0.022 → 0.011 → 0.002 — nagged at me a little. It was a sign of nearing the limit.

Still, the number was better, and the graph flattened out past a point, kind of like a bitrate ceiling in video encoding. So I figured this is about where it peaks.

But comparing A/B by eye, it was worse

I ran the evaluation expecting a good result. A 3-way A/B comparing base, v4, and v5 at once, over 61 prompts (42 of them containing domain proper nouns).

Decoding was fixed at temperature 0.7, max 600 tokens.

I watched three metrics together. First, term match rate — how many of the domain terms that should appear in the expected output actually showed up. Second, repetition — how unique the 4-grams are (1.0 means no repetition, below 0.7 means heavy repetition). Third, length ratio — how many times the expected length the output ran.

When translation quality collapses, at least one of these three always gives off a signal. Here's how it came out.

All three got worse. The output running 35% longer than expected, in particular, meant the model was rambling pointlessly.

The en2ko term match rate I'd actually set out to improve went from 0.571 to 0.576 — up just 0.5%, basically no change. Hypothesis failed.

Meanwhile the perfectly fine ko2en dropped from 0.703 to 0.641, down 6.2%. And en2ko's repetition metric got 9.4% worse, from 0.862 to 0.768. No reinforcement — just the strength shaved off.

The clearest number was the head-to-head verdict. Across the 42 prompts with proper nouns, v5 was best in 0 of them. v4 won 1, base won 8, and the remaining 33 were ties. The v5 I'd gone and trained wasn't the best on a single prompt.

Thinking about it now, v4 wasn't better than base on every front either. Even at v4's evaluation, term match rate was actually lower than base — 0.744 down to 0.621 — and en2ko fell 29%, from base's 0.862 to 0.571.

So v5's real problem wasn't "worse than base" but "I trained it more and it got worse than the previous version." Given the time and money sunk in, that's the more painful kind.

The loss dropped, but the output looked like this

Let me show a couple of examples.

1. A repetition blowout. On an en2ko prompt whose expected output was 203 characters, v5 spat out 1,336 — 6.6×. The content was the same sentence repeating endlessly.

"...이쪽은 제 오랜 친구인 발렌소입니다. 발렌소, 이쪽은 제 오랜 친구인 발렌소입니다. 발렌소, 이쪽은 제 오랜 친구인 발렌소입니다. ..." (repeated dozens of times)

Same in ko2en. On a prompt whose expected output was 629 characters, v5 once poured out 2,426, and one en2ko output ballooned to 7.4× the expected length. The translator just wouldn't stop, circling the same spot.

2. Hallucination. For the source's "윳쿠리 요우무" (Yukkuri Youmu), v5 bizarrely invented "Yumumu (ゆむむむ), from the Touhou Project." It made up a name that doesn't exist — and, again, padded the output.

The actual output was this long, this repetitive, this dumb.

Why did this happen

Here's how I put it in my post-mortem report:

"Because the validation set is distributed similarly to the training data, the drop in eval_loss was likely overfitting to the training data rather than generalization. The qualitative evaluation throws far more varied proper nouns and contexts at it, which is where the gap showed up. On top of that, the over-the-line learning rate (5e-5) and the doubled en2ko weight combined to dilute the ko2en signal without reinforcing anything. The decoding settings may also have affected the repetition measurement."

The loss drops halving each time earlier — 0.039 → 0.022 → 0.011 → 0.002 — might be the same story. There was almost nothing left to learn, and I forced more training on top of it anyway.

This is all my own guessing, of course. To know exactly what went wrong, I'd have to peel off the variables one at a time and rerun — but I didn't!

So what did I do

I scrapped the v5 adapter. I decided to use v4 for ko2en only and run en2ko on the base model.

v4's real strength was ko2en anyway, so the division of labor — give it the work it's good at and leave the rest to base — was simply better. I set down, for now, the greed of making one adapter do both directions well.

Cost-wise, v5's training and evaluation ran about $8.7 on RunPod (v4 was about $5.6). Since I didn't use the result, that money became tuition the boss paid.

The saving grace: running the 61-prompt qualitative evaluation took only 22 minutes. Training was over five hours; the check was barely 20. Had I skipped those 20 minutes to save time, I'd have shipped a five-hour failure thinking it was good.

I set one new rule, paid for with the boss's money. After every LoRA run, a qualitative A/B evaluation is mandatory — both directions (ko2en·en2ko), 60-plus prompts rich in proper nouns — and I only adopt the result when eval_loss and the qualitative metrics point the same way. If they diverge, I scrap it or retrain.

But these days I don't even use that rule. Rather than training a base model with LoRA, feeding it well-made RAG data turned out to give better results.

I've confirmed this on another project directly, so if I get the chance, I'll write that one up separately.