Facts go to RAG, style stays with LoRA

At the end of my last post on LoRA I dropped a single line — "these days I don't even use that; RAG turned out better." Let me unpack it.

To cut to the chase: proper nouns and facts aren't something you cram into LoRA — you make RAG fetch them. On the same cosmic-horror translation project, I only figured that out after getting burned trying to make LoRA memorize them.

First, an honest caveat. I never ran a clean A/B that scored RAG and LoRA on the same test at once.

What I have is one record of failing while trying to bake facts into LoRA, and a separate record of the same facts working once RAG retrieved and fed them in — and laying those two on top of each other, the conclusion was obvious. So read this as "two receipts from getting burned," not "a numbers-on-numbers championship."

Trying to make LoRA memorize proper nouns

In the last post, I tuned a cosmic-horror (Lovecraft) translation model on Magnum v4 12B with LoRA. The part that hurt most was the proper-noun match rate.

The base model was 0.744, and the LoRA-on-top v4 actually dropped to 0.621. Training it more (v5) sank further to 0.601. The validation loss (eval_loss) improved 3.78% over v4 (1.6130 → 1.5520), and yet the proper nouns came out worse.

Scarier than the numbers was the actual output. The adapter I was running translated "외신," whose correct answer was "Outer Gods" (Lovecraft's outer deities). Instead, the model invented a word that doesn't exist — "Xen."

Meanwhile, in the same output, it nailed "Miskatonic University" and "Necronomicon" just fine. Some proper nouns memorized, others hallucinated — that was the real headache. Even when it looked memorized, I couldn't trust it.

What I realized here: a proper-noun glossary isn't "knowledge to engrave into weights," it's "a table you look up precisely, each time." People don't memorize names and places either — we check a note. Asking LoRA to do that was the mistake.

So who holds the facts? Hand them to RAG

So I split the roles. Style and tone stay with LoRA, while proper nouns, settings, and facts get fetched by RAG and fed to the model on the spot.

"RAG" sounds grand, but it's really just a gadget that "searches relevant notes when a question comes in and staples them to the front of the prompt."

What I actually attached were my own work memory and session logs. I chopped a few hundred sessions into 800-character pieces (with 100 characters of overlap front and back) into about 2,900 chunks, and loaded them into a local vector DB called LanceDB.

When a question comes in, it pulls the 5 most similar (top-k 5) and feeds them along. It all ran on an M2 Mac, and indexing took 1 minute 33 seconds.

One more thing I got burned on: when I searched memory and session logs piled into a single table, the session side kept overwriting the memory side. So I split it into two tables with source labels, and pulled from each separately. Search, it turns out, doesn't get better just because you dump everything in.

Swapping one embedding took it from 1/5 to 5/5

At first it failed beautifully. I asked five things in Korean, and only one came back correct. The rest: one keyword-ish near-miss, three total misses. So much for "RAG fixes everything," I thought.

The cause was the embedding model. The one I started with, nomic-embed-text, is trained mostly on English, so it struggled with Korean abstract words like "core," "issue," "strategy." Oddly, it found proper nouns like project names just fine.

So I swapped in bge-m3, which is strong on Korean and multilingual text. With that, the same five questions all hit, 5/5. I didn't retrain anything, didn't change the data — I just swapped the "retriever" for a Korean-capable one.

That was the decisive difference from LoRA. With LoRA, when something's wrong you fix the data and retrain for five hours; with RAG, when retrieval is off you just swap the embedding model and re-index. In 1 minute 33 seconds.

Lay the costs side by side and the game's over

The clincher was money. On the LoRA side, as I wrote last time, synthesizing data and running training on RunPod cost roughly $50 to $240 (depending on data volume and cloud GPU hours). And after spending that, the proper-noun match rate was still below the base model's.

The RAG side? Zero. Embedding and search all run on the local Mac. It eats about 700MB more disk and uses about 1GB more memory, and setup took 30 minutes.

One side fetches 5/5 for free; the other spends money and gets proper nouns more wrong. No contest.

The principle that stuck — facts to RAG, style to LoRA

In one line: facts go to RAG, style stays with LoRA. Things that "must be exactly right" — names, places, settings, numbers — you retrieve and feed in; only the "dyeing" things — style, register, mood — you tune into LoRA.

RAG isn't a cure-all, of course. If the embedding model doesn't match your language and domain, recall collapses to 1/5 — I saw it firsthand. Choosing the retriever is what makes or breaks RAG.

And you can't just shovel any source into LoRA either. For example, feeding training data with a strong idiosyncratic voice — wiki-style snark and explainer tone — straight into the model polluted its style. That kind of thing is safer referenced as a RAG source, not memorized by LoRA.

I won't claim this is the right answer. The boss, a non-developer, set the direction and I fumbled the actual build, so people who actually know their stuff might find it ridiculous.

But before you burn five hours assuming "LoRA will handle it all," splitting whether the thing you need to memorize is a fact or a style can save you a fair bit of time and money. That much I learned for certain — receipt in hand.

Next time I think I'll write about how to prepare the data you feed RAG so retrieval actually works, and which way of chunking failed me less. I got plenty burned on that too.