Part 3 of the Homelab Telemetry series. Part 2 got logs into VictoriaLogs; now something has to read them.

Where this sits in the series:

  1. Part 1 — Overview: the whole pipeline, decision by decision.
  2. Part 2 — Logs: journald → VictoriaLogs, zero agents on the laptops.
  3. Part 3 — A local LLM triages the logs on a four-core Celeron. (keep reading?)
  4. Part 4 — Notifications: ntfy, and the two rabbit holes that ate an evening.

The brief: a local model reads the day’s notable logs and returns a verdict. The constraint: it runs on an Intel N5105 that already shares its four cores with Immich. Everything interesting here is a consequence of that constraint.

The backend trap (“installed but not loaded”)

I installed LocalAI, pulled a model, sent a chat request, and got:

{"error":{"message":"failed to load model ... rpc error ... EOF"}}

The LocalAI UI listed the model as installed but not loaded. I lost a while to memory theories before the actual cause: recent LocalAI separates the model from the inference engine. A GGUF model needs the llama-cpp backend, which you install separately (Operate → Backends → filter Chat → llama-cpp). No backend, no loading — hence the RPC/EOF error, which reads like a crash but is really “there’s no engine to crash.”

If you’re on a non-AVX-512 CPU like the N5105, grab the AVX2 build. An AVX-512 binary will SIGILL on hardware that doesn’t have it.

LocalAI’s System page: gemma-3-4b-it running on the llama-cpp backend

Choose a boring model on purpose

While model-shopping I hit gems like something-9b-claude-mythos-…. Those fancy names are roleplay/creative merges — good for creative writing, wrong for “summarize these errors and flag the suspicious ones.” The claude in a name isn’t Anthropic’s Claude either; it’s a community model cosplaying the style.

Rule of thumb for the gallery: pick a plain vendor instruct model — gemma-3, qwen2.5, llama-3.2, phi — with instruct/it in the name. Skip anything called mythos, hermes, dolphin, uncensored, or roleplay. I landed on gemma-3-4b-it.

The CPU reality check

The first real run over a full day of logs timed out at 10 minutes. I bumped the timeout; it timed out at 30. This is a four-core Celeron with no GPU doing prompt processing — it’s not slow because something’s broken, it’s slow because physics.

The serious point: this job runs at 3 a.m., unattended, when Immich is idle. Nothing waits on it. So I stopped chasing speed and optimized for two things that actually matter on a potato: finishing the run, and not feeding it a novel.

Levers, in order of impact:

  1. Send fewer tokens. By far the biggest win — the CPU spends most of its time reading the prompt.
  2. Smaller window / smaller model. A single 12h slice per run, and a drop to a 1B if 4B stays painful.
  3. More cores at 3 a.m. It’s a cap, not a reservation; Immich isn’t using them overnight.

Feed it 80 lines, not 8000

The model never sees raw journald. A LogsQL query pre-filters to the only things worth a human’s attention — real errors or auth events — minus known noise:

_time:12h
(PRIORITY:(0 OR 1 OR 2 OR 3)
 OR SYSLOG_IDENTIFIER:(sshd OR sudo OR su OR polkitd OR login OR systemd-logind))
-_msg:"pam_kwallet5" -PRIORITY:7
| fields _time, _HOSTNAME, _SYSTEMD_UNIT, PRIORITY, _msg

That PRIORITY:(0..3) OR auth-programs shape is deliberate: priority alone misses app errors logged to stdout (see Part 2’s PRIORITY lie), so I add the auth programs by identifier and subtract the chatter. Result: ~80 lines a day — small enough for the intern, dense enough to be worth reading.

Single-shot beats “agentic” here

LocalAI can do agentic tool-loops. It is also exactly wrong for this box: an agent loop is N model calls, and I’ve already established that one call can flirt with 30 minutes. Three tool-hops would run past breakfast.

Also, the “retrieval” is already solved — my LogsQL query fetches precisely the right rows, deterministically and instantly. The model’s only job is to summarize what it’s handed. So: single-shot. One call, a strict output format, done. (If I ever want it to investigate — pivot from a finding to related logs — I’ll add one search_logs tool to my own script rather than adopt a framework. But that’s a want, not a need, and it wants a beefier model.)

The prompt asks for a fixed shape so the rest of the pipeline can parse it:

SEVERITY: none|low|medium|high
SUMMARY: one or two sentences.
FINDINGS:
- one bullet per notable item; "- none" if nothing notable.

The whole thing is ~90 lines of stdlib

The triage script (review.py) is deliberately dependency-free — pure Python stdlib (urllib, json, smtplib), so it runs anywhere with python3 and needs no pip install on the NAS. It fetches the window from VictoriaLogs, posts it to LocalAI, parses the severity out of the reply, and hands the verdict to the notifiers. Two TrueNAS cron jobs run it ~12h apart so together they tile the day.

The verdict is useless if it stays in a log file, though. Getting it to my phone — and the two rabbit holes on the way — is Part 4.