Wanted a Jarvis. Ended up tinkering.

I've wanted a Jarvis kind of assistant for a while. Seeing Hermes, an AI agent you can run yourself, all over X (Twitter) finally got me to try it on my own VPS. I was curious about how its tools would work on my server, and whether I could get different models handling different parts of the work.

I also had a recurring annoyance: doing research, preparing documents from personal notes, then manually keeping track of everything. I wanted an assistant I could message through Telegram, with files I could inspect and tools running on my own server. Missing personal facts had to stay missing until I supplied them. Search results needed checking against the original page before they became evidence.

That turned into a few weeks of setting things up, breaking things, and getting a little more particular about how I wanted them to work.

First, the server. I updated Ubuntu, set up a separate admin with SSH keys, disabled root and password login, and configured firewalls and automatic security updates. I checked the effective settings with sudo sshd -T, then tested fresh connections. Kept the old session open the whole time because locking myself out would have been a pretty annoying start.

Once Tailscale access worked, public SSH went away. Then a clean snapshot, before adding the agent.

Hermes ran in a version-pinned Docker container. Its notes, configuration and sessions lived in a persistent mount so replacing the container wouldn't erase them. No Docker socket, no public agent ports, and Telegram only accepted my user ID. Its local terminal ran inside that container. That distinction mattered to me.

Getting a reply was one thing. I still had to test search, Chromium navigation and form inputs. DDGS was selectable, but the image was missing its Python package and automatic installs were disabled. A small child image fixed that:

FROM nousresearch/hermes-agent:v2026.8.3
USER root
RUN uv pip install --python /opt/hermes/.venv/bin/python \
    --no-cache-dir 'ddgs==9.14.4'

I checked both the import and normal startup before replacing the container. Later experiments included attaching a persistent browser at 127.0.0.1:9222, private to the container, and testing uploads separately.

Provider limits sent me down another rabbit hole. Switching providers exposed root-owned auth files the Hermes user couldn't read or refresh. I repaired ownership and made sure future commands that wrote agent state ran as that user.

Then I tested three models against the same source files. Each had to answer 20 factual questions, reread specified source lines to verify its answers, and stay within a tool-call budget. All three got the answers right. But Laguna skipped the verification read and claimed it had completed it. StepFun used 19 tool calls and reported 10. For the call counts, I checked Hermes' session database, state.db, rather than trusting the model's summary:

SELECT tool_call_count, api_call_count
FROM sessions
WHERE source = 'telegram'
ORDER BY started_at DESC
LIMIT 1;

Tool calls and model API rounds are separate counters. Hy3 followed the second-read requirement and correctly reported eight tool calls in that test. Reading only the final answers, I would have missed most of the difference.

Google had its own surprise. The installed authorization script requested broader access than I needed, despite its docs describing service selection. I patched it before granting access, narrowed the scope to drive.file, and made credential files private. A local CSV stayed authoritative, with scripts syncing a Sheets view and files into Drive. Hashes prevented repeat uploads of unchanged files; failed syncs left local work intact and raised a warning.

By then I'd also become less keen on automating every browser action. Bot checks, getting stuck, even white text on white backgrounds instructing AI agents what to write. I'd warned mine to expect that, and seeing it flag those instructions was weirdly satisfying. I kept research and document preparation with the agent. Form entry, uploads, consent and submission became entirely mine. The tracker also had to distinguish something prepared from something actually sent.

The custom model router came next: Luna for conversation and tools, Hy3 for bulk analysis, Sol for demanding writing. Hermes' installed delegation tool couldn't select providers per task, so I built a plugin around auxiliary tasks. It checks which model actually answered. Sol could fall back to Sol through another provider, but a fallback to Luna had to be rejected. A successful response alone wasn't enough to prove the route worked.

And after all that, Telegram started barely responding.

I'd left a browser running for days. Checking sudo docker stats --no-stream hermes showed the container near its memory limit; a separate process inspection found 50 Chromium processes. Stopping just that browser brought Telegram back. I didn't even need to restart the container.

That strongly implicated accumulated browser pressure, though I couldn't trace every renderer to the page that created it.

I added cleanup limits and a systemd watchdog outside the agent. It checks memory alongside that browser's processes and stops only that browser if pressure persists, sooner in an emergency. Recovery shouldn't depend on an already struggling agent remembering its instructions. After a full server reboot, I verified that the watchdog returned automatically and the browser stayed off until I started it.

I still like the Jarvis idea. Mostly, though, I like the continuity: context in Markdown files, changes I approve, documents and records I can open in my Google account. It feels like a companion now. And when something it does seems off, I can go look.