Two Things We Didn't Know About Our Own Stack

October 2026

Last week, two quiet discoveries in my very own infrastructure. Neither one had an impact on operations so they simply ran as dangling efforts never picked back up.

The first discovery started with time drift that uncovered a reboot nobody scheduled. A server was rebooting itself every three days — for five months. As I recall now it was a stopgap to clear memory while I sourced more but the entire VM got migrated with a new processor more memory and disk and not a thought to the lingering cron I left behind. This was not an active agent loose in the environment, but it is an artifact left behind via Agentic engineering: a cron setup at my request, which means a file installed last May, and documented nowhere, and remembered by no one. We only found it because the machine's uptime didn't match the story we'd assembled as part of an RCA.

Our job pipeline is written down as running at 10 AM. When I was setting up the thought was to hit mid morning Monday through Friday. It mostly fired by hand and then more or less randomly until I found one day it landed without my intervention all on its own at 6:31 AM...every day for several months now. It was there at any point in the business day so I never gave it much thought. However, time drift, or issues with event matching between hosts finally showed the primary worker host for my agentic flow runs on UTC time. The other agents were simply adjusting to it and nobody had ever compared the baseline setup doc against the machine. It ran fine. Nobody noticed, because the output still arrived.

Two boring findings. But look at what they share:

Nothing was broken. So the system simply carried on. However, both had been wrong for months, and neither ever threw an error, a warning, or a missed run. Nobody edited them and marked them wrong. They just quietly didn't match the reality me, my agents, and all my documentation thought was in place.

Monitoring found neither of them. Both surfaced because one number contradicted a story — an uptime that didn't fit, a run time that didn't fit what we knew for certain...our long pole for this RCA was a power outage I was present for. The time stamp was in my head as well as in the logs. Sort of locked down the notion of shared reality and provided an anchor my Agent Checking Agents setup had missed.

"It works" is a very low bar often cemented in place by automation that doesn't throw a clear failure. In Agentic Assisted delivery that can create a situation where hallucinations will anchor multiple LLM/Agent frameworks to a false version of reality. To put some skin on this we use a VM template. Two separate stacks independently confirmed EST as the default timezone on the template.... problem is I built the VM by hand. The assumption the template was used and the logic of checking the template led both to confirm the false narrative all Linux hosts are set to EST.

Every stack carries a share of this: the cron nobody owns, the timezone a default picked, the environment variable from a three-year-old tutorial, the "temporary" workaround that outlived the person who wrote it. Silent drift doesn't announce itself — it only becomes visible when something forces a comparison: a story checked against a number, a doc checked against the machine. And we don't make that comparison until something breaks — or until we go looking. What has changed is LLMs now use the tools to hand to confirm their inaccurate findings and tell the human operator "ALL Clear".

Question for those of you using these tools to deliver services back to the business: We talk a great deal about the pitfalls of vibe coding, but Agentic assisted Engineering at Scale has its own problem with increasing the attack surface available. When a single operator can run 4-5-6 Agents with elevated privileges concurrently who's responsible for the footprint left behind? Bottom line it's not just your functional workload, it's the accumulated muck of just standing it up which is increasing at speed. Anyone else found Agentic tech debt quietly *not* working — in your own stack?