AutomationOps is easiest to explain through someone’s worst morning, so consider this one. The call reached Priya at 02:14 on a Tuesday. Meridian Booking Systems — a mid-sized travel platform, eleven microservices, two data centres, a booking peak that follows the holiday season — had just lost user sessions across its live nodes. The cause, traced by sunrise, was almost comic: a disk-cleanup job written two monsoons ago by an engineer who had since left, running unattended, had followed a stale configuration file into a directory it should never have touched. Nothing malicious. No attacker. Just an orphan doing what orphans do. The discipline — treating the software that operates your software as a system with its own lifecycle, catalogued for reference at the Atlas’s AutomationOps entry — is what Meridian adopted next, and this fictionalised case study follows the ninety days that followed, because that arc is the clearest demonstration of the ideas we know.
Weeks One to Four: AutomationOps Begins With the Inventory
Priya’s first decision set the tone: no new controls until they knew what they controlled. Arun spent the first week compiling the true estate — crontabs across both data centres, three schedulers, a folder of scripts on a laptop that “somehow” still reached production. The final count was thirty-one unattended jobs. Nine had no identifiable owner. Four shared a single service account with rights neither Priya nor Arun could justify.
Week three introduced the register. Each job received a one-page entry as code: purpose in one sentence, tier, limits, running identity, review date. Tiering followed the Atlas’s proposal — read-only at the bottom, reversible single-target work next, reversible multi-target work above that, and anything irreversible or data-destroying at the top with the strictest requirements. The cleanup job that started the story was placed, without argument, in the top tier, where the requirements are explicit: a second person approves, deletion is staged rather than instant, rollout proceeds in steps. NIST’s dual-authorisation pattern (AC-3(2), CM-5(4)) gave Priya the vocabulary to defend the overhead to management: this is not bureaucracy, it is the same control standard auditors already recognise.
The register surfaced one job nobody could explain. It had run every Sunday for two years, succeeding quietly, doing work whose purpose had left with its author. The team switched it off and nothing happened. That moment — a job nobody missed — did more to convince Meridian’s leadership than any slide. It also reframed the work for the engineers themselves: registration was not surveillance of their craft, but the difference between being the person who secretly keeps a system alive and being the person officially trusted to. Two of the nine ownerless jobs turned out to duplicate each other; consolidating them removed a weekly conflict the team had been absorbing as “that odd Sunday error” without ever investigating it.
Weeks Five to Eight: Fitting the AutomationOps Guards
With tiers assigned, controls followed in order of risk. The multi-target jobs — configuration sweeps, certificate rotations — received three additions. First, a rehearsal mode run before every real execution, with one honest caveat the team learned early: certain command-style steps stay unexecuted under default check modes, so the rehearsal script explicitly exercised the risky probes rather than trusting a green tick. Second, empty-target refusal: any job handed a blank selection halts and pages, because an empty list after a filter change signals a broken upstream — the industry’s shorthand for the rule is that emptiness never grants permission, a lesson Google’s Diskerase backup loss made famous. Third, batching with a canary: small first wave, health checks between waves, and a halt once failures crossed a threshold agreed in advance.
The irreversible jobs were harder. Meridian’s billing reconciliation could not simply stop, so it gained a caller-supplied request ID recorded atomically with each action — the standard remedy for actions that must never repeat silently. Restarts, clean-ups and the like were rewritten toward desired-state form: declare “service running” rather than command “start”, so a re-run corrects nothing because there is nothing to correct. Google’s repair culture was the inspiration here too; the Atlas notes their repair jobs tolerated being replayed on a quarter-hour schedule, because idempotency was designed in rather than hoped for.
Then came the drill. Arun, on call, pulled the global brake on a quiet Saturday morning with Priya watching the clock. Stop-to-verified time: eleven minutes, mostly spent finding one job’s runner. The second drill, six weeks later, took ninety seconds. The Atlas recommends exactly this cadence — twice-yearly stop drills for the riskier tiers — and the rule that whoever carries the pager pulls the switch without seeking permission. Approvals were reshaped in the same fortnight: per-run sign-offs, which reviewers had been waving through, were abolished for everything below the top tier; the top tier kept its second approver, now actually reading, because scarcity had returned to the role. Policy landed as code, too — each register entry checked in the pipeline, so a job drifting outside its recorded limits failed its build before it could fail a night.
Weeks Nine to Twelve: Numbers, Then Calm
The last month was measurement, because governance without numbers is just new opinions. Meridian’s ledger, adapted from the Atlas’s formulas and kept deliberately short — five lines, reviewed monthly in fifteen minutes:
- Net hours returned. Monthly runs, times manual minutes saved, minus residual human minutes, minus maintenance hours, divided into hours. Across eleven jobs: modest but real, and it grew as the team promoted jobs from supervised to unattended runs.
- Human fallback rate. How often people stepped in to rescue a run. Halved once rehearsals started catching template and targeting mistakes.
- Induced-incident rate. Modelled on DORA’s change-failure measure. It answered the question leadership actually cared about: was the automation causing what it prevented? After the guard set landed, the answer settled at zero.
- Guard trip rate. Times the controls stopped something. Each trip was reviewed as a near-miss, not a nuisance.
- Ownership coverage and the unregistered count. Coverage finished the quarter at ninety-one percent; the unregistered count finished at zero.
The quiet ending deserves recording, because case studies usually end at the crisis. Three months after the Tuesday-morning page, a configuration sweep met a broken filter mid-run. The job refused the empty selection, paged, and stopped. Arun cleared the condition in ten minutes from his desk, unheroically, during office hours. No incident review, no apology email, no 02:14 phone call. That is what the discipline buys: not the absence of failures, but the conversion of disasters into errands. Promotion followed evidence rather than optimism — a job moved from supervised to unattended runs only after a month of clean numbers, the same way a pilot’s first solo follows a logged record, not a feeling.
Meridian’s stack, for readers mapping this onto their own platforms, sat on managed Kubernetes, and the site’s coverage of Kubernetes training for production teams describes the platform skills involved. The monitoring that fed Meridian’s candidate list is the subject of the piece on AI-powered IT operations, and their earlier migration off aging hardware — which created the stale paths behind the original cleanup bug — mirrors the story in cloud migration for legacy infrastructure. Teams starting from a similar 02:14 should resist the urge to begin with tooling. Begin, as Meridian did, with the list of what already runs, however embarrassing the list is. The rest of the discipline hangs off that list.
