An AIOps Readiness Checklist: 22 Checks Before You Buy Anything

An AIOps Readiness Checklist: 22 Checks Before You Buy Anything
Tech How-To

Chennai’s IT corridor along the Old Mahabalipuram Road is full of teams that were handed a monitoring stack years ago and have been adding alert rules ever since. Before anyone buys an AIOps product, there is an audit worth running: a short list of yes-or-no checks that tells you whether the groundwork is ready. This checklist follows the ordering used in the Ops Atlas, which lays out AIOps practices by payoff. Print it, tick the boxes honestly, and treat every “no” as a to-do rather than a failure.

How to run it

Book a room for an hour and a half and share one screen. For every item, record the evidence: a screenshot, a query result or a link to a document. If you cannot produce evidence, score it a no. The reasoning comes from the Atlas page for AIOps, which treats clean inputs as the prerequisite for anything smarter.

Section A: foundations

  1. Do the alert rules actually work? Pick ten from the last month and confirm each one fired on a real problem. The Atlas lists working rules as a prerequisite, not an optional extra.
  2. Does every service have a named owner? Not a team alias that nobody reads, but a person or rota that answers.
  3. Is there a dependency list? A page showing what each service calls and what calls it.
  4. Is there a change log? Deployments, configuration pushes and infrastructure edits, with timestamps.

If any of these four is a no, stop here and fix it. Roughly 70% of outages originate in changes to production, a figure the Atlas takes from the Google SRE book, so without a change log every later step is weaker.

Section B: the label contract

  1. Is a minimum label set defined? In the Atlas sample the set covers the service, owning team, severity, a remediation hint, the affected cluster, the evaluating Prometheus and a runbook URL.
  2. Is compliance measured? A linter that checks rule files before they are merged, and a report on how many live alerts carry every required label. To leave the first maturity level, the Atlas expects 95% of alerts to comply.
  3. Does any label carry two meanings? Search for tags such as “cluster” that sometimes mean the affected system and sometimes the observer. The Atlas describes a worked runbook in which this ambiguity silenced a real outage; the cure was two separate labels.

Section C: deduplication and grouping

  1. Are repeats collapsed by a fingerprint? Check which labels go into the fingerprint and remove anything volatile, such as short-lived pod names.
  2. Is grouping done on meaningful keys? Service and upstream dependency beat a bare time window. The Atlas lab found a five-minute window compressed alerts 5.5 times while merging unrelated incidents.
  3. Is inhibition guarded? Review each suppression rule and ask what it could hide. The Atlas flags a quirk worth testing: if every label named in the “equal” list is missing from both the source and target alerts, the suppression still applies.
  4. Can you list what is currently suppressed? The Atlas suggests the first line of an outage runbook be a command that lists inhibited alerts. If the team does not know it, add it today.

Section D: correlation and detection

  1. Do alerts link to changes and topology? If a deployment happened ten minutes before the first alarm, does anyone see that automatically?
  2. Is the topology current? A lab experiment in the Atlas removed one dependency edge, and pairwise recall slid from 0.93 to roughly 0.53. Out-of-date maps cost real accuracy.
  3. Do calendars exist for planned events? Holidays, sales and maintenance windows. One calendar line removed 32 false alerts in the Atlas seasonal-baseline test.
  4. Do critical alerts need two viewpoints? For cluster-unreachable style alarms, the Atlas suggests firing only if two independent vantage points agree, to avoid being fooled by a broken observer.

Section E: measurement

  1. Is there a labelled incident set? The Atlas rule of thumb is about one hundred before trialling learned grouping.
  2. Do you score precision and recall? Not compression ratio. Precision divides correct hits by everything flagged; recall divides correct hits by everything that should have been flagged.
  3. Is there a cadence? Atlas guidance sets three review beats: noise gets looked at each week, results each month, and the platform’s fitness every quarter.

Section F: safety

  1. Who can reach the alert manager’s interface? The Atlas points out that a permissive cross-origin header on the Alertmanager API means one crafted request could silence an outage. Restrict access.
  2. Do ingest webhooks require authentication? The event-driven Ansible alertmanager source in the collection version the Atlas checks has no authentication, so keep it on the loopback address.
  3. Do automated fixes follow guard rails? Allow-listed actions, reversibility, rate limits, verification, and an escalation to a person on failure. Run scripts as the owner of the log files, never as root.
  4. Is alert text treated as untrusted input? This matters most if any language model reads it.

Typical findings, and what they usually mean

After many audits of this kind, the same patterns tend to appear. They are my observations from the type of exercise rather than Atlas statistics, so use them as prompts and not as facts about your estate.

  • Owner fields filled with a distribution list. Usually means incidents wait for someone to volunteer. Replace with a rota.
  • Label sets that differ between teams. Common after mergers. Pick one set and migrate rule files gradually, with the linter blocking new violations.
  • A deduplication key that includes the instance name. Every restart creates a fresh incident. Remove the changing field from the key.
  • Suppression rules nobody remembers writing. Review each; delete or document.
  • A dependency diagram last edited a year ago. Assign an owner and a monthly refresh.

Evidence worksheet

For each check, keep one line: item number, answer, evidence link, owner of the fix, due date. A simple spreadsheet is enough. Run the audit again next quarter and compare: some items will have moved while others sit still, and the stalled ones are usually the ones that need management attention rather than more engineering effort.

Scoring yourself

Count your yeses. Between 0 and 8, you are in foundation territory: skip platforms and focus on Sections A and B. Between 9 and 16, you are ready to experiment with grouping and correlation changes, and with learned detection in shadow mode, where results are scored but nobody sees them. Above 16, you can plan a canary on one or two services, with a written rollback trigger. These bands are my own rule of thumb, not from the Atlas, so adjust them to your situation.

What to do with the results

Rank every “no” by effort and impact. Labels and deduplication usually come out cheapest and highest in impact. Resist the urge to jump to Section D because it feels more interesting. To build the skills for running an audit like this, read the article on a career journey through DevOps skills, tools and growth gives a sense of the learning ladder.

Finally, a thought for the weekend. The same checklist habit helps when you plan travel: confirm the basics before anything exotic. For example, this feature on planning a first-ever Chennai trip follows that principle for visitors to the city.