It begins on a humid Saturday morning, the busiest booking hour of the week for a small Chennai walking-tour company. The owner’s phone fills with messages: the payment page will not load for some customers, though others pay without trouble. Support insists nothing was changed. The developer finds no errors. For two hours nobody can explain why a slice of visitors is being turned away. The cause, when it finally surfaces, has nothing to do with the tour business directly and everything to do with how threat information gets used. This is a diagnosis story, and the cure turns out to be a ThreatOps habit.
Symptom: some customers locked out, no error anywhere
Patchy failures are the hardest kind. When everyone fails, you hunt for a broken part. When a random-looking subset fails, you hunt for a rule. The owner’s checklist that morning shows the website is up, the payment provider’s status page is green, and the hosting dashboard is calm. Customers on one mobile network cannot reach checkout, while customers on a broadband line sail through.
That asymmetry is a clue. Something is deciding who may pass, and it is doing so by address. The team eventually finds a firewall rule added a week earlier, fed from an outside list of addresses labelled suspicious. The rule had been switched from observation to automatic blocking after a colleague read that blocking was “more secure”.
Root cause: a list trusted beyond its record
The Ops Atlas describes this exact family of mistake. It says that an early-stage programme should treat outside lists as material for flagging and alerting, and that unattended blocking calls for measured precision, a strong source grade and a check against the organisation’s own allowlists. The tour company had none of those. Nobody had counted how often the list was right. The list included addresses from a shared delivery network, the kind of infrastructure that serves thousands of innocent sites alongside the odd bad one, so a legitimate payment widget was caught in the net.
The Atlas offers a worked runbook for precisely this event. The block is reversed within minutes by whoever is on call, the offending feed is moved to warning-only before the hour is out, and the team tallies reversed blocks monthly as a health signal. The tour company had no runbook, so the repair took the whole morning.
The fix, in order
- Undo first, explain later. The developer disabled the rule. Payments resumed within a minute. Rolling back must be allowed for any person on duty, with no committee required.
- Demote the source. The list went back to observation mode, where it raises a notice and blocks nothing.
- Write down the entry. A short note recorded what was blocked, for how long, and which customers were affected, based on payment attempts that failed during the window.
- Review within the hour. The owner checked that the revert held and no other rule used the same source.
- Add the lasting guardrails. This is where the ThreatOps work begins, described below.
Prevention one: grade the source and measure it
The team created a small register with one row per outside list. Each row records who supplies it, who reads it and a grade. The grading approach the Atlas outlines gives the provider a letter and the claim a number, so a respected supplier’s new list starts around the middle rather than at the top. The register also gets a measured column: of the last hundred flags, how many turned out to be real? When the Atlas’s worked example scored a list at about 15 percent, with 18 true hits in 120, it concluded the list should stay in observation mode and become a candidate for cancellation. The tour company’s own list, once counted, looked uncomfortably similar.
Prevention two: give every entry a shelf life
Old entries cause as much damage as bad ones. An address that was hostile in the spring may be a customer’s phone in the autumn. The standard format for exchanging such items allows the end date to be left empty, which the Atlas reads as no limit at all. The tour company now stamps lifetimes on entries and prunes weekly, with shorter lives for addresses than for file fingerprints.
Prevention three: a narrow question instead of a general worry
The incident exposed a deeper problem. The company had adopted a blocking list because of a vague fear, not because of a question. The Atlas expects every standing question to be built around one request, one piece of evidence and one choice to be made. After the incident the owner rewrote the vague fear into three: Are fake rebooking emails going to operators like us? Is anyone replaying stolen passwords on our booking page? Are impersonators asking our suppliers to change bank details? Each points to something the company can do.
Prevention four: look, do not just block
Blocking only stops what you already know about. Hunting finds what you do not. Once a fortnight the team now runs a one-page search guided by a hypothesis, such as “a customer account is being used from several cities within an hour”. Each search ends in something filed. A quiet result still counts for the Atlas as long as the logs were complete enough to have exposed the behaviour; if they were not, the output is a ticket asking for better logs.
The Atlas also names an anti-pattern worth remembering: using the hunting slot as overflow for the daily queue of alerts. The tour owner protects the fortnightly hour in the diary, the way a guide protects the early slot before the crowds arrive.
What the owner learned about people, not software
The technical repair was quick. The slower lesson was cultural. The colleague who switched on blocking had acted in good faith, and the first instinct of several staff was to hide the mistake rather than report it. The owner made a point of thanking that colleague in front of the team, because the Atlas’s approach depends on anyone on call feeling free to reverse a block without fear of blame. A rule that nobody dares to undo is a rule that fails the next customer too.
The owner also asked the support team to log every customer message that mentioned failed payments, with the time and network if known. Those notes became the evidence that tied the outage to the rule and now serve as a model of what a good incident record looks like: timestamped, factual and free of guesswork.
Finally, the company agreed a plain communication rule. When something breaks, the opening message to customers covers the facts so far, the work under way and a promised time for the next note. It avoids guessing at causes, which is also the habit that careful analysts apply to their own written conclusions.
How to tell it is working
The team picked three numbers for the next quarter and ignored vanity counts. They track how many outside lists were measured and graded, how many blocks had to be reversed (the target is a falling trend), and how many hunts produced a filed result. After three months, one list had been cancelled for poor accuracy, one had earned limited blocking, and no customer had been turned away by a rule.
If you are planning a trip around the city yourself, our guides on working out the ideal trip length for Chennai and on enjoyable activities for a memorable Chennai visit may help, and they are good examples of the local businesses that depend on smooth online payments.
The wider framework, including a staged maturity ladder and a short guide to choosing between building, buying or advising, is set out on an Ops Atlas page devoted to ThreatOps, which also describes the runbook mentioned here in full.
The moral of the Saturday story is simple. A rule you cannot measure and cannot quickly undo is a liability, however sensible it sounded in a blog post. Measure it, label it, shorten its life and keep the undo button within reach.
