A couple of months ago, my understanding of detection engineering was roughly "write alert, alert goes off." Then the words started showing up. Drift. DaC. CI/CD. Normalization. Baselining. Parsers. Lookback versus schedule. Event time versus ingest time, and the delta between them. p95 and p99. Let the rule soak before you promote it. Precision versus recall. I nodded through every single one and understood maybe a third.

Zach Galifianakis trying to do the math
Me, twenty tabs deep into detection engineering docs. Via GIPHY.

Detection engineering ain't sweet. So I'm learning it the slow way, in public.

This is Part 1 of Becoming a Detection and Response Engineer. That's the goal, and this series is the paper trail. Each post covers one topic I studied, got quizzed on until my explanation held up, and then used on our own SIEM. No textbook theory, just what actually happened.

First up is drift, because it was the first word I got confidently wrong.

Detection drift is a rule that used to work going quiet because the data under it changed. The rule still runs and still shows as active. It just stops matching.

I thought I already knew what drift was

Our detections live as YAML in Git and deploy to the SIEM through a CI/CD pipeline. One of the jobs in that pipeline is literally called drift. Every night it compares the rules running in the SIEM with the rules in Git and pulls any console edits back into the repo.

So when I first heard the phrase "detection drift," I figured I had it covered. I didn't. That nightly job catches config drift, where the text of a rule changes. It never looks at the data. A rule can match Git perfectly and still be dead, and that second problem is what detection drift actually means.

Everything was green

I found out the hard way. Our web proxy vendor sends us a single connection that carries three datasets: web traffic, DNS and firewall. The SIEM's connections page showed it as Active, with about 20 GB ingested in the last day and the last event 38 seconds ago. By every signal on that screen, it was healthy.

It wasn't. The DNS and firewall feeds had stopped four weeks earlier, at the same minute. Every rule reading them had been returning zero ever since, and every one of those rules still showed as active. The connections page reports on the connection as a whole, so as long as one dataset kept flowing, it stayed green. Web traffic was enough to hide the other two. It was red light, green light, except the logs stopped moving and the light stayed green.

Diagram of one vendor connection with three feeds into the SIEM. Web is flowing, DNS and firewall lines are cut, and the connection still shows Active.
One live feed kept the whole connection green, while every DNS and firewall rule sat at zero hits.

The SOC Prime write-up I read afterward has a line for exactly this: "green means broken, not quiet." An active rule with zero hits could be quiet, or it could be dead, and the status alone can't tell you which.

Asking the data instead

What finally showed the real picture was skipping the status page and asking the data directly, one dataset at a time:

#repo=<vendor> | groupBy([#event.dataset], function=[count(), max(@ingesttimestamp)])

Web came back with about 16 million events in the last day. DNS and firewall came back with nothing, and their last arrival times were the same minute four weeks earlier.

I ran one more query to check for events that were arriving but failing to parse, by looking for anything with @error set. There were none. That mattered more than I expected. Parse errors would have pointed at the SIEM's parser. No events at all pointed at the source, which meant the logs weren't breaking on the way in. They had simply stopped being sent.

As for why they stopped, the team's working theory is that we had only ever licensed the web feed, and the other two had been running without a license until the vendor switched them off. That isn't confirmed yet. But if it holds, nothing technical broke at all. No code changed and no parser failed. Drift can come from a licensing decision just as easily as from engineering.

Five ways a rule goes quiet

My first attempt at a definition was "drift is when the ingestion pipeline breaks." That turned out to be too narrow. The SOC Prime piece lists five causes, and in every one of them the rule keeps running and reporting success while matching nothing:

  1. A field keeps its name but changes meaning.
  2. A field stops being sent and shows up empty.
  3. A field is renamed or retyped.
  4. The parser fails, so events never reach the detection.
  5. The source is turned off, retired or migrated.

Only the last two are really about the pipeline. The first three happen while logs flow in perfectly, because the source changed underneath the rule.

The first one is the sneakiest. Here's a made-up example. Say you have an exfiltration rule that fires when bytes_out is over a billion, meaning more than 1 GB is leaving. The vendor ships an update, and the field is still called bytes_out, but now it counts kilobytes. An attacker sends 5 GB out, the field reads 5,000,000, and the rule stays silent. The field exists, the rule runs, nothing errors, and you miss the attack. SOC Prime's real-world version is an identity provider that drops the "challenge" value from a login-outcome field, leaving an MFA-bypass rule unable to tell a bypass from a normal login.

Never worked is a different problem

While cleaning up our rules, I also found a batch that could never have fired. They looked for fields our tenant doesn't send. I called that drift too, and I was wrong. Drift means a rule once worked. These never did.

The analogy that made it click for me was a smoke detector. One was installed without a battery. The other's battery died two years later. You press the test button on both, but the fixes are different. The first needs a better install checklist. The second needs someone checking batteries on a schedule.

Rules work the same way. A rule that never worked needs a gate before deploy that proves the fields exist and the rule can fire. A rule that drifted needs monitoring after deploy. Our CI only checked that the YAML was well-formed. It never ran a query or checked that a field existed, which is how that batch of dead rules got through.

What I'd tell someone starting out

Config drift and detection drift are different problems, and your Git sync job won't save you from the second one. Don't trust a green status. Ask the data, per dataset, and check separately whether events are arriving and whether they're parsing, because each answer points at a different owner. And when a feed is gone for good, mark its rules inactive with a note explaining why, so the next engineer knows they're parked, not forgotten.

Next up, Part 2: detection-as-code and CI/CD, and what a pipeline can and can't prove about a rule.

Readings that helped

All three come from vendors or practitioners with something to sell, so I trusted their explanations more than their numbers.