Detection Engineering Fundamentals

Detection engineering represents the systematic practice of creating, testing, and maintaining the rules that identify security threats. While security tools come with built-in detections, organizations need custom rules tailored to their specific environment, threats, and risk tolerance.

The Detection Lifecycle

Effective detection requires more than writing alert rules. A complete lifecycle includes identifying what to detect, creating the detection logic, testing for effectiveness and false positives, deploying to production, maintaining rule performance, and eventually retiring rules that no longer provide value.

Detection ideas come from many sources. Threat intelligence reveals adversary techniques worth detecting. Incident investigations highlight gaps where attacks succeeded without detection. Framework mappings like MITRE ATT&CK provide systematic coverage goals.

Before writing detection logic, understand the data available. Detection rules can only work with observable evidence. If your logs don't capture the relevant activity, even perfect logic won't generate alerts.

Writing Detection Logic

Good detection rules balance sensitivity with specificity. Highly sensitive rules catch more malicious activity but generate more false positives. Highly specific rules produce fewer false alerts but might miss variations of the attack they target.

Start with broad detection and refine based on results. If a rule generates too many false positives, add conditions that exclude benign activity. If it misses known attacks, examine why and adjust the logic. This iterative refinement continues throughout a rule's lifetime.

Context often distinguishes malicious from benign activity. Running PowerShell is normal; running it with encoded commands from a Word document is suspicious. Effective detection often combines multiple conditions rather than matching single indicators.

Temporal patterns add another dimension. A single failed login is routine; hundreds in a minute suggest attack. Aggregating and thresholding transforms normal activity into detectable anomalies.

Testing Detections

Detection rules require testing before deployment and ongoing validation afterward. Without testing, you don't know if rules work as intended or generate acceptable alert volumes.

Atomic Red Team and similar frameworks provide test cases for common attack techniques. Execute these tests in controlled environments and verify your detections fire appropriately. Document which tests each rule should detect.

False positive rates matter as much as detection rates. A rule that catches every attack but alerts on normal activity too frequently becomes useless—analysts learn to ignore it or eventually disable it. Test rules against representative samples of normal activity to estimate false positive rates.

Maintaining Detection Systems

Detection rules require ongoing maintenance as environments and threats evolve. Applications update, changing normal behavior. Adversaries modify techniques to evade known detections. Rules that worked last year might produce only noise today.

Regular review assesses whether rules still provide value. Low-frequency rules might be working correctly or might be missing everything. High-frequency rules need constant tuning to maintain signal quality.

Document why rules exist and what they detect. Future analysts (including future you) need this context to make informed maintenance decisions.