Detection Triage
Does the model correctly distinguish true positives from noise under time pressure?
PlannedVersion v0.1·Runtime Threat & Incident Remediation
Runtime Remediation Benchmark
Measures AI models on remediating active runtime threats — live exploit attempts, anomalous behavior, workload quarantine decisions, and compensating-control authoring.
Scope
What this benchmark measures
Correct identification of active vs. dormant threats
Quality of containment recommendations
Appropriate use of compensating controls
Decision-making under partial information
Coordination across detection signals (EDR, network, audit)
Evaluation
Evaluation dimensions
Each model response is judged on these dimensions by an LLM-as-Judge calibrated against expert annotations.
Containment
Are containment actions proportionate and reversible?
Compensating Controls
Are temporary controls correctly authored when a permanent fix is not yet possible?
Runbook Quality
Is the recommended response sequence clear, ordered, and operator-ready?
Cross-Signal Correlation
Does the model combine EDR, network, and audit signals to scope the incident?
Example
Sample benchmark case
An illustrative case to show how the model is presented with a problem and how its response is scored.
Finding
Suspicious outbound connection from pod payments-api to 185.x.x.x on port 4444; reverse-shell pattern.
Prompt summary
Generate a containment plan. Decide whether to kill the pod, isolate it, or apply a NetworkPolicy.
Ideal answer
Quarantine via NetworkPolicy denying egress except to in-cluster services. Snapshot memory before kill. Trigger incident workflow. Flag CI image for SBOM re-scan.
Scoring note
Suggestions that immediately kill the pod (destroying evidence) are penalized.
Roadmap
Current version: v0.1
All Remediation Labs benchmarks evolve through three versions, increasing in difficulty and realism.
Current
v0.1Finding-only
Models are given the raw security finding and asked to recommend a remediation. No environment context provided.
v0.2Context-aware
Models receive the finding plus relevant system, code, and policy context. Evaluated on contextual reasoning quality.
v0.3Executable remediation
Models must produce a remediation that can be safely executed end-to-end — code patches, IaC changes, runbook steps — with verification.
Want the full Runtime Remediation report?
Get model leaderboards, scoring methodology, dataset details, and roadmap access — or discuss generating proprietary remediation datasets for your environment.
Or email us directly at info@remediationlabs.com