Correctness
Does the proposed fix resolve the underlying issue without breaking functionality?
PublishedVersion v0.1·Static Application Security Testing
SAST Remediation Benchmark
Measures whether AI models can recommend, generate, and validate fixes for vulnerabilities found by static analysis tools across major programming languages and SAST engines.
Scope
What this benchmark measures
Accuracy of root-cause identification for SAST findings
Quality of suggested code fix or patch
Whether the fix preserves application behavior
Whether the fix avoids introducing new vulnerabilities
Coverage across OWASP Top 10 and CWE Top 25 categories
Evaluation
Evaluation dimensions
Each model response is judged on these dimensions by an LLM-as-Judge calibrated against expert annotations.
Safety
Does the fix avoid introducing new CVEs, regressions, or side-channel risks?
Context-Awareness
Does the model reason about the calling context, framework idioms, and data flow?
Executability
Is the fix directly applicable as a PR or patch without manual rewriting?
Explanation Quality
Does the model explain the root cause, blast radius, and trade-offs clearly?
Example
Sample benchmark case
An illustrative case to show how the model is presented with a problem and how its response is scored.
Finding
SQL injection in user.controller.ts line 42 — raw concatenation in WHERE clause.
Prompt summary
Given the file, the finding, and surrounding call sites, generate a fix that resolves the SQLi while preserving the API contract.
Ideal answer
Replace string concatenation with a parameterized prepared statement using the existing ORM. Add input validation. Justify the change in the PR description.
Scoring note
LLM-as-Judge rubric weighs Correctness (40%), Safety (25%), Context-Awareness (15%), Executability (15%), Explanation (5%).
Roadmap
Current version: v0.1
All Remediation Labs benchmarks evolve through three versions, increasing in difficulty and realism.
Current
v0.1Finding-only
Models are given the raw security finding and asked to recommend a remediation. No environment context provided.
v0.2Context-aware
Models receive the finding plus relevant system, code, and policy context. Evaluated on contextual reasoning quality.
v0.3Executable remediation
Models must produce a remediation that can be safely executed end-to-end — code patches, IaC changes, runbook steps — with verification.
Want the full SAST Remediation report?
Get model leaderboards, scoring methodology, dataset details, and roadmap access — or discuss generating proprietary remediation datasets for your environment.
Or email us directly at info@remediationlabs.com