Reachability
Does the model correctly distinguish reachable from unreachable vulnerable code paths?
PlannedVersion v0.1·Software Composition Analysis
SCA Remediation Benchmark
Measures AI models on remediating vulnerable open-source dependencies — choosing safe upgrade paths, evaluating breaking changes, and validating reachability.
Scope
What this benchmark measures
Identification of true reachable vulnerabilities vs. noise
Selection of safest viable upgrade version
Awareness of transitive dependency impact
Detection of breaking API changes in target versions
License compatibility consideration
Evaluation
Evaluation dimensions
Each model response is judged on these dimensions by an LLM-as-Judge calibrated against expert annotations.
Upgrade Safety
Does the suggested version avoid known regressions or breaking changes?
Transitive Impact
Does the model account for downstream dependencies affected by the upgrade?
License Awareness
Does the model verify license compatibility of the upgrade target?
Patch Quality
Is the resulting lockfile and manifest change minimal and reviewable?
Example
Sample benchmark case
An illustrative case to show how the model is presented with a problem and how its response is scored.
Finding
log4j-core 2.14.1 transitive dependency with CVE-2021-44228; called via spring-boot-starter-logging.
Prompt summary
Recommend an upgrade path or compensating control. Show the manifest changes and explain reachability.
Ideal answer
Upgrade log4j-core to 2.17.1 via dependencyManagement override. Confirm spring-boot-starter-logging compatibility. Note that JNDI lookup is reachable based on configured appenders.
Scoring note
Models that recommend major-version jumps without justifying breaking-change handling are penalized.
Roadmap
Current version: v0.1
All Remediation Labs benchmarks evolve through three versions, increasing in difficulty and realism.
Current
v0.1Finding-only
Models are given the raw security finding and asked to recommend a remediation. No environment context provided.
v0.2Context-aware
Models receive the finding plus relevant system, code, and policy context. Evaluated on contextual reasoning quality.
v0.3Executable remediation
Models must produce a remediation that can be safely executed end-to-end — code patches, IaC changes, runbook steps — with verification.
Want the full SCA Remediation report?
Get model leaderboards, scoring methodology, dataset details, and roadmap access — or discuss generating proprietary remediation datasets for your environment.
Or email us directly at info@remediationlabs.com