Image Slimming
Does the fix reduce attack surface (smaller base, removed tools)?
PlannedVersion v0.1·Container & Binary Artifact Remediation
Binary & Image Remediation Benchmark
Measures AI models on remediating vulnerabilities in container images and binary artifacts — including base image selection, layer-level patches, and SBOM-driven fixes.
Scope
What this benchmark measures
Selection of minimal-CVE base images
Quality of Dockerfile changes to reduce surface area
SBOM-aware identification of vulnerable components
Awareness of multi-stage build optimization opportunities
Signature and provenance preservation across rebuilds
Evaluation
Evaluation dimensions
Each model response is judged on these dimensions by an LLM-as-Judge calibrated against expert annotations.
CVE Resolution
Does the proposed image actually resolve the flagged CVEs?
Build Reproducibility
Are the Dockerfile changes deterministic and cache-friendly?
SBOM Alignment
Does the model correctly identify which SBOM component carries the CVE?
Supply Chain
Is signature/provenance preserved in the proposed remediation?
Example
Sample benchmark case
An illustrative case to show how the model is presented with a problem and how its response is scored.
Finding
Container image ghcr.io/acme/api:1.4 ships glibc 2.31 with CVE-2024-2961.
Prompt summary
Propose a Dockerfile change that fixes the CVE while preserving runtime behavior. Justify base image choice.
Ideal answer
Switch base from debian:bullseye-slim to debian:bookworm-slim (glibc 2.36). Validate runtime libs. Rebuild with cosign signing in CI.
Scoring note
Models that suggest "alpine" without verifying glibc-dependent runtime behavior are penalized.
Roadmap
Current version: v0.1
All Remediation Labs benchmarks evolve through three versions, increasing in difficulty and realism.
Current
v0.1Finding-only
Models are given the raw security finding and asked to recommend a remediation. No environment context provided.
v0.2Context-aware
Models receive the finding plus relevant system, code, and policy context. Evaluated on contextual reasoning quality.
v0.3Executable remediation
Models must produce a remediation that can be safely executed end-to-end — code patches, IaC changes, runbook steps — with verification.
Want the full Binary & Image Remediation report?
Get model leaderboards, scoring methodology, dataset details, and roadmap access — or discuss generating proprietary remediation datasets for your environment.
Or email us directly at info@remediationlabs.com