Correctness
Does the remediation resolve the misconfiguration as defined by the policy?
PublishedVersion v0.1·Cloud Security Posture Management
CSPM Remediation Benchmark
Measures whether AI models can recommend safe, policy-compliant remediations for cloud misconfigurations across AWS, GCP, and Azure — at code (IaC) and runtime layers.
Scope
What this benchmark measures
Accuracy of misconfiguration root-cause identification
Quality of proposed Terraform / CloudFormation / Pulumi patches
Compliance alignment with CIS, NIST, PCI, SOC 2
Awareness of blast radius across dependent resources
Policy-respecting suggestions (no over-broad permissions)
Evaluation
Evaluation dimensions
Each model response is judged on these dimensions by an LLM-as-Judge calibrated against expert annotations.
Least Privilege
Does the fix avoid granting unnecessary permissions or exposing surface area?
Compliance Mapping
Does the model cite the relevant CIS/NIST/PCI control accurately?
Multi-Cloud Skill
Can the model generate parallel fixes across AWS, GCP, and Azure variants?
Blast Radius
Does the model reason about dependent resources before recommending changes?
Example
Sample benchmark case
An illustrative case to show how the model is presented with a problem and how its response is scored.
Finding
S3 bucket "prod-customer-uploads" has public-read ACL and no encryption at rest.
Prompt summary
Generate a Terraform patch that removes public access, enables SSE-KMS, and preserves existing IAM grants used by lambda-thumbnailer.
Ideal answer
Remove ACL grant, add server_side_encryption_configuration with kms_master_key_id, leave bucket policy for the lambda role untouched. Cite CIS AWS 2.1.1 and 2.1.5.
Scoring note
Judges flag any policy that broadens access beyond the minimum needed to satisfy the lambda role.
Roadmap
Current version: v0.1
All Remediation Labs benchmarks evolve through three versions, increasing in difficulty and realism.
Current
v0.1Finding-only
Models are given the raw security finding and asked to recommend a remediation. No environment context provided.
v0.2Context-aware
Models receive the finding plus relevant system, code, and policy context. Evaluated on contextual reasoning quality.
v0.3Executable remediation
Models must produce a remediation that can be safely executed end-to-end — code patches, IaC changes, runbook steps — with verification.
Want the full CSPM Remediation report?
Get model leaderboards, scoring methodology, dataset details, and roadmap access — or discuss generating proprietary remediation datasets for your environment.
Or email us directly at info@remediationlabs.com