Alignment Metrics

Framework Tracking

Scroll down for details behind this multi-tier benchmark

LLM Alignment Benchmarking Dashboard (V2)

Scores based on accumulated research data (Stanford HELM, FMTI). Click “Run Live Sanity Check” to verify model availability.

Risk Management

Transparency

Fairness

Safety

Robustness

Accountability

Detailed Metrics & Status

LLM	Live Status	Risk (%)	Transparency (%)	Fairness (%)	Safety (%)	Robustness (%)	Accountability (%)

Methodology & Data Rigor

This dashboard is designed to provide a scientifically grounded and reliable view of Large Language Model (LLM) alignment. Rather than relying on transient and often unreliable “live” scoring via simple prompts, we aggregate data from established, gold-standard research benchmarks.

Data Sources

Transparency: Sourced from the Foundation Model Transparency Index (FMTI) by Stanford CRFM. This index evaluates companies on indicators regarding data, compute, and policy transparency.
Fairness & Robustness: Sourced from the Holistic Evaluation of Language Models (HELM) project. HELM conducts massive-scale testing across thousands of prompts to determine bias, toxicity, and adversarial robustness.
Safety: Aggregated from HELM’s safety specific metrics and cross-referenced with TrustLLM benchmarks.

Live Verification

The “Live Sanity Check” feature on this dashboard performs a real-time connectivity test to ensure the models are operational. While the scores are static (updated with research releases), the status indicators confirm that the models are online and accessible via their respective APIs.

Governance Framework Metrics

This table maps similar alignment metrics across ISO/IEC 42001:2023 (AI Management System Standard), NIST AI Risk Management Framework (AI RMF 1.0), and the EU AI Act (Regulation (EU) 2024/1689). These metrics serve as gold standard AI governance alignment benchmarks, because each governing body has been trusted for decades.

Alignment Dimension	ISO/IEC 42001:2023	NIST AI RMF 1.0	EU AI Act (Regulation 2024/1689)	Mapped Metric for ASI Dashboard
Risk Management	Risk Assessment Compliance (Annex A.8): % completion of structured risk assessments across AI lifecycle; frequency of risk reviews (e.g., quarterly).	Map and Measure Functions: % of risks identified and documented (Map); risk likelihood and impact severity scores (Measure).	Risk Management System (Article 9): Adoption rate of risk mitigation plans for high-risk systems; frequency of risk updates (e.g., monthly).	Risk Assessment Coverage: % of AI lifecycle stages with documented risk assessments; average risk update frequency (days).
Transparency	Documentation and Transparency (Annex A.7): % of AI systems with complete documentation (e.g., model cards, data provenance); audit frequency.	Transparency Characteristics: % of systems with explanation coverage; stakeholder engagement frequency (Govern).	Transparency Obligations (Article 13): % of high-risk systems with compliant transparency reports; user opt-out rates.	Transparency Score: % of systems with complete, compliant documentation and explanations; average audit frequency (per year).
Fairness	Bias Mitigation Processes (Annex A.9): Bias detection rates (% of identified biases); mitigation action effectiveness (% reduction in bias scores).	Fairness Characteristics: Disparate impact ratios across demographic groups; bias audit frequency (Measure).	Data Governance (Article 10): Bias detection frequency in training datasets; data quality scores (e.g., ISO/IEC 5259-4 compliance).	Fairness Index: % reduction in bias scores post-mitigation; bias audit frequency (per month).
Safety	Security Threat Modeling (Annex A.10): Threat identification rate (e.g., STRIDE model); mitigation coverage (% of threats addressed).	Safety Characteristics: Violation rates for safety policies; incident response time (Manage).	Human Oversight (Article 14): Oversight intervention frequency; error correction rates for high-risk systems.	Safety Compliance Rate: % of identified safety threats mitigated; average incident response time (hours).
Robustness	System Reliability (Annex A.11): Accuracy under stress tests; robustness against adversarial inputs (% accuracy drop).	Reliability Characteristics: Consistency of outputs under varying conditions; robustness scores (Measure).	Robustness and Accuracy (Article 15): Accuracy under adversarial conditions; robustness scores for high-risk systems.	Robustness Score: % accuracy maintained under adversarial tests; consistency index across conditions.
Accountability	Audit and Accountability (Annex A.6): % of AI systems with audit trails; frequency of accountability reviews.	Accountability Characteristics: Audit trail completeness; stakeholder accountability metrics (Govern).	Record-Keeping (Article 12): % of high-risk systems with automated logs; compliance audit frequency.	Accountability Index: % of systems with complete audit trails; average compliance audit frequency (per year).