Presentation
Veriscore: Measuring LLM Verification Quality with Mutation- and Soundness-Based Metrics
DescriptionLarge language models are increasingly used to generate hardware verification artifacts from natural language prompts, yet evaluation is often limited to syntactic or build success. We present VeriScore, an open evaluation framework that measures end to end verification quality by executing generated verification artifacts on correct and systematically mutated RTL designs. VeriScore provides standardized prompts with explicit assumptions, golden RTL implementations, and controlled bug injected mutants representing common hardware error patterns. Generated artifacts such as assertions, checkers, testbenches, and harness code are built and executed under fixed resource budgets and scored based on their ability to detect injected bugs without producing false failures. In addition, VeriScore can optionally benchmark models against a curated suite of known correct verification artifacts derived from design specifications, enabling direct comparison to reference assertions and checkers. The resulting report decomposes performance into buildability, soundness, sensitivity, and non triviality and aggregates these into a transparent prompt to verification accuracy score. VeriScore enables reproducible tool agnostic comparison of verification pipelines and establishes a benchmark for evaluating verification effectiveness beyond compilation.
Event Type
Engineering Poster
TimeTuesday, July 285:00pm - 6:00pm PDT
LocationDAC Pavilion, Exhibit Floor