Validation and research

Compliance software is only useful if you can trust its findings. BIM Guard's evaluation harnesses live in an independent public repository, and results are published together with what they cannot show.

What the evaluation currently supports

• 22 of 22 architecture-engine benchmark cases classified correctly (13 true positives, 9 true negatives, no false positives or negatives). The cases are few and procedurally generated, so the 95% Wilson confidence interval for accuracy, 85.1% to 100%, matters more than the point estimate.
• 60 of 60 NLP annotation test cases passing, fully reproducible.

What is not claimed

Some harnesses in the repository simulate their inputs. Their numbers are labelled as such and are not evidence about BIM Guard. Every headline number in the repository is listed in a claims ledger with its producing script, artifact and verification status.

Read the evidence

• BIM-Guard Evaluation site - overview and legend
• Results - the architecture-engine confusion matrix
• Claims ledger - every headline number and its evidence
• Limitations - methodological gaps, stated plainly
• Reproduce - commands to re-run each harness
• bim-guard-evaluation on GitHub - source and citation file
• bim-guard on GitHub - the platform itself

Related

Automated IFC compliance checking · All features · Documentation