Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali

Summary

BILAL-BENCH is a 240-question, expert-verified benchmark for open-ended Islamic-source fabrication in Bengali, covering Qur'anic attribution, hadith, fiqh, contemporary issues and false-premise prompts. Evaluating seven open-weight models from 1B to 120B on 1,680 double-annotated responses, six of seven confidently fabricated on 79.3-95.3% of source-production questions, with no monotonic improvement from scale. Hadith provenance was the weakest category, with only two of 315 responses correct and correctly sourced.