Evaluating and Mitigating Relationship Hallucinations in Large Vision-Language Models
- Published
- Record updated
Summary
R-Bench is a new benchmark for measuring hallucinations about relationships between objects in Large Vision-Language Models, using image-level questions on whether a relationship exists and instance-level questions on local visual comprehension. The authors trace these hallucinations to relationship-relationship, subject-relationship and relationship-object co-occurrences, worsened by long-tail distributions in visual datasets. They report that region-level image-text alignment reduces them and propose a baseline, Region-Aware Alignment Mitigation (RA$^{2}$2M), that directs model attention to relevant regions.
Mitigation
Region-level image-text alignment helps mitigate relationship hallucinations; the authors propose RA$^{2}$2M (Region-Aware Alignment Mitigation) as a new baseline that enhances model attention to relevant regions.