NobleBlocks

Research indexThesis

RelCheck

Patil

1 Jan 2026

DOI10.31979/etd.7wq3-5nzv

Paywalled

The full text is at the publisher

You can read the abstract here, or open it at the publisher.

Read at the publisher

Abstract

Multimodal large language models (MLLMs), such as LLaVA, may be able to create rich descriptions of an image; however they suffer from the hallucination problem, in which their output does not reflect what was visually presented. The main goal of this project is to address limitations of Woodpecker, a state-of-the-art method of correcting hallucinations in multimodal language models through post-hoc methods. Woodpecker uses objects and attributes in relation to an image to ground it. We propose RelCheck, a training-free technique that adds a relational component to the object-level visual grounding. To do so, we built a multi-layered visual knowledge base using object-level detection with GroundingDINO, learned relational triples with RelTR (a scene graph generation model), and deterministic spatial relations derived from bounding box coordinates. This evidence then helps guide a LLM in its reasoning to correct the hallucination. We test on the two main hallucination benchmarks, POPE, and the hallucination subtasks of MME, with our target model being LLaVA v1 13B. We compare our system against the raw MLLM and Woodpecker-style pipeline. Our results show promising improvement across the benchmarks, as well as a breakdown across different types of queries. These demonstrate the usefulness of relational knowledge in correcting hallucinations in MLLMs.

Discussion

More

Suggest a correction or report