Evaluating Semantic Representations in Multimodal Word Grounding

Word grounding tasks aim to associate individual words with
corresponding elements in visual scenes, enabling machines to link language with perception for effective human–machine interaction. However,
existing grounding models struggle to generalize to synonyms or unseen
lexical variants, limiting their performance in open-domain scenarios. In
this paper, we present a Bayesian multimodal grounding model that incorporates word embeddings as priors within a probabilistic generative
process to improve robustness under lexical variation. We compare the
effects of static FastText and contextual BERT embeddings on grounding accuracy by conditioning word–visual associations on their semantic
representations. Experiments use CLEVR-generated 3D scenes paired
with structured compositional descriptions to test the grounding of object categories, colors, and spatial relations across lexical shifts. Results
show that contextual embeddings such as BERT consistently outperform static embeddings like FastText in overall grounding accuracy and
in resolving spatial relations. We demonstrate that integrating structured
probabilistic inference with rich semantic embeddings offers a principled
and scalable solution for robust, interpretable word grounding.