Advances in Representation: Interactive Heritage Site Mobile Application on Artworks

In this work, we are introducing early technological prototypes of an interactive mobile application aimed at heritage sites and museums. This mobile application will comprise three main components as follows: (1) Image Captioning, (2) Indoor Navigation, and (3) Augmented Reality. To address the Image Captioning component, we aim to build a mobile application for the guidance of heritage site visitors with a caption with relevant information about each artwork. This application can be helpful for visually impaired visitors, and it can provide some extra information about the museum’s objects. We will build this project based on computer vision and AI techniques. The main methodology of this work is according to the Image Captioning algorithms which by giving any input image, the AI model can provide a relevant caption to describe the input image through the textual and voice outputs. For the Indoor Heritage Site Navigation, we propose a mobile application that will be connected to an indoor positioning system which will be able to locate each piece of artwork of the museum and indicate directions to the museum visitors. Furthermore, this application would be beneficial for visually impaired people to follow properly all the artwork of the museum and find their locations. The Augmented Reality technology will provide the interactive aspect of an application for the heritage site visitors to obtain extra information about the artworks. The goal is to show animated visualizations of the artwork and/or to show detailed information about each part of the artwork by pointing out each area and adding some more textual or visual data around that. The proposed application will be a holistic interactive application which will be comprised of all the three above-mentioned packages in one. The key characteristic of this work stems from building a lightweight mobile application to include three platforms together.