The MSK museum in Ghent is filled with paintings. For a customized automated guided tour, the visitor's location should be known. This is implemented using camera footage in this project. Painting recognition provides information about the camera position, all with only a delay of 33ms.

Team members

Electrical Engineering Technology (Automation)

Glenn De Loose 

Calvin Roets

Information Engineering Technology

Bart van de Meerendonk

Freya Van Speybroeck

Painting detection and extraction

A contour analysis is performed for every 10 frames. The process begins by converting the frame to grayscale, followed by applying Canny edge detection with Otsu's threshold. This step highlights only the distinct lines. Subsequently, morphological operations are applied to connect these lines. By setting limits on the maximum and minimum contour size, the majority of non-painting rectangles are eliminated. Finally, a mask is employed to isolate only the painting.

Painting matching

Transfer learning is employed to leverage ResNet-18. This not only saves time but can also enhances performance compared to creating a network from scratch. The utilized network is pre-trained on millions of images from ImageNet for a classification task involving 1000 classes. Since only one labeled painting was provided, data augmentation was necessary for training the last fully connected layer. By applying a confidence threshold of 0.6, the performance achieved over 96% accuracy.

Painting localization

For locating the paintings, a connectivity matrix prevented impossible movements through the building. Additionally, a buffer of samples was implemented. This introduced a slight delay but further enhanced the performance.