Fraunhofer Institute for Production Systems and Design Technology
Going Once, Going Twice, Going Three Times – Found!
With the help of AI-based methods, current and historical images of art objects can be compared with each other. Pictured: Bodhisattva Mahasthamaprapta, (Dashizhi) 13th century.
(left) Public Domain, The Metropolitan Museum of Art; (right) German Sales, University Library Heidelberg
If one wants to research the art market of the first half of the 20th century, there is no getting around auction catalogs. They are an indispensable research tool, especially for provenance researchers: They document where a work of art came from, who it once belonged to, and where and when it changed hands, often over several centuries. »German Sales« is a database that has been growing for around ten years. It digitally records and centrally collects information on the historical art market in Germany, Austria, and Switzerland. It was launched as a collaborative project between Heidelberg University Library, the Kunstbibliothek of the Staatliche Museen zu Berlin, and the Getty Research Institute in Los Angeles. Around 11,000 auction catalogs along with gallery, storage, and antiquarian catalogs are now available online and via open access.
On behalf of the Rhineland Regional Council (Landschaftsverband Rheinland, LVR), researchers at Fraunhofer IPK have investigated the extent to which modern computer vision methods can help with researching works of art in such digital auction catalogs. Guido Kohlenbach, head of the Department of Culture at LVR, explains: »Automated identification of cultural assets or art objects in online cata-logs would make it easier to research the whereabouts of objects at specific times as well as stored data in the publications and thus obtain valuable information about owners.«
As part of a feasibility study, the Fraunhofer researchers developed AI-based image search methods that can reliably match images or art objects. The methods were validated using pairs of images, each representing matches between auction catalogs and other digital image collections. The researchers were confronted with the challenge that images of one and the same art object can vary greatly depending on the date and technique of image acquisition, image quality, perspective, or even the type of object itself (2D or 3D).
The preparation and quality assurance of the image data therefore played a central role in the project, starting with the extraction of images from over 11,000 auction catalogs in the »German Sales« database. »Due to the large number of images, purely manual processing would be very time-consuming and expensive,« explains Raúl Vicente-García, project lead at Fraunhofer IPK. »That‘s why we are using state-of-the-art computer vision methods that have been proven successful in the automated analysis and segmentation of documents, and adapting them with the help of AI to the special features of historical images.«
Automated matching process of two images of the same object in detail. Pictured: Lidded vase, Cornelius Funcke, around 1710
From a database of almost 600,000 objects, the objects matching the query are output automatically.
The results of the feasibility study are promising: Around 560,000 images of paintings and sculptures, items of art, and even everyday objects such as furniture or cutlery can be searched within just a few seconds using the Fraunhofer methods. During this process, AI-based image features are collected for each individual item so as to achieve a high recognition rate despite the high variability in the type and quality of the images. The image features which are automatically analyzed on different scales range from contours and textures to item details such as the eye of a person depicted in the image. Thanks to the adaptability of the AI-based methods, it is also possible to match current and historical images, which exhibit a much lower image quality.
With their feasibility study, the Fraunhofer experts prove that AI-based image search methods are suitable for provenance research. In the next step, which involves the development of a prototype software solution, the scientists are currently looking for museums, foundations, and libraries that are interested in researching and documenting the provenance and ownership history of their cultural assets more efficiently.
Results of the automatic segmentation of a page from an auction catalog