Results of Year 4+
Video archives, both public and private ones, continue to expand at a dazzling pace. That might seem a good thing at first – as more content becomes available on a growing number of topics – but it also comes with its challenges. After all, finding the exact content you are looking for becomes more difficult, and takes considerably longer. Current search engines rely on human annotations, known as ‘metadata’, to retrieve relevant material. AXES, a tool resulting from a recently-ended European research project, changes that. AXES’ video search results are an order of magnitude better than anything that is on the market today. It allows users to mine video archives rapidly, intuitively and with minimal effort. And it comes with a public test version that is freely available.
The amount of video material that has been – and is being – produced, is enormous. The BBC Archives, for instance, date back to 1890 and include more than 1 million hours of playable material. A wealth of information, in other words, but these libraries’ sheer size makes accessing its contents a real challenge – especially since existing tools require quite some human interaction to get to satisfactory results (both before and during the actual search).
AXES: a game-changer in mining video archives
Existing search engines require upfront (manual) tagging to make the video content searchable (you can only get out what you put in). An army of archivists spends big part of their daily work time on adding metadata to old or new video footage – often using a fixed thesaurus. And even as search results are shown, more manual effort is required from the enduser, fast-forwarding the selected videos in order to find the specific fragment he/she is interested in. Finally, current tools often require the user to upload the content to public databases; this is often not an option due to copyrights, privacy concerns, etc. But now there is AXES, a real game-changer in interacting with video archives. The AXES consortium built an intelligent tool that allows users to search video archives without having to tag everything in advance. Bringing together the knowledge from European experts in a wide range of technology domains, they created a system that automatically recognizes buildings, places, faces, objects and events within the video files. And thanks to its integrated speech recognition technology, AXES also searches on spoken words within the video fragments. Another unique feature of AXES is that users end up at the exact video fragment that matches their query; no more fast-forwarding is required. And also the upload issue is addressed: the AXES technology can simply be downloaded and safely applied to users’ private video collections.
Search, explore, connect
Being able to access the vast amount of valuable content in our archives appropriately is key in many use scenarios encountered in the context of archives, ranging from re-use to research and entertainment. Allowing users to find what they are (not) looking for and enabling the exploration of both content and context – via links within the archive itself, or even links that cross archive boundaries—requires archival content to be labeled beyond descriptive metadata on the document level. By enriching traditional metadata with time-coded jump-in points, or ‘link anchors‘ as we call them from a connectivity perspective, the richness of the (long tail of) archives can be fully exploited. AXES developed a range of advanced access technology components for generating these labels automatically, based on audio or visual tools such as speech recognition and visual search. These also provide links based on which users can browse and explore the archive content. Moreover, AXES has a strong focus on deploying the automatically generated labels in practical application scenarios, addressing audiovisual search optimization, recommendation and video hyperlinking in prototype systems fed with representative data sets.
A self-learning system improving over time
What makes AXES unique is the system’s ability to learn. This means that as a user you’re not restricted to a predefined thesaurus, but can use any search term. If users insert a query such as ‘horn rimmed glasses’, and the system does not have a model for that concept yet, the tool will consult the Internet to learn what ‘horn rimmed glasses’ are – and use that knowledge to search the video archive.
This provides the user with an unprecedented level of flexibility. For instance, queries can be posed in any language. Moreover, this whole procedure is efficient and scalable: models can be trained and applied to the whole archive in just a matter of seconds.
A dedicated system for media professionals, journalists, researchers and consumers
In co-development with real end users, AXES produced archival search systems in three flavors: AXES PRO for broadcast professionals, AXES RESEARCH for journalists and researchers, and AXES HOME for public users. The tool comes with a dedicated user interface that helps each of them get the most out of their video archives. Of course, the results of the project in terms of technology and best-practices are available for take up by industry.
AXES PRO for media professionals
Based on requirements of professional users at or linked to our content partners (BBC, Deutsche Welle and Netherlands Institute for Sound and Vision), AXES built a system that specifically focused on the re-use of archival material: AXES-Pro. One of the main topics evaluated with this prototype was how speech search and visual search could help professionals to find clips for new productions. Next to speed the clarity of the user interface and ultimately, the relevance of results produced by such a system received special attention. The general conclusion was that innovative search approaches increase the amount of search options and improve access to the long tail of archives. Given the variance in performance levels of automatic analysis tools, it is important though that in a professional environment new functionalities are provided with care and with a strong focus on expectation management.
AXES RESEARCH for journalists and researchers
Researchers and journalists are specific user groups of archive material. They use the archive to investigate a topic or hypothesis. Evidently, the requirements of this user group significantly differ from the broadcast professionals. Obvious examples are the relatively stronger focus of this group on recall (“Didn’t I miss something crucial?”), the time they are willing to spend on diving into the archive, and the possibility of exploring links within and across collections. We evaluated AXES-Research with journalists and researchers, the latter via our connections with the Digital Humanities community. For both user groups, having advanced audiovisual search tools available was a new and welcome experience. One important outcome of our studies was that the deployment of advanced search tools should be accompanied with background information on how the technology works. Interestingly, playing with the AXES tools stimulated the discussion on exploring content based on ‘narratives’ (e.g., storytelling). An online demo of the AXES RESEARCH system is available at https://axesdemo.ch.bbc.co.uk/axes/research/open
AXES HOME for public users
The ultimate user group AXES addresses, is the general public, obviously an important user group for archives. The access models that we implemented and the tablet-based design of the AXES HOME system focuses on entertainment and “edutainment”. The interface is kept as simple as possible but at the same time, invites users to explore content using the advanced features made possible by AXES’ audio and visual analysis tools. For example, by providing a browsing option on the basis of visual categories that allows a user to browse through all elephants found in the collection.
An example of a novel access model is based upon the observation that archives should bring content to where the user is, instead of the traditional model that requires a user to come to the content (e.g., website of the archive).
AXES implements this by providing users with relevant content from the archives dynamically based on the contents of external sources such as newsfeeds, blogs and websites.
Open AXES: AXES technology at your fingertips
In an effort to consolidate the tools we have developed, making sure they can be used and exploited further, we have made available Open-AXES, a version of our system with a selection of AXES components that can be freely downloaded for non-commercial use. All you need is a powerful desktop, running Linux and with the right set of (generic) libraries preinstalled.
You download Open-AXES, install it and off you go: you can now index your own videos (which takes roughly 3 minutes per minute of video uploaded), then search or browse your collection, using the AXES RESEARCH interface. Also see: Use Open AXES