AI-PRODUCER: AI-based Video Clipping and Summarization of Sport Events

The goal of the AI-producer project has been to develop an automated AI-based video producer generating clipped sport events and summaries. Such a service will reduce costs and additionally enable production services on both existing archives and in leagues not having the resources for a production.The project has developed some simple models to find good clipping points for events like goals and bookings in soccer, and can generate videos from compilations of events prioritised by the event annotators.

Automatic clipping: In this project, we have automated the process of highlight generation using logo transition detection and scene boundary detection. We have experiment with various approaches on different datasets. These models are evaluated both quantitatively and qualitatively, and the results show that we can detect logo and scene transitions with high accuracy and generate highlight clips that are highly acceptable for viewers. We conclude that there is considerable potential in automating the overall soccer video clipping process. There is also ongoing work to further refine the developed approaches. Automatic thumbnail selection: We have developed an automatic thumbnail selection system for soccer videos which uses machine learning to deliver representative thumbnails with high relevance to video content and high visual quality. Our proposed system combines logo detection, close-up shot detection, face detection, and image quality analysis into a modular and customizable pipeline, and a subjective evaluation framework for the evaluation of results. We have evaluated our proposed pipeline quantitatively using various soccer datasets, in terms of complexity, runtime, and adherence to a pre-defined rule-set, as well as qualitatively through a user study, in terms of the perception of output thumbnails by end-users. Our results show that an automatic end-to-end system for the selection of thumbnails based on contextual relevance and visual quality can yield attractive highlight clips. Summary generation: When the events are annotated and clipped, the next step is to compile selected events into a highlight summary. Based on priorities set by the annotators, we can compile the AI-clipped events into a summary by concatenating the selected events into a video, adding transitions and overlay graphics. This is today already in use in Sweden. However, to automatically generate various summaries, there is still a lot of R&D to be performed.

Video productions are part of an enormous industry where soccer alone has a market share of about 45% of the $500 billion global sports industry. There is a huge interest in sport event and summary consumption for both broadcasted and streamed on the web. A lot of resources are put into producing the content in an efficient and high-quality manner. However, generating such summaries and events requires expensive equipment, includes a lot of tedious expensive manual labor, and the operation is often performed redundantly by different actors for different purposes. Forzasys has already developed an improved annotation and production system optimising the manual procedures and reducing the need for expensive equipment. In this project, we are targeting an AI-based solution assisting the current production personnel in their daily work and extending the current system, i.e., allowing a production of highlights at a higher pace and enriched with additional metadata. Building on our experience in sport analysis, sport video delivery systems, and use of AI for video and image analysis in both sport and medicine, the idea is to use AI to analyse the video from the area around a detected event, using for example scene change detection, logo identifications, audio analysis, change of cameras, game transitions and replay recognition, i.e., all information to be used to enable an accurate automated AI-based clipping operation. Furthermore, other enhancement methods will also be investigated, like adding more detailed text descriptions, tags and appropriate thumbnail selection describing the events. Moreover, targeting the labour expensive generation of highlights after the game, and even in the half-time break, we also aim for AI-automated solutions for collecting the most relevant events and combining and clipping these into a highlight summary package to be used by various actors like the league streaming services, media and TV broadcasters.