DETECT SPORTS VIDEO HIGHLIGHTS FOR MOBILE COMPUTING DEVICES

• US 20160292510A1
• Filed: 03/31/2015
• Published: 10/06/2016
• Est. Priority Date: 03/31/2015
• Status: Active Grant

• Current Assignee: Beijing Shunyuan Kaihua Technology Limited
• Original Assignee: Zepp Labs, Inc. (Zepp Health Corporation)
• Inventors: Han, Zheng, Dai, Xiaowei, Huang, Xianjun, Yang, Fan
• Granted Patent: US 10,572,735 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/06 Cutting and rejoining; Notching, or perforating record carriers otherwise than by recording styli record carriers with provision for splicing G11B23/20
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A solution is provided for detecting in real time video highlights in a sports video at a mobile computing device. A highlight detection module of the mobile computing device extracts visual features from each video frame of the sports video using a trained feature model and detects a highlight in the video frame based on the extracted visual features of the video frame using a trained detection model. The feature model and detection model are trained with a convolutional neural network on a large corpus of videos to generate category level and pair-wise frame feature vectors. Based on the detection, the highlight detection module generates a highlight score for each video frame of the sports video and presents the highlight scores to users of the computing device. The feature model and detection model are dynamically updated based on the real time highlight detection data collected by the mobile computing device.

Claims

1. A computer-implemented method for detecting highlights in a sports video at a mobile computing device, comprising:

receiving a sports video having a plurality of video frames at the mobile computing device;

buffering a segment of the sports video, the segment comprising a plurality of video frames of the sports video;

for each buffered video frame of the sports video;

extracting a plurality of visual features of the video frame;

detecting a highlight in the video frame based on the extracted visual features of the video frame; and

generating a video highlight score for the video frame based on the detection.

2. The method of claim 1, wherein extracting a plurality of visual features of the video frame comprises:

applying a trained feature model to the video frame; and

generating a plurality of feature vectors based on the application of the trained feature model to the video frame, the generated feature vectors representing category level visual characteristics of the video frame of the sports video.

3. The method of claim 2, wherein the feature model is trained with a convolutional neural network on a large corpus of videos, and wherein the trained feature model is configured to classify the large corpus of videos into a plurality of classes, and each class of the sports videos is associated with a plurality feature vectors describing category level visual characteristics of the class.

4. The method of claim 1, wherein detecting a highlight in the video frame comprises:

applying a trained detection model to the extracted visual features of the video frame; and

comparing the extracted visual features of the video frame with pair-wise frame feature vectors associated with the trained detection model.

5. The method of claim 4, wherein the pair-wise frame feature vectors comprise a feature vector describing visual characteristics of a first video frame having a highlight and a feature vector describing visual characteristics of a second video frame having no highlight, wherein the first video frame and the second video frame are semantically similar to the buffered video frame.

6. The method of claim 4, wherein comparing the extracted visual features of the video frame with the pair-wise frame feature vectors comprises:

generating a distance between the extracted visual features of the video frame of the sports video and the feature vector of the first video frame having a highlight; and

generating the highlight score for the video frame of the sports video based on the generated distance.

7. The method of claim 1, wherein the highlight score of the video frame represents a prediction of the video frame having a highlight.

8. The method of claim 1, further comprising:

presenting the highlight scores of the plurality of video frames of the sports video in a graphical user interface; and

monitoring user interactions with the presented highlight scores of the plurality of video frames of the sports video.

9. The method of claim 8, further comprising:

responsive to detecting a user interaction with a highlight score of a video frame of the sports video, providing highlight detection data of the sports video and user interaction information to a computer server to update a trained detection model.

10. The method of claim 1, further comprising:

providing highlight detection data of the sports video to a computer server to update a trained feature model.

13. The computer readable storage medium of claim 2, wherein the feature model is trained with a convolutional neural network on a large corpus of videos, and wherein the trained feature model is configured to classify the large corpus of videos into a plurality of classes, and each class of the sports video is associated with a plurality feature vectors describing the category level visual characteristics of the class.

18. The computer readable storage medium of claim 1, further computer program instructions when executed by a computer processor cause the computer processor to:

present the highlight scores of the plurality of video frames of the sports video in a graphical user interface; and

monitor user interactions with the presented highlight scores of the plurality of video frames of the sports video.

19. The computer readable storage medium of claim 8, further computer program instructions when executed by a computer processor cause the computer processor to:

responsive to detecting a user interaction with a highlight score of a video frame of the sports video, provide highlight detection data of the sports video and user interaction information to a computer server to update a trained detection model.

20. The computer readable storage medium of claim 1, further computer program instructions when executed by a computer processor cause the computer processor to:

provide highlight detection data of the sports video to a computer server to update a trained feature model.

11. A non-transitory computer readable storage medium storing executable computer program instructions for or detecting highlights in a sports video at a mobile computing device, the instructions when executed by a computer processor cause the computer processor to:

receive a sports video having a plurality of video frames at the mobile computing device;

buffer a segment of the sports video, the segment comprising a plurality of video frames of the sports video;

for each buffered video frame of the sports video;

extract a plurality of visual features of the video frame;

detect a highlight in the video frame based on the extracted visual features of the video frame; and

generate a video highlight score for the video frame based on the detection.

12. The computer readable storage medium of claim 11, wherein the instructions for extracting a plurality of visual features of the video frame comprise instructions when executed by the computer processor that cause the computer processor to:

apply a trained feature model to the video frame; and

generate a plurality of feature vectors based on the application of the trained feature model to the video frame, the generated feature vectors representing category level visual characteristics of the video frame of the sports video.

14. The computer readable storage medium of claim 11, wherein the instructions for detecting a highlight in the video frame comprise instructions when executed by the computer processor that cause the computer processor to:

apply a trained detection model to the extracted visual features of the video frame; and

compare the extracted visual features of the video frame with pair-wise frame feature vectors associated with the trained detection model.

15. The computer readable storage medium of claim 14, wherein the pair-wise frame feature vectors comprise a feature vector describing visual characteristics of a first video frame having a highlight and a feature vector describing visual characteristics of a second video frame having no highlight, wherein the first video frame and the second video frame are semantically similar to the buffered video frame.

16. The computer readable storage medium of claim 14, wherein the instructions for comparing the extracted visual features of the video frame with the pair-wise frame feature vectors comprise instructions when executed by the computer processor that cause the computer processor to:

generate a distance between the extracted visual features of the video frame of the sports video and the feature vector of the first video frame having a highlight; and

generate the highlight score for the video frame of the sports video based on the generated distance.

17. The computer readable storage medium of claim 11, wherein the highlight score of the video frame represents a prediction of the video frame having a highlight.