Detect sports video highlights for mobile computing devices
First Claim
1. A computer-implemented method for detecting highlights in a sports video of a sport at a mobile computing device, comprising:
- receiving a sports video having a plurality of video frames at the mobile computing device; and
for each video frame of the plurality of video frames;
extracting, using a feature model that is trained to identify classes of sports in a single video frame, a plurality of visual features of the video frame, the feature model is trained, using images of the sports, to extract frame-based features;
identifying a class of sport of the video frame;
identifying pair-wise frame feature vectors that are for the class of sport of the video frame, each pair-wise frame feature vectors comprising;
a first feature vector describing visual characteristics of a first video frame having a highlight and having the class of sport, anda second feature vector describing visual characteristics of a second video frame having no highlight and having the class of sport,wherein the first video frame and the second video frame are images of the class of sport of the video frame, andwherein the pair-wise frame feature vectors are generated during a training phase and based on a model previously trained using a training set comprising first sports images of the class of sport that include highlights and second sports images of the class of sport that do not include highlights, and first frame-based features extracted by the feature model and corresponding to the first sports images, and second frame-based features extracted by the feature model and corresponding to the second sports images; and
generating a highlight score for the video frame by;
determining first distances between the extracted visual features and respective first feature vectors of the pair-wise frame feature vectors;
determining second distances between the extracted visual features and respective second feature vectors of the pair-wise frame feature vector; and
combining the first distances and the second distances to generate the highlight score for the video frame.
3 Assignments
0 Petitions
Accused Products
Abstract
A solution is provided for detecting in real time video highlights in a sports video at a mobile computing device. A highlight detection module of the mobile computing device extracts visual features from each video frame of the sports video using a trained feature model and detects a highlight in the video frame based on the extracted visual features of the video frame using a trained detection model. The feature model and detection model are trained with a convolutional neural network on a large corpus of videos to generate category level and pair-wise frame feature vectors. Based on the detection, the highlight detection module generates a highlight score for each video frame of the sports video and presents the highlight scores to users of the computing device. The feature model and detection model are dynamically updated based on the real time highlight detection data collected by the mobile computing device.
59 Citations
20 Claims
-
1. A computer-implemented method for detecting highlights in a sports video of a sport at a mobile computing device, comprising:
-
receiving a sports video having a plurality of video frames at the mobile computing device; and for each video frame of the plurality of video frames; extracting, using a feature model that is trained to identify classes of sports in a single video frame, a plurality of visual features of the video frame, the feature model is trained, using images of the sports, to extract frame-based features; identifying a class of sport of the video frame; identifying pair-wise frame feature vectors that are for the class of sport of the video frame, each pair-wise frame feature vectors comprising; a first feature vector describing visual characteristics of a first video frame having a highlight and having the class of sport, and a second feature vector describing visual characteristics of a second video frame having no highlight and having the class of sport, wherein the first video frame and the second video frame are images of the class of sport of the video frame, and wherein the pair-wise frame feature vectors are generated during a training phase and based on a model previously trained using a training set comprising first sports images of the class of sport that include highlights and second sports images of the class of sport that do not include highlights, and first frame-based features extracted by the feature model and corresponding to the first sports images, and second frame-based features extracted by the feature model and corresponding to the second sports images; and generating a highlight score for the video frame by; determining first distances between the extracted visual features and respective first feature vectors of the pair-wise frame feature vectors; determining second distances between the extracted visual features and respective second feature vectors of the pair-wise frame feature vector; and combining the first distances and the second distances to generate the highlight score for the video frame. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A non-transitory computer readable storage medium storing executable computer program instructions for or detecting highlights in a sports video of a sport at a mobile computing device, the instructions when executed by a computer processor cause the computer processor to:
-
receive a sports video having a plurality of video frames at the mobile computing device; and for each video frame of the plurality of video frames; extract, using a feature model that is trained to identify classes of sports in a single video frame, a plurality of visual features of the video frame, the feature model is trained, using images of the sports, to extract frame-based features; identify a class of sport of the video frame; identify pair-wise frame feature vectors that are for the class of sport of the video frame, each pair-wise frame feature vectors comprising; a first feature vector describing visual characteristics of a first video frame having a highlight and having the class of sport, and a second feature vector describing visual characteristics of a second video frame having no highlight and having the class of sport, wherein the first video frame and the second video frame are images of the class of sport of the video frame, and wherein the pair-wise frame feature vectors are generated during a training phase and based on a model previously trained using
a training set comprising first sports images of the class of sport that include highlights and second sports images of the class of sport that do not include highlights, and
first frame-based features extracted by the feature model and corresponding to the first sports images, and second frame-based features extracted by the feature model and corresponding to the second sports images; andgenerate a highlight score for the video frame by; determining first distances between the extracted visual features and respective first feature vectors of the pair-wise frame feature vectors; determining second distances between the extracted visual features and respective second feature vectors of the pair-wise frame feature vector; and combining the first distances and the second distances to generate the highlight score for the video frame. - View Dependent Claims (9, 10, 11, 12, 13)
-
-
14. An apparatus for detecting highlights in a sports video of a sport, comprising:
-
a non-transitory memory; and a processor configured to execute instructions stored in the non-transitory memory to; receive a sports video having a plurality of video frames at the apparatus; and for each video frame of the plurality of video frames; extract, using a feature model that is trained to identify classes of sports in a single video frame, a plurality of visual features of the video frame, the feature model is trained, using images of the sports, to extract frame-based features; identify a class of sport of the video frame; identify pair-wise frame feature vectors that are for the class of sport of the video frame, each pair-wise frame feature vectors comprising; a first feature vector describing visual characteristics of a first video frame having a highlight and having the class of sport, and a second feature vector describing visual characteristics of a second video frame having no highlight and having the class of sport, wherein the first video frame and the second video frame are images of the class of sport of the video frame, and wherein the pair-wise frame feature vectors are generated during a training phase and based on a model previously trained using a training set comprising first sports images of the class of sport that include highlights and second sports images of the class of sport that do not include highlights, and first frame-based features extracted by the feature model and corresponding to the first sports images, and second frame-based features extracted by the feature model and corresponding to the second sports images; and generate a highlight score for the video frame by; determining first distances between the extracted visual features and respective first feature vectors of the pair-wise frame feature vectors; determining second distances between the extracted visual features and respective second feature vectors of the pair-wise frame feature vector; and combining the first distances and the second distances to generate the highlight score for the video frame. - View Dependent Claims (15, 16, 17, 18, 19, 20)
-
Specification