DEEP LEARNING ON IMAGE FRAMES TO GENERATE A SUMMARY

• US 20190180109A1
• Filed: 03/16/2018
• Published: 06/13/2019
• Est. Priority Date: 12/12/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: SINHA, Utkarsh, MAKWANA, Kandarpkumar J., REGALIA, Melissa, CHEN, Wei-Chih, WEISBERG, Joshua B., WANG, Baoyuan, NAHMIAS, Gil M., VESDAPUNT, Noranart
• Granted Patent: US 10,445,586 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

Techniques for automatically selecting image frames from a video and providing the selected image frames to a device for display are disclosed.

Claims

1. A system, comprising:

at least one non-transitory memory storing instructions;

one or more hardware processors that are coupled to the at least one non-transitory memory and that are configured to execute the instructions to cause the system to perform operations comprising;

obtain a plurality of image frames from a video;

generate a score for each of the plurality of image frames of the video based upon a video summarization model;

rank the score for each of the plurality of image frames to generate a ranking;

compute a distance between a top ranked frame from the ranking and each other frame of the plurality of image frames;

generate suppression scores based upon the distance computation;

apply a suppression curve to the generated suppression scores to minimize a likelihood that one of the plurality of image frames proximal to the top ranked frame is selected, wherein the suppression scores for the each other frame of the plurality of image frames are multiplied by a value corresponding to the suppression curve;

after applying the suppression curve to the suppression scores, rank the resultant values to identify a next top ranked image frame;

select the top ranked image frame and the next top ranked image frame for display on a device; and

display the top ranked image frame and the next top ranked image frame on the device.

2. The system of claim 1, wherein the operations further comprise:

iteratively applying the suppression curve to the each other frame of the plurality of image frames, ranking a next resultant values, and selecting a third ranked image frame, wherein any frame selected as a highest ranking frame is excluded from subsequent iterations.

3. The system of claim 1, wherein the device performs the operations and the device is selected from the group consisting of a smartphone, a smartwatch, a tablet, a laptop computer.

4. The system of claim 1, wherein the operations further comprise training a video summarization model by:

obtain a training set of image frames;

obtain ground truth information for each of the training set of image frames, the ground truth information comprising;

a pairwise comparison between each possible pair of image frames in the training set of image frames;

generate a probability for the pairwise comparison to form ground truth information;

apply a machine learning algorithm to the training set of image frames to produce a score for each image in the training set of image frames;

compute a difference between scores obtained for each pair of image frames in the training set of image frames;

apply a normalization function to the difference to generate a normalized difference value;

compare the normalized difference value to the ground truth for each pair of images to ascertain incorrect and correct predictions by the machine learning algorithm; and

provide an indication to the machine learning algorithm of the incorrect and correct predictions to train the machine learning algorithm to generate the video summarization model;

5. The system of claim 1, wherein the operations further comprise:

displaying the top ranked image frame and the next top ranked image frame in a comic strip format.

6. The system of claim 1, wherein the operations further comprise:

providing the top ranked image frame and the next top ranked image frame to a photo-sharing application operating on the device; and

receiving, by the photo-sharing application operating on the device, one or more modifications to the top ranked image frame and the next top ranked image frame.

7. The system of claim 6, wherein the operations further comprise:

wherein the one or more modifications is selected from the group consisting of;

adding text, adding an icon, adding a geometric shape or line, and applying an image filter.

8. The system of claim 1, wherein the suppression curve is an absolute curve and has the top ranked frame centered at a minimum value of 0.

9. The system of claim 1, wherein the operations further comprise:

reducing the frame rate of the video, wherein the plurality of images are obtained from the reduced frame rate video.

10. A computer-implemented method, comprising:

obtaining a plurality of image frames from a video;

generating a score for each of the plurality of image frames of the video based upon a video summarization model;

ranking the score for each of the plurality of image frames to generate a ranking;

computing a distance between a top ranked frame from the ranking and each other frame of the plurality of image frames to generate suppression scores;

applying a suppression curve to the generated suppression scores to minimize a likelihood that one of the plurality of image frames proximal to the top ranked frame is selected, wherein the suppression scores for the each other frame of the plurality of image frames are multiplied by a value corresponding to the suppression curve;

after applying the suppression curve to the suppression scores, ranking the resultant values to identify a next top ranked image frame;

selecting the top ranked image frame and the next top ranked image frame for display on a device; and

displaying the top ranked image frame and the next top ranked image frame on the device.

11. A non-transitory computer readable medium having stored thereon computer readable instructions that are executable to cause one or more processors to perform operations, comprising:

obtaining a plurality of image frames from a video;

generating a score for each of the plurality of image frames of the video based upon a video summarization model;

ranking the score for each of the plurality of image frames to generate a ranking;

computing a distance between a top ranked frame from the ranking and each other frame of the plurality of image frames to generate suppression scores;

applying a suppression curve to the generated suppression scores to minimize a likelihood that one of the plurality of image frames proximal to the top ranked frame is selected, wherein the suppression scores for the each other frame of the plurality of image frames are multiplied by a value corresponding to the suppression curve;

after applying the suppression curve to the suppression scores, ranking the resultant values to identify a next top ranked image frame;

selecting the top ranked image frame and the next top ranked image frame for display on a device; and

displaying the top ranked image frame and the next top ranked image frame on the device.