Method and system for creating an automatic video summary

• US 10,459,975 B1
• Filed: 12/20/2016
• Issued: 10/29/2019
• Est. Priority Date: 12/20/2016
• Status: Active Grant

• Current Assignee: Shutterstock Incorporated
• Original Assignee: Shutterstock Incorporated
• Inventors: Malpani, Vaibhav
• Primary Examiner(s): Daye, Chelcie L
• Application Number:
  US15/385,737
• Time in Patent Office: 1,043 Days
• Field of Search: 707706
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/735 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

A method is provided for receiving a selection of a key frame in a video and including the key frame within a first plurality of frames from the video. The method includes adjusting at least one weighting factor for a value entry in the first plurality of frames to obtain a second plurality of frames having at least one reduced dimensionality and obtaining a one-dimensional representation of the first plurality of frames with a plurality of weighted entry values from the second plurality of frames. Further, the method includes correlating two entry values in the one-dimensional representation to obtain a selector, storing the at least one weighting factor as a network entry in a process network when the selector picks the key frame from the first plurality of frames, and applying the process network to a third plurality of frames in the video to form a summary of the video.

Claims

1. A computer-implemented method, comprising:

receiving an input selection of a key frame in a video clip;

including the key frame within a first plurality of frames selected from the video clip;

adjusting at least one weighting factor for a value entry in the first plurality of frames to obtain a second plurality of frames having at least one reduced dimensionality;

obtaining a one-dimensional representation of the first plurality of frames with a plurality of weighted entry values from the second plurality of frames;

correlating at least two entry values in the one-dimensional representation to obtain a selector, wherein the selector is a one-dimensional array with each entry comprising a probability that each of a second plurality of frames be selected as the key frame in a video summary of the video clip;

storing the at least one weighting factor as a network entry in a process network when the selector picks the key frame from the first plurality of frames; and

applying the process network to a third plurality of frames in the video clip to form the video summary of the video clip.

2. The computer-implemented method of claim 1, wherein adjusting the at least one weighting factor comprises convolving a plurality of entry values in the first plurality of frames in at least one dimension.

3. The computer-implemented method of claim 1, wherein adjusting the at least one weighting factor comprises sub sampling a plurality of entry values in the first plurality of frames in at least one dimension.

4. The computer-implemented method of claim 1, wherein adjusting the at least one weighting factor comprises training a three-dimensional convolved neural network on the first plurality of frames.

5. The computer-implemented method of claim 1, wherein correlating at least two entry values in the one-dimensional representation comprises adjusting at least another weighting factor to ensure that a value of ‘

one’

is associated with the key frame.

6. The computer-implemented method of claim 1, wherein correlating at least two entry values in the one-dimensional representation comprises adjusting at least another weighting factor to ensure that a value of ‘

zero’

is associated with a frame in the first plurality of frames other than the key frame.

7. The computer-implemented method of claim 1, wherein receiving an input selection of a key frame in a video clip comprises receiving an input selection of a first key frame and a second key frame and selecting the network entry in the process network when the selector picks the first key frame and the second key frame from the first plurality of frames.

8. The computer-implemented method of claim 1, wherein receiving the input selection of the key frame in a video clip comprises receiving a text string describing a feature of the key frame, and storing the text string in an annotated training database.

9. The computer-implemented method of claim 1, further comprising automatically obtaining the key frame when a user uploads the video clip.

10. A system comprising:

one or more processors; and

a non-transitory, computer-readable storage medium coupled to the one or more processors, the non-transitory, computer-readable storage medium including instructions that, when executed by the one or more processors, cause the one or more processors to;

receive a video clip from a user;

request a user selection of a key frame in the video clip;

receive, from the user, the key frame in the video clip;

form a selector with a one-dimensional array with each entry comprising a probability that differentiates each of a plurality of frames adjacent to the key frame from the key frame;

select, with the selector, a plurality of frames from the video clip based on the key frame selected by the user;

form a video summary of the video clip based on the plurality of frames; and

configure the video summary of the video clip to play when the video clip is selected by a search engine and a pointer device hovers over a thumbnail of the video clip.

11. The system of claim 10, wherein to select a plurality of frames from the video clip based on the key frame selected by the user, the instructions cause the processor to select a frame sequence with a frame captured before the key frame and a frame captured after the key frame.

12. The system of claim 10, wherein to form a video summary of the video clip based on the plurality of frames, the instructions cause the processor to apply a process network to a sequence of frames from the video clip.

13. The system of claim 10, wherein to form a video summary of the video clip based on the plurality of frames, the instructions cause the processor to identify a sequence of frames in the video clip as a relevant scene based on a selector in a process network.

14. The system of claim 10, wherein to form a video summary of the video clip based on the plurality of frames, the instructions cause the processor to identify a sequence of frames in the video clip as a relevant scene based on an interaction history with the user.

15. The system of claim 10, wherein to form a video summary of the video clip based on the plurality of frames, the instructions cause the processor to identify a sequence of frames in the video clip as a relevant scene based on an annotated training database including an input from a second user.

16. The system of claim 10, wherein to request a user selection of the key frame in the video clip, the instructions cause the processor to prompt the user to select the key frame from a video clip with a scrolling tool on a display in a client device.

17. The system of claim 10, wherein the instructions cause the processor to form the video summary of the video clip based on the plurality of frames and on a text description of relevant features of the video clip, the text description of relevant features being provided by the user.

18. A computer-implemented method, comprising:

uploading, by a user via an application on a client device, a video clip onto a server;

receiving, by the user in the client device, a request from the server to select a key frame in the video clip; and

providing, to a server, the key frame in the video clip based on a relevant feature to be emphasized in a video summary of the video clip, wherein providing to the server the key frame in the video clip comprises selecting the key frame from the video clip with a scrolling tool provided in a user interface displayed on the client device, and providing a text indicative of a user description of a relevant feature in the key frame so that the server selects multiple frames from the video clip based on a probability to select each of the frames as the key frame.

19. The computer-implemented method of claim 18, wherein providing to the server the key frame in the video clip comprises providing a text description of a relevant feature of the video clip in a user interface displayed on the client device.

20. The computer-implemented method of claim 18, further comprising directing a pointing device over a thumbnail of a second video clip to play a second video summary of the second video clip when the second video clip is listed in a search result of a search engine in the server, wherein the second video clip comprises a second key frame selected by a second user.