SELECTING REPRESENTATIVE VIDEO FRAMES FOR VIDEOS

• US 20160378863A1
• Filed: 06/24/2015
• Published: 12/29/2016
• Est. Priority Date: 06/24/2015
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Abu-El-Haija, Sami Ahmad, Shlens, Jonathon, Toderici, George Dan
• Application Number:
  US14/749,436
• Publication Number:
  US 20160378863A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for selecting representative frames for videos. One of the methods includes receiving a search query; determining a query representation for the search query; obtaining data identifying a plurality of responsive videos for the search query, wherein each responsive video comprises a plurality of frames, wherein each frame has a respective frame representation; selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video; and generating a response to the search query, wherein the response to the search query includes a respective video search result for each of the responsive videos, and wherein the respective video search result for each of the responsive videos includes a presentation of the representative video frame from the responsive video.

Claims

1. A method comprising:

receiving a search query, wherein the search query comprises one or more query terms;

determining a query representation for the search query, wherein the query representation is a vector of numbers in a high-dimensional space;

obtaining data identifying a plurality of responsive videos for the search query, wherein each responsive video comprises a plurality of frames, wherein each frame has a respective frame representation, and wherein each frame representation is a vector of numbers in the high-dimensional space;

selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video; and

generating a response to the search query, wherein the response to the search query includes a respective video search result for each of the responsive videos, and wherein the respective video search result for each of the responsive videos includes a presentation of the representative video frame from the responsive video.

2. The method of claim 1, wherein the respective video search result for each of the responsive videos includes a link to playback of the responsive video starting from the representative frame from the responsive video.

3. The method of claim 1, wherein selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video comprises:

computing a respective distance measure between the query representation and each of the frame representations for the frames in the responsive video frame.

4. The method of claim 3, wherein selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video further comprises:

selecting as the representative frame a frame having a frame representation that is closest to the query representation according to the distance measure.

5. The method of claim 3, wherein selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video further comprises:

generating a respective probability for each of the frames from the distance measures;

determining whether a highest probability for any of the frames exceeds a threshold value;

when the highest probability exceeds the threshold value, selecting the frame having the highest probability as the representative frame.

6. The method of claim 5, wherein selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video further comprises:

when the highest probability does not exceed the threshold value, selecting a default frame as the representative frame.

7. The method of claim 1, wherein determining the query representation for the search query comprises:

determining a respective term representation for each of the one or more terms in the search query, wherein the term representation is a representation of the term in the high-dimensional space; and

determining the query representation from the one or more term representations.

8. The method of claim 1, further comprising:

determining, for each of the responsive videos, the respective frame representation for each of the plurality of frames from the responsive video.

9. The method of claim 8, wherein determining the respective frame representation for each of the plurality of frames from the responsive video comprises:

maintaining data mapping each label in a predetermined set of labels to a respective label representation, wherein each label representation is a vector of numbers in the high-dimensional space;

processing the frame using a deep convolutional neural network to generate a set of label scores for the frame, wherein the set of label scores includes a respective score for each label in the predetermined set of labels, and wherein the respective score for each of the labels represents a likelihood that the frame contains an image of an object from an object category labeled by the label; and

computing the frame representation for the frame from the set of label scores for the frame and the label representations.

10. The method of claim 8, wherein computing the frame representation for the frame from the set of label scores for the frame and the label representations comprises:

computing, for each of the labels, a weighted representation for the label by multiplying the label score for the label by the label representation for the label; and

computing the frame representation for the frame by computing a sum of the weighted representations.

11. The method of claim 8, wherein determining the respective frame representation for each of the plurality of frames from the responsive video comprises:

processing the frame using a modified image classification neural network to generate the frame representation for the frame, wherein the modified image classification neural network comprises;

an initial image classification neural network configured to process the frame to generate a respective label score for each label of a predetermined set of labels, and

an embedding layer configured to receive the label scores and to generate the frame representation for the frame.

12. The method of claim 11, wherein the modified image classification convolutional neural network has been trained on a set of training triplets, each training triplet comprising a respective training frame from a respective training video, a positive query representation, and a negative query representation.

13. The method of claim 12, wherein the positive query representation is a query representation for a search query that is associated with the training video and the negative query representation is a query representation for a search query that is not associated with the training video.

14. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving a search query, wherein the search query comprises one or more query terms;

determining a query representation for the search query, wherein the query representation is a vector of numbers in a high-dimensional space;

obtaining data identifying a plurality of responsive videos for the search query, wherein each responsive video comprises a plurality of frames, wherein each frame has a respective frame representation, and wherein each frame representation is a vector of numbers in the high-dimensional space;

selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video; and

generating a response to the search query, wherein the response to the search query includes a respective video search result for each of the responsive videos, and wherein the respective video search result for each of the responsive videos includes a presentation of the representative video frame from the responsive video.

15. The system of claim 14, wherein the respective video search result for each of the responsive videos includes a link to playback of the responsive video starting from the representative frame from the responsive video.

16. The system of claim 14, wherein selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video comprises:

computing a respective distance measure between the query representation and each of the frame representations for the frames in the responsive video frame.

17. The system of claim 16, wherein selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video further comprises:

selecting as the representative frame a frame having a frame representation that is closest to the query representation according to the distance measure.

18. The system of claim 16, wherein selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video further comprises:

generating a respective probability for each of the frames from the distance measures;

determining whether a highest probability for any of the frames exceeds a threshold value;

when the highest probability exceeds the threshold value, selecting the frame having the highest probability as the representative frame.

19. The system of claim 14, wherein determining the query representation for the search query comprises:

determining a respective term representation for each of the one or more terms in the search query, wherein the term representation is a representation of the term in the high-dimensional space; and

determining the query representation from the one or more term representations.

20. A computer program product encoded on one or more non-transitory computer readable media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving a search query, wherein the search query comprises one or more query terms;

determining a query representation for the search query, wherein the query representation is a vector of numbers in a high-dimensional space;

obtaining data identifying a plurality of responsive videos for the search query, wherein each responsive video comprises a plurality of frames, wherein each frame has a respective frame representation, and wherein each frame representation is a vector of numbers in the high-dimensional space;

selecting, for each responsive video, a representative frame from the responsive video using the query representation and the frame representations for the frames in the responsive video; and

generating a response to the search query, wherein the response to the search query includes a respective video search result for each of the responsive videos, and wherein the respective video search result for each of the responsive videos includes a presentation of the representative video frame from the responsive video.