Collage of interesting moments in a video

• US 10,157,638 B2
• Filed: 06/24/2016
• Issued: 12/18/2018
• Est. Priority Date: 06/24/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Ramaswamy, Sharadh, Grundmann, Matthias, Conley, Kenneth
• Primary Examiner(s): Dang, Hung Q
• Assistant Examiner(s): Park, Sunghyoun
• Application Number:
  US15/192,851
• Publication Number:
  US 20170372749A1
• Time in Patent Office: 907 Days
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/237 Communication with additional data server
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

A computer-implemented method includes determining interesting moments in a video. The method further includes generating video segments based on the interesting moments, wherein each of the video segments includes at least one of the interesting moments from the video. The method further includes generating a collage from the video segments, where the collage includes at least two windows and wherein each window includes one of the video segments.

Claims

1. A computer-implemented method to generate a collage, the method comprising:

determining interesting moments in a video by;

identifying audio in the video;

identifying a type of audio in the video;

generating a respective interest score for two or more instances of audio in the video based on the type of audio; and

determining the interesting moments based at least in part on corresponding interest scores;

selecting video segments based on the interesting moments, wherein each of the video segments includes at least one of the interesting moments from the video; and

generating a collage from the video segments, wherein;

the collage includes at least two windows,

each window includes one of the video segments; and

the windows are displayed with different sizes based on the interest score for the audio for each corresponding video segment.

2. The method of claim 1, further comprising:

receiving a selection of one of the video segments in the collage; and

causing the video to be displayed at a time position in the video that corresponds to the selection.

3. The method of claim 1, wherein a largest window size corresponds to a highest interest score for the audio.

4. The method of claim 1, wherein at least a first segment of the video segments in the collage is configured to play at a different frame rate than other video segments in the collage.

5. The method of claim 1, wherein:

determining the interesting moments in the video further includes receiving an identification of at least one of the interesting moments from a user; and

generating video segments based on the interesting moments includes;

determining a beginning and an end of continual motion of at least a first object in the video that appears in the video at one of the interesting moments; and

cutting the video into a first video segment that includes at least a first frame that corresponds to the beginning of the continual motion and at least a second frame that corresponds to the end of the continual motion.

6. The method of claim 1, wherein generating video segments based on the interesting moments includes:

determining markers that indicate different sections within the video, wherein the markers include at least one of metadata, black frames, white frames, a title card, and a chapter card; and

generating video segments that include interesting moments that occur within a single section.

7. The method of claim 1, wherein the sizes of the windows are further based on at least one of a length of each of the video segments or an artistic effect.

8. A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that, when executed by one or more computers, cause the one or more computers to perform operations to generate a collage comprising:

determining interesting moments in a video by;

identifying two or more instances of audio that are associated with motion in the video;

identifying a type of audio in the video;

generating an interest score for each type of motion in the video based on the type of audio; and

determining the interesting moments based at least in part on corresponding interest scores;

generating video segments based on the interesting moments, wherein each of the video segments includes at least one of the interesting moments from the video; and

generating a collage from the video segments, wherein;

the collage includes at least two windows;

each window includes one of the video segments; and

the windows are displayed with different sizes based on the interest score for the audio for each corresponding video segment.

9. The computer storage medium of claim 8, wherein the instructions cause the one or more computers to further perform operations comprising:

receiving a selection of one of the video segments in the collage; and

causing the video to be displayed at a time position in the video that corresponds to the selection.

10. The computer storage medium of claim 8, wherein one or more of the video segments in the collage are configured to play automatically.

11. The computer storage medium of claim 8, wherein at least a first segment of the video segments in the collage is configured to play at a different frame rate than other video segments in the collage.

12. The computer storage medium of claim 8, wherein:

determining the interesting moments in the video further includes receiving an identification of at least one of the interesting moments from a user; and

generating video segments based on the interesting moments includes;

determining a beginning and an end of continual motion of at least a first object in the video that appears in the video at one of the interesting moments; and

cutting the video into a first video segment that includes at least a first frame that corresponds to the beginning of the continual motion and at least a second frame that corresponds to the end of the continual motion.

13. The computer storage medium of claim 8, wherein generating video segments based on the interesting moments includes:

determining markers that indicate different sections within the video, wherein the markers include at least one of metadata, black frames, and white frames; and

generating video segments that include interesting moments that occur within a single section.

14. The computer storage medium of claim 8, wherein the sizes of the windows are further based on at least one of a length of each of the video segments or an artistic effect.

15. The computer storage medium of claim 8, wherein the at least two windows are organized based on a chronology of the video segments.

16. A system comprising:

one or more hardware processors; and

a memory that stores instructions that, when executed by the one or more hardware processors, causes the one or more hardware processors to perform operations comprising;

determining interesting moments in a video by;

identifying audio in the video;

identifying a type of audio in the video;

generating a respective interest score for two or more instances of audio in the video based on the type of audio; and

determining the interesting moments based at least in part on corresponding interest scores;

generating video segments based on the interesting moments, wherein each of the video segments includes at least one of the interesting moments from the video; and

generating a collage from the video segments, wherein;

the collage includes at least two windows;

each window includes one of the video segments; and

the windows are displayed with different sizes based on the interest score for the audio for each corresponding video segment.

17. The system of claim 16, wherein the operations further comprise:

receiving a selection of one of the video segments in the collage; and

causing the video to be displayed at a time position in the video that corresponds to the selection.

18. The system of claim 16, wherein generating the interest score for each instance of audio in the video includes determining a type of motion based on the type of audio and generating the interest score based on the type of motion.

19. The system of claim 16, wherein at least a first segment of the video segments in the collage is configured to play at a different frame rate than other video segments in the collage.

20. The system of claim 16, wherein:

determining the interesting moments in the video includes receiving an identification of at least one of the interesting moments from a user; and

generating video segments based on the interesting moments includes;

determining a beginning and an end of continual motion of at least a first object in the video that appears in the video at one of the interesting moments; and

cutting the video into a first video segment that includes at least a first frame that corresponds to the beginning of the continual motion and at least a second frame that corresponds to the end of the continual motion.