Scene and activity identification in video summary generation

• US 10,074,013 B2
• Filed: 10/13/2014
• Issued: 09/11/2018
• Est. Priority Date: 07/23/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Silver, Adam
• Primary Examiner(s): Casanova, Jorge A
• Application Number:
  US14/513,149
• Publication Number:
  US 20160027470A1
• Time in Patent Office: 1,429 Days
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G11B 27/002 Programmed access in sequence to a plurality of record carriers or indexed parts, e.g. tracks, thereof, e.g. for editing; transfer of record carriers from magazine G11B15/68, G11B17/10; G11B17/22
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/13 the information being derived from movement of the record carrier, e.g. using tachometer
• CPC Class Codes: G11B 27/22 Means responsive to presence or absence of recorded information signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: G11B 27/34 : Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 13/172 : image signals comprising non-image signal components, e.g. headers or format information
• CPC Class Codes: H04N 13/178 : Metadata, e.g. disparity information
• CPC Class Codes: H04N 19/513 : Processing of motion vectors
• CPC Class Codes: H04N 21/2353 : specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/8549 : Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 2201/3228 : further additional information (metadata) being comprised in the identification information
• CPC Class Codes: H04N 2201/3229 : further additional information (metadata) being comprised in the file name (including path, e.g. directory or folder names at one or more higher hierarchical levels)
• CPC Class Codes: H04N 2201/3256 : colour related metadata, e.g. colour, ICC profiles for use made of colour related metadata see the appropriate place, e.g. H04N1/60, H04N1/60 and subgroups
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 5/77 : between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 : the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/91 : Television signal processing therefor
• CPC Class Codes: H04N 9/8205 : involving the multiplexing of an additional signal and the colour video signal

Abstract

Video and corresponding metadata is accessed. Events of interest within the video are identified based on the corresponding metadata, and best scenes are identified based on the identified events of interest. A video summary can be generated including one or more of the identified best scenes. The video summary can be generated using a video summary template with slots corresponding to video clips selected from among sets of candidate video clips. Best scenes can also be identified by receiving an indication of an event of interest within video from a user during the capture of the video. Metadata patterns representing activities identified within video clips can be identified within other videos, which can subsequently be associated with the identified activities.

Claims

1. A method of generating a video summary of a video, the method comprising:

accessing metadata associated with a video, the accessed metadata representative of one or more aspects of the capture of the video as a function of time during capture of the video;

identifying patterns in the metadata as a function of time that correspond to performance of one or more activities being performed by a subject of the video;

determining the one or more activities being performed by the subject of the video during specific portions of the video based on the identifications of the patterns in the metadata as a function of time, the one or more activities including a first activity performed by the subject of the video during a first portion of the video, the first activity being of a given type of activity;

identifying moments within the video at which events of interest are captured in the video based on the accessed metadata, the moments including a first moment during performance of the first activity by the subject of the video at which a first event of interest occurs, the first event being of a given type of event;

identifying individual highlight scenes in the video for the individual events of interest, wherein lengths of footage in the video included in the highlight scenes before and after the moments at which the events of interest occur are based on types of the activities being performed by the subject of the video at the moments in the video at which the events of interest are captured and types of the events such that a first scene in the video is identified for the first event of interest and a length of footage included in the first scene before and after the first moment is a first length based on the given type of activity being of a first type of activity and the given type of event being of a first type of event, a second length based on the given type of activity being of the first type of activity and the given type of event being of a second type of event, a third length based on the given type of activity being of a second type of activity and the given type of event being of the first type of event, and a fourth length based on the given type of activity being of the second type of activity and the given type of event being of the second type of event, the first length different from the second length and the third length, the third length different from the fourth length; and

generating a video summary of the video for playback, the video summary including at least one of the highlight scenes.

2. The method of claim 1, wherein generating the video summary comprises concatenating a plurality of the highlight scenes.

3. The method of claim 1, wherein the accessed metadata is generated by a camera during the capture of the video.

4. The method of claim 3, wherein the accessed metadata comprises telemetry data describing a motion of the camera during the capture of the video.

5. The method of claim 3, wherein the accessed metadata comprises location data describing a location of the camera during the capture of the video.

6. The method of claim 3, wherein the accessed metadata comprises biometric data describing characteristics of a user of the camera during the capture of the video.

7. The method of claim 1, wherein the accessed metadata is accessed from an external entity after the capture of the video.

8. The method of claim 7, wherein the accessed metadata comprises environment data describing characteristics of an environment in which the video was captured.

9. The method of claim 1, further comprising:

ranking the identified highlight scenes based on a likelihood that the identified highlight scene will be of interest to a user;

wherein the at least one highlight scene included in the video summary are selected based on the ranking.

10. The method of claim 1, wherein the video includes multiple video segments captured at non-contiguous times.

11. A system that generates a video summary of a video, the system comprising:

a non-transitory computer-readable storage medium storing instructions configured to, when executed;

access metadata associated with a video, the accessed metadata representative of one or more aspects of capture of the video as a function of time during capture of the video;

identify patterns in the metadata as a function of time that correspond to performance of one or more activities being performed by a subject of the video;

determine the one or more activities being performed by the subject of the video during specific portions of the video based on the identifications of the patterns in the metadata as a function of time, the one or more activities including a first activity performed by the subject of the video during a first portion of the video, the first activity being of a given type of activity;

identify moments within the video at which events of interest are captured in within the video based on the accessed metadata, the moments including a first moment during performance of the first activity by the subject of the video at which a first event of interest occurs, the first event being of a given type of event;

identify individual highlight scenes in the video for the individual events of interest, wherein lengths of footage in the video included in the highlight scenes before and after the moments at which the events of interest occur are based on types of the activities being performed by the subject of the video at the moments in the video at which the events of interest are captured and types of the events such that a first scene in the video is identified for the first event of interest and a length of footage included in the first scene before and after the first moment is a first length based on the given type of activity being of a first type of activity and the given type of event being of a first type of event, a second length based on the given type of activity being of the first type of activity and the given type of event being of a second type of event, a third length based on the given type of activity being of a second type of activity and the given type of event being of the first type of event, and a fourth length based on the given type of activity being of the second type of activity and the given type of event being of the second type of event, the first length different from the second length and the third length, the third length different from the fourth length; and

generate a video summary of the video for playback, the video summary including at least one of the highlight scenes; and

a processor configured to execute the instructions.

12. The system of claim 11, wherein the accessed metadata is generated by a camera during the capture of the video and comprises one or more of:

telemetry data describing a motion of the camera during the capture of the video, location data describing a location of the camera during the capture of the video, and biometric data describing characteristics of a user of the camera during the capture of the video.

13. The system of claim 11, wherein the accessed metadata is accessed from an external entity after the capture of the video and comprises environment data describing characteristics of an environment in which the video was captured.

14. The system of claim 11, wherein the instructions are further configured to:

rank the identified highlight scenes based on a likelihood that the identified highlight scene will be of interest to a user;

wherein the at least one highlight scene included in the video summary based on the ranking.

15. A non-transitory computer-readable storage medium storing instructions for identifying scenes in captured video for inclusion in a video summary, the instructions configured to, when executed:

access metadata associated with a video, the accessed metadata representative of one or more aspects of capture of the video as a function of time during capture of the video;

identify patterns in the metadata as a function of time that correspond to performance of one or more activities being performed by a subject of the video;

determine the one or more activities being performed by the subject of the video during specific portions of the video based on the identifications of the patterns in the metadata as a function of time, the one or more activities including a first activity performed by the subject of the video during a first portion of the video, the first activity being of a given type of activity;

identify moments within the video at which events of interest are captured in within the video based on the accessed metadata, the moments including a first moment during performance of the first activity by the subject of the video at which a first event of interest occurs, the first event being of a given type of event;

identify individual highlight scenes in the video for the individual events of interest, wherein lengths of footage in the video included in the highlight scenes before and after the moments at which the events of interest occur are based on types of the activities being performed by the subject of the video at the moments in the video at which the events of interest are captured and types of the events such that a first scene in the video is identified for the first event of interest and a length of footage included in the first scene before and after the first moment is a first length based on the given type of activity being of a first type of activity and the given type of event being of a first type of event, a second length based on the given type of activity being of the first of activity and the given type of event being of a second type of event, a third length based on the given type of activity being of a second type of activity and the given type of event being of the first type of event, and a fourth length based on the given type of activity being of the second type of activity and the given type of event being of the second type of event, the first length different from the second length and the third length, the third length different from the fourth length; and

generate a video summary of the video for playback, the video summary including at least one of the highlight scenes.

16. The computer-readable storage medium of claim 15, wherein the accessed metadata is generated by a camera during the capture of the video and comprises one or more of:

telemetry data describing a motion of the camera during the capture of the video, location data describing a location of the camera during the capture of the video, and biometric data describing characteristics of a user of the camera during the capture of the video.

17. The computer-readable storage medium of claim 15, wherein the accessed metadata is accessed from an external entity after the capture of the video and comprises environment data describing characteristics of an environment in which the video was captured.

18. The computer-readable storage medium of claim 15, wherein the instructions are further configured to:

rank the identified highlight scenes based on a likelihood that the identified highlight scene will be of interest to a user;

wherein the at least one highlight scene included in the video summary based on the ranking.