Video scene classification by activity

• US 9,984,293 B2
• Filed: 10/13/2014
• Issued: 05/29/2018
• Est. Priority Date: 07/23/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Hodulik, Nick, Taylor, Jonathan
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): NGUYEN, CHAN T H
• Application Number:
  US14/513,151
• Publication Number:
  US 20160027475A1
• Time in Patent Office: 1,324 Days
• Field of Search: 3482313
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G11B 27/002 Programmed access in sequence to a plurality of record carriers or indexed parts, e.g. tracks, thereof, e.g. for editing; transfer of record carriers from magazine G11B15/68, G11B17/10; G11B17/22
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/13 the information being derived from movement of the record carrier, e.g. using tachometer
• CPC Class Codes: G11B 27/22 Means responsive to presence or absence of recorded information signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: G11B 27/34 : Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 13/172 : image signals comprising non-image signal components, e.g. headers or format information
• CPC Class Codes: H04N 13/178 : Metadata, e.g. disparity information
• CPC Class Codes: H04N 19/513 : Processing of motion vectors
• CPC Class Codes: H04N 21/2353 : specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/8549 : Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 2201/3228 : further additional information (metadata) being comprised in the identification information
• CPC Class Codes: H04N 2201/3229 : further additional information (metadata) being comprised in the file name (including path, e.g. directory or folder names at one or more higher hierarchical levels)
• CPC Class Codes: H04N 2201/3256 : colour related metadata, e.g. colour, ICC profiles
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 5/77 : between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 : the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/91 : Television signal processing therefor
• CPC Class Codes: H04N 9/8205 : involving the multiplexing of an additional signal and the colour video signal

Abstract

Video and corresponding metadata is accessed. Events of interest within the video are identified based on the corresponding metadata, and best scenes are identified based on the identified events of interest. A video summary can be generated including one or more of the identified best scenes. The video summary can be generated using a video summary template with slots corresponding to video clips selected from among sets of candidate video clips. Best scenes can also be identified by receiving an indication of an event of interest within video from a user during the capture of the video. Metadata patterns representing activities identified within video clips can be identified within other videos, which can subsequently be associated with the identified activities.

Claims

1. A method for identifying scenes in videos, the method comprising:

obtaining one or more electronic files defining a video captured with a camera, the one or more electronic files including event of interest information indicating reception of verbal input during the capture of the video, the verbal input identifying an occurrence of an event of interest within the video, the event of interest occurring at an event moment within the video, the event of interest information identifying (i) a given input type of the verbal input, input types of the verbal input including a first input type, a second input type, and a third input type, and (ii) an input moment during the capture of the video at which the verbal input was received, wherein the first input type indicates the input moment occurring before the event moment, the second input type indicates the input moment occurring during the event moment, and the third input type indicates the input moment occurring after the event moment;

identifying the input moment based on the event of interest information;

identifying the given input type of the verbal input based on the event of interest information;

identifying the event moment based on the input moment and the given input type, wherein the event moment is identified to occur before the input moment based on the given input type being the first type, the event moment is identified to occur during the input moment based on the given input type being the second type, and the event moment is identified to occur after the input moment based on the given input type being the third type;

identifying a portion of the video as a video clip associated with the event of interest based on the event of interest information, the video clip comprising a first time amount of the video occurring before the event moment and a second time amount of the video occurring after the event moment, the first time amount and the second time amount being determined based on a type of an activity captured within the video; and

storing clip information indicating the association of the video clip with the event of interest and the portion of the video included in the video clip.

2. The method of claim 1, wherein the occurrence of the event of interest is identified further based on metadata associated with the video, the metadata captured during the capture of the video, the metadata characterizing velocity or acceleration of the activity captured within the video.

3. The method of claim 2, wherein a metadata criteria for identifying the occurrence of the event of interest based on the metadata associated with the video is based on the type of the activity captured within the video such that a first metadata criteria is used to identify the occurrence of the event of interest based on the activity being of a first type and a second metadata criteria is used to identify the occurrence of the event of interest based on the activity being of a second type, the first metadata criteria being different from the second metadata criteria.

4. The method of claim 1, wherein the verbal input comprises a spoken command associated with tagging events of interest.

5. The method of claim 1, wherein the event moment includes a point in time within the video or a duration of time within the video.

6. The method of claim 1, further comprising:

identifying one or more non-event moments within the video, the one or more non-event moments not associated with any event of interest within the video; and

storing non-event information indicating the one or more non-event moments.

7. The method of claim 6, wherein the one or more non-event moments are identified based on matching metadata associated with the video with a metadata pattern determined to not be of interest to a user, the metadata characterizing velocity of the activity captured within the video, acceleration of the activity captured within the video, visuals captured within the video, or audio captured within the video.

8. A system for identifying scenes in videos, the system comprising:

one or more physical processors configured by computer-readable instructions to;

obtain one or more electronic files defining a video captured with a camera, the one or more electronic files including event of interest information indicating reception of verbal input during the capture of the video, the verbal input identifying an occurrence of an event of interest within the video, the event of interest occurring at an event moment within the video, the event of interest information identifying (i) a given input type of the verbal input, input types of the verbal input including a first input type, a second input type, and a third input type, and (ii) an input moment during the capture of the video at which the verbal input was received, wherein the first input type indicates the input moment occurring before the event moment, the second input type indicates the input moment occurring during the event moment, and the third input type indicates the input moment occurring after the event moment;

identify the input moment based on the event of interest information;

identify the given input type of the verbal input based on the event of interest information;

identify the event moment based on the input moment and the given input type, wherein the event moment is identified to occur before the input moment based on the given input type being the first type, the event moment is identified to occur during the input moment based on the given input type being the second type, and the event moment is identified to occur after the input moment based on the given input type being the third type;

identify a portion of the video as a video clip associated with the event of interest based on the event of interest information, the video clip comprising a first time amount of the video occurring before the event moment and a second time amount of the video occurring after the event moment, the first time amount and the second time amount being determined based on a type of an activity captured within the video; and

store clip information indicating the association of the video clip with the event of interest and the portion of the video included in the video clip.

9. The system of claim 8, wherein the occurrence of the event of interest is identified further based on metadata associated with the video, the metadata captured during the capture of the video, the metadata characterizing velocity or acceleration of the activity captured within the video.

10. The system of claim 9, wherein a metadata criteria for identifying the occurrence of the event of interest based on the metadata associated with the video is based on the type of the activity captured within the video such that a first metadata criteria is used to identify the occurrence of the event of interest based on the activity being of a first type and a second metadata criteria is used to identify the occurrence of the event of interest based on the activity being of a second type, the first metadata criteria being different from the second metadata criteria.

11. The system of claim 8, wherein the verbal input comprises a spoken command associated with tagging events of interest.

12. The system of claim 8, wherein the event moment includes a point in time within the video or a duration of time within the video.

13. The system of claim 8, wherein the one or more physical processors are further configured by the computer-readable instructions to:

identify one or more non-event moments within the video, the one or more non-event moments not associated with any event of interest within the video; and

store non-event information indicating the one or more non-event moments.

14. The system of claim 13, wherein the one or more non-event moments are identified based on matching metadata associated with the video with a metadata pattern determined to not be of interest to a user, the metadata characterizing velocity of the activity captured within the video, acceleration of the activity captured within the video, visuals captured within the video, or audio captured within the video.

15. A non-transitory computer-readable storage medium storing instructions for identifying scenes in videos, the instructions, when executed by one or more physical processors, configured to cause the one or more physical processors to:

obtain one or more electronic files defining a video captured with a camera, the one or more electronic files including event of interest information indicating reception of verbal input during the capture of the video, the verbal input identifying an occurrence of an event of interest within the video, the event of interest occurring at an event moment within the video, the event of interest information identifying (i) a given type of the verbal input, input types of the verbal input including a first input type, a second input type, and a third input type, and (ii) an input moment during the capture of the video at which the verbal input was received, wherein the first input type indicates the input moment occurring before the event moment, the second input type indicates the input moment occurring during the event moment, and the third input type indicates the input moment occurring after the event moment;

identify the input moment based on the event of interest information;

identify the given input type of the verbal input based on the event of interest information;

identify the event moment based on the input moment and the given input type, wherein the event moment is identified to occur before the input moment based on the given input type being the first type, the event moment is identified to occur during the input moment based on the given input type being the second type, and the event moment is identified to occur after the input moment based on the given input type being the third type;

identify a portion of the video as a video clip associated with the event of interest based on the event of interest information, the video clip comprising a first time amount of the video occurring before the event moment and a second time amount of the video occurring after the event moment, the first time amount and the second time amount being determined based on a type of an activity captured within the video; and

store clip information indicating the association of the video clip with the event of interest and the portion of the video included in the video clip.

16. The computer-readable storage medium of claim 15, wherein the occurrence of the event of interest is identified further based on metadata associated with the video, the metadata captured during the capture of the video, the metadata characterizing velocity or acceleration of the activity captured within the video.

17. The computer-readable storage medium of claim 16, wherein a metadata criteria for identifying the occurrence of the event of interest based on the metadata associated with the video is based on the type of the activity captured within the video such that a first metadata criteria is used to identify the occurrence of the event of interest based on the activity being of a first type and a metadata second criteria is used to identify the occurrence of the event of interest based on the activity being of a second type, the first metadata criteria being different from the second metadata criteria.

18. The computer-readable storage medium of claim 15, wherein the verbal input comprises a spoken command associated with tagging events of interest.

19. The computer-readable storage medium of claim 15, wherein the event moment includes a point in time within the video or a duration of time within the video.

20. The computer-readable storage medium of claim 15, wherein the instructions, when executed by the one or more physical processors, are further configured to cause the one or more physical processors to,

identify one or more non-event moments within the video, the one or more non-event moments not associated with any event of interest within the video;

and

store non-event information indicating the one or more non-event moments.