Scene and activity identification in video summary generation based on motion detected in a video

• US 9,666,232 B2
• Filed: 04/05/2016
• Issued: 05/30/2017
• Est. Priority Date: 08/20/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Adsumilli, Balineedu Chowdary, Lustig, Ryan, Newman, David A.
• Primary Examiner(s): JONES, HEATHER RAE
• Application Number:
  US15/091,539
• Publication Number:
  US 20160224834A1
• Time in Patent Office: 420 Days
• Field of Search: None
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/13 the information being derived from movement of the record carrier, e.g. using tachometer
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 9/8205 : involving the multiplexing of an additional signal and the colour video signal

Abstract

Video and corresponding metadata is accessed. Events of interest within the video are identified based on the corresponding metadata, and best scenes are identified based on the identified events of interest. In one example, best scenes are identified based on the motion values associated with frames or portions of a frame of a video. Motion values are determined for each frame and portions of the video including frames with the most motion are identified as best scenes. Best scenes may also be identified based on the motion profile of a video. The motion profile of a video is a measure of global or local motion within frames throughout the video. For example, best scenes are identified from portion of the video including steady global motion. A video summary can be generated including one or more of the identified best scenes.

Claims

1. A system configured to generate a video summary from recorded video footage, the system comprising:

storage media that stores video including multiple frames and audio captured contemporaneously with the frames, the video having an unedited viewing time;

one or more physical processor configured by machine readable instructions to;

obtain a user-selected summary length for a video summary of the video;

obtain metadata for the video, the metadata including biometric information of a user captured contemporaneously with capture of the video, motion values that characterize motion of objects visible within the frames of the video, and camera motion information characterizing position and/or motion of the camera during capture of the video;

analyze the biometric information of the user captured contemporaneously with capture of the video, the motion values that characterize motion of objects visible within the frames of the video, the camera motion information characterizing position and/or motion of the camera during capture of the video, and the audio captured contemporaneously with the frames of the video to identify events of interest within the video;

rank the events of interest within the video based on the biometric information of the user captured contemporaneously with capture of the individual events of interest, the motion values that characterize motion of objects visible within the frames of the individual events of interest, the camera motion information characterizing position and/or motion of the camera during capture of the individual events of interest, and the audio captured contemporaneously with capture of the individual events of interest;

select portions of the video for inclusion in the video summary based on the identified events of interest, the ranking, and the user-selected summary length; and

generate an electronic file defining the video summary from the selected portions of the video so that the video summary has the user-selected summary length.

2. The system of claim 1, wherein the events of interest include a first event of interest identified on the basis of the biometric information of the user captured contemporaneously with capture of the first event of interest.

3. The system of claim 2, wherein the one or more physical processors are configured such that identification of the first event of interest is based on one or more of the biometric information of the user captured contemporaneously with capture of the first event of interest crossing a first threshold, a maximum and/or a minimum in the biometric information of the user captured contemporaneously with capture of the first event of interest, and/or a change in the biometric information of the user captured contemporaneously with the first event of interest.

4. The system of claim 3, wherein the events of interest include a second event of interest identified on the basis of motion of an object visible within the frames of the second event of interest as characterized by a motion value for the object.

5. The system of claim 4, wherein the one or more physical processors are configured such that identification of the second event of interest is based on one or more of the motion value that characterizes motion of the object visible within the frames of the second event of interest crossing a second threshold, a maximum and/or a minimum of the motion value that characterizes motion of the object visible within the frames of the second event of interest, and/or a change in the motion value that characterizes motion of the object visible within the frames of the second event of interest.

6. The system of claim 5, wherein the events of interest include a third event of interest identified on the basis of a change in the position and/or motion of the camera during capture of the video during capture of the third event of interest as characterized by the camera motion information.

7. The system of claim 6, wherein the one or more physical processors are configured such that identification of the third event of interest is based on the camera motion information for the third event of interest indicating a change in the position and/or motion of the camera.

8. The system of claim 1, wherein the events of interest include a second event of interest identified on the basis of motion of an object visible within the frames of the second event of interest as characterized by a motion value for the object.

9. The system of claim 8, wherein the one or more physical processors are configured such that identification of the second event of interest is based on one or more of the motion value that characterizes motion of the object visible within the frames of the second event of interest crossing a second threshold, a maximum and/or a minimum of the motion value that characterizes motion of the object visible within the frames of the second event of interest, and/or a change in the motion value that characterizes motion of the object visible within the frames of the second event of interest.

10. The system of claim 9, wherein

the one or more motion values that characterize motion of objects visible within the frames of the video include a differential motion parameter associated with a frame of the frames of the video;

the differential motion parameter is being determined based on a difference of the one or more motion values associated with the frame and the one or more motion values associated with another frame preceding the frame.

11. The system of claim 1, wherein the events of interest include a third event of interest identified on the basis of a change in the position and/or motion of the camera during capture of the video during capture of the third event of interest as characterized by the camera motion information.

12. The system of claim 11, wherein the one or more physical processors are configured such that identification of the third event of interest is based on the camera motion information for the third event of interest indicating a change in the position and/or motion of the camera.

13. The system of claim 1, wherein the events of interest include a fourth event of interest identified on the basis of the audio captured contemporaneously with the fourth event of interest.

14. The system of claim 13 wherein one or more physical processors are configured such that identification of the fourth event of interest is based on one or more of a level change of the audio captured contemporaneously with the fourth event of interest, recognition of a word or phrase in the audio captured contemporaneously with the fourth event of interest, and/or a non-verbal pattern or noise present in the audio captured contemporaneously with the fourth event.

15. The system of claim 1, wherein the one or more physical processors are configured such that an amount of separate portions of the video selected for inclusion in the video summary is based on the user-selected summary length.

16. The system of claim 1, wherein the one or more motion values that characterize motion of objects visible within the frames of the video include a differential motion parameter associated with a frame of the frames of the video.

17. The system of claim 16, wherein:

the differential motion parameter is being determined based on a difference of the one or more motion values associated with the frame and the one or more motion values associated with another frame preceding the frame.

18. The system of claim 16, wherein:

the one or more motion values include a first motion value associated with a portion of the frame at a first location within the frame;

the one or more motion values include a second motion value associated with another portion of another frame at a second location within another frame, the second location within another frame corresponding to the first location within the frame; and

the differential motion parameter is being determined based on a difference operation between the first motion value and the second motion value.