Scene and activity identification in video summary generation based on motion detected in a video

• US 9,646,652 B2
• Filed: 05/06/2015
• Issued: 05/09/2017
• Est. Priority Date: 08/20/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Adsumilli, Balineedu Chowdary, Lustig, Ryan, Newman, David A.
• Primary Examiner(s): Jones, Heather
• Application Number:
  US14/705,864
• Publication Number:
  US 20160055381A1
• Time in Patent Office: 734 Days
• Field of Search: None
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/13 the information being derived from movement of the record carrier, e.g. using tachometer
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 9/8205 : involving the multiplexing of an additional signal and the colour video signal

Abstract

Video and corresponding metadata is accessed. Events of interest within the video are identified based on the corresponding metadata, and best scenes are identified based on the identified events of interest. In one example, best scenes are identified based on the motion values associated with frames or portions of a frame of a video. Motion values are determined for each frame and portions of the video including frames with the most motion are identified as best scenes. Best scenes may also be identified based on the motion profile of a video. The motion profile of a video is a measure of global or local motion within frames throughout the video. For example, best scenes are identified from portion of the video including steady global motion. A video summary can be generated including one or more of the identified best scenes.

Claims

1. A method for identifying scenes in captured video for inclusion in a video summary, the method comprising:

accessing a video, the video including a plurality of frames;

determining a velocity of content of individual frames of the video;

determining an acceleration of the content of the individual frames of the video based on at least two of the velocities of the content of the individual frames;

determining a score for the individual frames of the video based on the velocity and the acceleration of the content of the individual frames;

selecting one or more of the individual frames of the video based on the determined scores;

identifying, for individual selected frames, a corresponding video scene, the corresponding video scene comprising a first amount of the video occurring before the selected frame and a second amount of the video occurring after the selected frame; and

selecting one or more of the identified video scenes for inclusion in the video summary.

2. The method of claim 1, further comprising:

receiving a request for the video summary from a user; and

generating the video summary including the selected video scenes.

3. The method of claim 2, wherein generating the video summary comprises concatenating the selected video scenes.

4. The method of claim 1, wherein determining the velocity of the content of the individual frames of the video comprises:

determining a magnitude of one or more motion vectors in the individual frames; and

determining the velocity of the content of the individual frames based on the magnitude of the one or more motion vectors in the individual frames.

5. The method of claim 1, wherein determining the acceleration of the content of the individual frames of the video comprises:

determining the velocities of the content of two adjacent individual frames; and

determining the acceleration of the content of the individual frames based on the difference between the velocities of the content of the two adjacent individual frames.

6. The method of claim 5, further comprising:

aggregating a threshold number of the accelerations of the content of the individual frames in a sequence of frames; and

determining a single acceleration for the individual frames in the sequence of frames based on the aggregated accelerations.

7. The method of claim 1, further comprising:

ranking the identified video scenes based on a likelihood that the identified video scenes will be of interest to a user;

wherein the one or more of the identified video scenes are selected for inclusion in the video summary based on the ranking.

8. A system for identifying scenes in captured video for inclusion in a video summary, the system comprising:

a non-transitory computer-readable storage medium comprising executable computer instructions configured to, when executed, cause a hardware processor to perform steps comprising;

accessing a video, the video including a plurality of frames;

determining a velocity of content of individual frames of the video;

determining an acceleration of the content of the individual frames of the video based on at least two of the velocities of the content of the individual frames;

determining a score for the individual frames of the video based on the velocity and the acceleration of the content of the individual frames;

selecting one or more of the individual frames of the video based on the determined scores;

identifying, for individual selected frames, a corresponding video scene, the corresponding video scene comprising a first amount of the video occurring before the selected frame and a second amount of the video occurring after the selected frame; and

selecting one or more of the identified video scenes for inclusion in the video summary; and

the hardware processor.

9. The system of claim 8, wherein the computer instructions are further configured to, when executed, cause the hardware processor to perform further steps comprising:

receiving a request for the video summary from a user; and

generating the video summary including the selected video scenes.

10. The system of claim 9, wherein generating the video summary comprises concatenating the selected video scenes.

11. The system of claim 8, wherein determining the velocity of the content of the individual frames of the video comprises:

determining a magnitude of one or more motion vectors in the individual frames; and

determining the velocity of the content of the individual frames based on the magnitude of the one or more motion vectors in the individual frames.

12. The system of claim 8, wherein determining the acceleration of the content of the individual frames of the video comprises:

determining the velocities of the content of two adjacent individual frames; and

determining the acceleration of the content of the individual frames based on the difference between the velocities of the content of the two adjacent individual frames.

13. The system of claim 12, wherein the computer instructions are further configured to, when executed, cause the hardware processor to perform further steps comprising:

aggregating a threshold number of the accelerations of the content of the individual frames in a sequence of frames; and

determining a single acceleration for the individual frames in the sequence of frames based on the aggregated accelerations.

14. The system of claim 8, wherein the computer instructions are further configured to, when executed, cause the hardware processor to perform further steps comprising:

ranking the identified video scenes based on a likelihood that the identified video scenes will be of interest to a user;

wherein the one or more of the identified video scenes are selected for inclusion in the video summary based on the ranking.

15. A non-transitory computer-readable storage medium storing executable computer instructions configured to, when executed by a processor, cause the processor to perform steps comprising:

accessing a video, the video including a plurality of frames;

determining a velocity of content of individual frames of the video;

determining an acceleration of the content of the individual frames of the video based on at least two of the velocities of the content of the individual frames;

determining a score for the individual frames of the video based on the velocity and the acceleration of the content of the individual frames;

selecting one or more of the individual frames of the video based on the determined scores;

identifying, for individual selected frames, a corresponding video scene, the corresponding video scene comprising a first amount of the video occurring before the selected frame and a second amount of the video occurring after the selected frame; and

selecting one or more of the identified video scenes for inclusion in a video summary.

16. The computer-readable storage medium of claim 15, wherein the computer instructions are further configured to, when executed, cause the processor to perform further steps comprising:

receiving a request for the video summary from a user; and

generating the video summary including the selected video scenes.

17. The computer-readable storage medium of claim 16, wherein generating the video summary comprises concatenating the selected video scenes.

18. The computer-readable storage medium of claim 15, wherein determining the velocity of the content of the individual frames of the video comprises:

determining a magnitude of one or more motion vectors in the individual frames; and

determining the velocity of the content of the individual frames based on the magnitude of the one or more motion vectors in the individual frames.

19. The computer-readable storage medium of claim 15, wherein determining the acceleration of the content of the individual frames of the video comprises:

determining the velocities of the content of two adjacent individual frames; and

determining the acceleration of the content of the individual frames based on the difference between the velocities of the content of the two adjacent individual frames.

20. The computer-readable storage medium of claim 19, wherein the computer instructions are further configured to, when executed, cause the processor to perform further steps comprising:

aggregating a threshold number of the accelerations of the content of the individual frames in a sequence of frames; and

determining a single acceleration for the individual frames in the sequence of frames based on the aggregated accelerations.

21. The computer-readable storage medium of claim 15, wherein the computer instructions are further configured to, when executed, cause the processor to perform further steps comprising:

ranking the identified video scenes based on a likelihood that the identified video scenes will be of interest to a user;

wherein the one or more of the identified video scenes are selected for inclusion in the video summary based on the ranking.