Scene and activity identification in video summary generation based on motion detected in a video

• US 10,643,663 B2
• Filed: 01/24/2019
• Issued: 05/05/2020
• Est. Priority Date: 08/20/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Adsumilli, Balineedu Chowdary, Lustig, Ryan, Newman, David A.
• Primary Examiner(s): Jones, Heather R
• Application Number:
  US16/256,669
• Publication Number:
  US 20190180110A1
• Time in Patent Office: 467 Days
• Field of Search: None
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/13 the information being derived from movement of the record carrier, e.g. using tachometer
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 9/8205 : involving the multiplexing of an additional signal and the colour video signal

Abstract

Video and corresponding metadata is accessed. Events of interest within the video are identified based on the corresponding metadata, and best scenes are identified based on the identified events of interest. In one example, best scenes are identified based on the motion values associated with frames or portions of a frame of a video. Motion values are determined for each frame and portions of the video including frames with the most motion are identified as best scenes. Best scenes may also be identified based on the motion profile of a video. The motion profile of a video is a measure of global or local motion within frames throughout the video. For example, best scenes are identified from portion of the video including steady global motion. A video summary can be generated including one or more of the identified best scenes.

Claims

1. A method for identifying video clips for inclusion in a video summary, the method comprising:

accessing a video, the video including video frames;

dividing individual video frames into tiles;

determining motion values of individual tiles of the individual video frames, the motion values characterizing movement of visual content within a corresponding tile between proximate video frames, wherein the motion values includes a velocity value and an acceleration value;

identifying tiles in the video frames having motion values that satisfy one or more motion criteria,

identifying one or more of the video frames for inclusion in the video summary based on numbers of the tiles within the video frames that satisfy the one or more motion criteria; and

generating the video summary, the video summary including the one or more identified video frames.

2. The method of claim 1, wherein the tiles are laid out in an array, individual ones of the tiles including multiple pixels.

3. The method of claim 1, wherein the one or more motion criteria includes a first motion criterion defining a first threshold velocity value, the individual tiles satisfying the first motion criterion based on the motion values of the individual tiles being greater than the first threshold velocity value.

4. The method of claim 3, wherein the one or more motion criteria includes a second motion criterion defining a second threshold velocity value, the second threshold velocity value being less than the first threshold velocity value, the individual tiles satisfying the second motion criterion based on the motion values of the individual tiles being less than the second threshold velocity value.

5. The method of claim 1, wherein the one or more motion criteria includes a first motion criterion defining a threshold velocity value and a second motion criterion defining a threshold acceleration value, the individual tiles satisfying the first motion criterion based on a comparison of the motion values of the individual tiles to the threshold velocity value and the individual tiles satisfying the second motion criterion based on a comparison of the motion values of the individual tiles to the threshold acceleration value.

6. The method of claim 1, wherein the one or more motion criteria includes a band motion criterion defining a range of threshold velocity values or a range of threshold acceleration values, the individual tiles satisfying the band motion criterion based on the motion values of the individual tiles falling within the range of threshold velocity values or the range of threshold acceleration values.

7. The method of claim 1, wherein determining the motion values of the individual tiles comprises:

determining one or more motion vectors in the individual tiles; and

determining the motion values of the individual tiles based on the one or more motion vectors in the individual tiles.

8. The method of claim 1, wherein determining the motion values of the individual tiles comprises:

determining one or more motion vectors in two corresponding tiles in two adjacent video frames; and

determining the motion values of the individual tiles based on the difference between the one or more motion vectors in the two corresponding tiles in the two adjacent video frames.

9. A system for identifying video clips for inclusion in a video summary, the system comprising:

one or more physical processors configured by machine readable instructions to;

access a video, the video including video frames;

divide individual video frames into tiles;

determine motion values of individual tiles of the individual video frames, the motion values characterizing movement of visual content within a corresponding tile between proximate video frames, wherein the motion values includes a velocity value and an acceleration value;

identify tiles in the video frames having motion values that satisfy one or more motion criteria,

identify one or more of the video frames for inclusion in the video summary based on numbers of the tiles within the video frames that satisfy the one or more motion criteria; and

generate the video summary, the video summary including the one or more identified video frames.

10. The system of claim 9, wherein the tiles are laid out in an array, individual ones of the tiles including multiple pixels.

11. The system of claim 9, wherein the one or more motion criteria includes a first motion criterion defining a first threshold velocity value, the individual tiles satisfying the first motion criterion based on the motion values of the individual tiles being greater than the first threshold velocity value.

12. The system of claim 11, wherein the one or more motion criteria includes a second motion criterion defining a second threshold velocity value, the second threshold velocity value being less than the first threshold velocity value, the individual tiles satisfying the second motion criterion based on the motion values of the individual tiles being less than the second threshold velocity value.

13. The system of claim 9, wherein the one or more motion criteria includes a first motion criterion defining a threshold velocity value and a second motion criterion defining a threshold acceleration value, the individual tiles satisfying the first motion criterion based on a comparison of the motion values of the individual tiles to the threshold velocity value and the individual tiles satisfying the second motion criterion based on a comparison of the motion values of the individual tiles to the threshold acceleration value.

14. The system of claim 9, wherein the one or more motion criteria includes a band motion criterion defining a range of threshold velocity values or a range of threshold acceleration values, the individual tiles satisfying the band motion criterion based on the motion values of the individual tiles falling within the range of threshold velocity values or the range of threshold acceleration values.

15. The system of claim 9, wherein determining the motion values of the individual tiles comprises:

determining one or more motion vectors in the individual tiles; and

determining the motion values of the individual tiles based on the one or more motion vectors in the individual tiles.

16. The system of claim 9, wherein determining the motion values of the individual tiles comprises:

determining one or more motion vectors in two corresponding tiles in two adjacent video frames; and

determining the motion values of the individual tiles based on the difference between the one or more motion vectors in the two corresponding tiles in the two adjacent video frames.