Systems and methods for editing videos based on shakiness measures

• US 9,916,863 B1
• Filed: 02/24/2017
• Issued: 03/13/2018
• Est. Priority Date: 02/24/2017
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Medioni, Tom, Garcia, Vincent
• Primary Examiner(s): Chevalier, Robert
• Application Number:
  US15/442,050
• Time in Patent Office: 382 Days
• Field of Search: 386278, 386281, 386290, 386323, 386326
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Video information defining video content may be accessed. The video content may include video frames. Motion vectors for the video frames may be determined. The motion vectors may represent motion of one or more visuals captured within individual video frames. A transformation matrix for the video frames may be determined based on the motion vectors. The transformation matrix may characterize rigid transformations between pairs of the video frames. Shakiness metrics for the video frames may be determined based on the transformation matrix. A shakiness threshold may be obtained. One or more of the video frames may be identified based on the shakiness metrics, the shakiness threshold, and algorithms with hysteresis or finite-state machines. A video summary of the video content may be generated. The video summary may include the one or more identified video frames.

Claims

1. A system for editing videos based on movement captured within the videos, the system comprising:

physical storage media storing video information defining video content, the video content including video frames;

one or more physical processors configured by machine readable instructions to;

access the video information defining the video content;

determine motion vectors for the video frames, the motion vectors representing motion of one or more visuals captured within individual video frames;

determine a transformation matrix for the video frames based on the motion vectors, the transformation matrix characterizing rigid transformations between pairs of the video frames;

determine shakiness metrics for the video frames based on the transformation matrix;

obtain a shakiness threshold; and

identify one or more of the video frames based on the shakiness metrics and the shakiness threshold;

generate a video summary of the video content, the video summary including the one or more identified video frames.

2. The system of claim 1, wherein the shakiness metrics for the video frames are determined based on a rate of change in a lateral translation, a rate of change in a vertical translation, a rate of change in an angular translation, and a rate of change in a linear first-order approximation of a scaling factor of the transformation matrix.

3. The system of claim 2, wherein the rate of change in the lateral translation is normalized based on a width of the video content, the rate of change in the vertical translation is normalized based on a height of the video content, the rate of change in the angular translation is matched to the lateral translation and the vertical translation via a first matching factor, and the rate of change in the linear first-order approximation of the scaling factor is matched to the lateral translation and the vertical translation via a second matching factor.

4. The system of claim 1, wherein the rigid transformations are estimated using a RANSAC algorithm.

5. The system of claim 1, wherein the motion vectors for the video frames are determined based on movement of a pixel group.

6. The system of claim 5, wherein the pixel group includes an eight-by-eight pixel block.

7. The system of claim 1, wherein the one or more physical processors are further configured by machine readable instruction to apply one or more filters to remove noise from the video frames.

8. The system of claim 7, wherein the video content is captured at thirty frames per second, and the one or more filters are applied using a time scale of fives frames.

9. The system of claim 7, wherein the video content is captured at sixty frames per second, and the one or more filters are applied using a time scale of ten frames.

10. A method for editing videos based on movement captured within the videos, the method implemented in a system including one or more physical processors, the method comprising:

accessing, by the one or more physical processors, video information defining video content, the video content including video frames;

determining, by the one or more physical processors, motion vectors for the video frames, the motion vectors representing motion of one or more visuals captured within individual video frames;

determining, by the one or more physical processors, a transformation matrix for the video frames based on the motion vectors, the transformation matrix characterizing rigid transformations between pairs of the video frames;

determining, by the one or more physical processors, shakiness metrics for the video frames based on the transformation matrix;

obtaining, by the one or more physical processors, a shakiness threshold; and

identifying, by the one or more physical processors, one or more of the video frames based on the shakiness metrics and the shakiness threshold;

generating, by the one or more physical processors, a video summary of the video content, the video summary including the one or more identified video frames.

11. The method of claim 10, wherein the shakiness metrics for the video frames are determined based on a rate of change in a lateral translation, a rate of change in a vertical translation, a rate of change in an angular translation, and a rate of change in a linear first-order approximation of a scaling factor of the transformation matrix.

12. The method of claim 11, wherein the rate of change in the lateral translation is normalized based on a width of the video content, the rate of change in the vertical translation is normalized based on a height of the video content, the rate of change in the angular translation is matched to the lateral translation and the vertical translation via a first matching factor, and the rate of change in the linear first-order approximation of the scaling factor is matched to the lateral translation and the vertical translation via a second matching factor.

13. The method of claim 10, wherein the rigid transformations are estimated using a RANSAC algorithm.

14. The method of claim 10, wherein the motion vectors for the video frames are determined based on movement of a pixel group.

15. The method of claim 14, wherein the pixel group includes an eight-by-eight pixel block.

16. The method of claim 10, further comprising applying, by the one or more physical processors, one or more filters to remove noise from the video frames.

17. The method of claim 16, wherein the video content is captured at thirty frames per second, and the one or more filters are applied using a time scale of fives frames.

18. The method of claim 16, wherein the video content is captured at sixty frames per second, and the one or more filters are applied using a time scale of ten frames.

19. A system for editing videos based on movement captured within the videos, the system comprising:

physical storage media storing video information defining video content, the video content including video frames;

one or more physical processors configured by machine readable instructions to;

access the video information defining the video content;

determine motion vectors for the video frames, the motion vectors representing motion of one or more visuals captured within individual video frames, wherein the motion vectors for the video frames are determined based on movement of an eight-by-eight pixel block;

determine a transformation matrix for the video frames based on the motion vectors, the transformation matrix characterizing rigid transformations between pairs of the video frames;

determine shakiness metrics for the video frames based on the transformation matrix, wherein the shakiness metrics for the video frames are determined based on a rate of change in a lateral translation, a rate of change in a vertical translation, a rate of change in an angular translation, and a rate of change in a linear first-order approximation of a scaling factor of the transformation matrix;

obtain a shakiness threshold; and

identify one or more of the video frames based on the shakiness metrics and the shakiness threshold;

generate a video summary of the video content, the video summary including the one or more identified video frames.

20. The system of claim 19, wherein the rate of change in the lateral translation is normalized based on a width of the video content, the rate of change in the vertical translation is normalized based on a height of the video content, the rate of change in the angular translation is matched to the lateral translation and the vertical translation via a first matching factor, and the rate of change in the linear first-order approximation of the scaling factor is matched to the lateral translation and the vertical translation via a second matching factor.