Systems and methods for determining a sample frame order for analyzing a video via segments

• US 10,268,898 B1
• Filed: 09/21/2016
• Issued: 04/23/2019
• Est. Priority Date: 09/21/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Wills, Jonathan, Tse, Daniel, Chik, Desmond
• Primary Examiner(s): Youssef, Menatoallah
• Application Number:
  US15/271,998
• Time in Patent Office: 944 Days
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

System and/or method configured to determine a sample frame order for analyzing a video. The video may have multiple frames ordered in a sequence from a beginning to an end. Segments of the video may be determined based on an analysis of the frames of the video. A first sample frame order for analyzing the video may be determined based on multiple iterations performed on individual ones of the segments of the video. Determining the first sample frame order may include determining initial frames for a first iteration, and determining secondary frames for a second iteration based on a function of frame position in the sequence of frames. The initial frames and the secondary frames may be associated with sample positions in the sample frame order. A first feature of the video may be determined based on an analysis of the frames performed in the first sample frame order.

Claims

1. A system configured to determine a sample frame order for analyzing a video, the system comprising:

one or more processors configured by machine-readable instructions to;

obtain one or more electronic files that define multiple frames of the video, the multiple frames of the video being ordered in a sequence from a beginning to an end;

determine segments of the video based on an analysis of the multiple frames of the video, such that a first segment having a first beginning and a first end, and a second segment having a second beginning and a second end, are determined;

determine a first sample frame order for analyzing the video based on multiple iterations performed on individual ones of the segments of the video, wherein determining the first sample frame order includes;

determining, for a first iteration of the first segment, a first initial frame based on a function of frame position in the sequence of frames between the first beginning of the first segment and the first end of the first segment, the first initial frame having a first initial frame position in the sequence of frames;

determining, for the first iteration of the second segment, a second initial frame based on a function of frame position in the sequence of frames between the second beginning of the second segment and the second end of the second segment, the second initial frame having a second initial frame position in the sequence of frames;

associating the first initial frame and the second initial frame with primary sample positions in a first sample frame order that is different from the sequence of frames;

determining, for a second iteration of the first segment, first secondary frames based on a function of frame position in the sequence of frames between the first beginning of the first segment and the first initial frame position, and between the first initial frame position and the first end of the first segment;

determining, for a second iteration of the second segment, second secondary frames based on a function of frame position in the sequence of frames between the second beginning of the second segment and the second initial frame position, and between the second initial frame position and the second end of the second segment; and

associating the first secondary frames and the second secondary frames with secondary sample positions in the first sample frame order;

analyze the video according to the first sample frame order to determine one or more features of the video;

determine a first feature of the video based on a feature analysis of the frames in the video performed on the frames in the first sample frame order

identify one or more frames from the multiple frames based on the first feature; and

output the one or more frames identified.

2. The system of claim 1, wherein the first initial frame is a first middle frame of the first segment such that determining the first initial frame based on the function of frame position in the sequence of frames between the first beginning of the first segment and the first end of the first segment includes determining a frame in the sequence of frames that is an equal number of frames from both the first beginning of the first segment and the first end of the first segment, and

wherein the second initial frame is a second middle frame of the second segment such that determining the second initial frame based on the function of frame position in the sequence of frames between the second beginning of the second segment and the second end of the second segment includes determining a frame in the sequence of frames that is an equal number of frames from both the second beginning of the second segment and the second end of the second segment.

3. The system of claim 1, wherein the first secondary frames have first secondary frame positions in the sequence of frames, the first secondary frame positions located at, or adjacent to, midpoints between:

the first beginning of the first segment and the first initial frame position, and the first initial frame position and the first end of the first segment, and

wherein the second secondary frames have second secondary frame positions in the sequence of frames, the second secondary frame positions located at, or adjacent to, midpoints between;

the second beginning of the second segment and the second initial frame position, and the second initial frame position and the second end of the second segment.

4. The system of claim 1, wherein the first feature includes one or more of a goodness score, a motion metric, a shape metric, a face metric, and/or a persistence metric.

5. The system of claim 1, wherein the multiple iterations include a quantity of iterations beyond the first iteration and the second iteration, and wherein determining the first sample frame order further includes determining, for individual ones of the quantity of iterations of individual ones of the segments, other sets of frames to be associated with other sample positions in the first sample frame order based on a function of frame position in the sequence of frames between a position of a previously determined frame determined during an immediately preceding iteration within the multiple iterations of a segment, and one of the first beginning of the first segment, the second beginning of the second segment, the first end of the first segment, the second end of the second segment, or another position of another previously determined frame determined during a second preceding iteration that immediately preceded the immediately preceding iteration of the segment.

6. The system of claim 1, wherein the segments of the video are determined based on similarity between neighboring frames in the sequence such that individual ones of the segments of the video comprise multiple similar sequential frames and determining the segments of the video includes determining the similarity between the neighboring frames in the sequence.

7. The system of claim 1, wherein the primary sample positions in the first sample frame order represent a first depth pass of the video and the secondary sample positions in the first sample frame order represent a second depth pass of the video.

8. The system of claim 1, wherein determining the first sample frame order further includes:

determining, for a third iteration of the first segment, first tertiary frames based on a function of frame position in the sequence of frames between the first beginning of the first segment and one of the first secondary frames, one of the first secondary frames and the first initial frame, the first initial frame and one of the secondary frames, and one of the secondary frames and the first end of the first segment;

determining, for a third iteration of the second segment, second tertiary frames based on a function of frame position in the sequence of frames between the second beginning of the second segment and one of the second secondary frames, one of the second secondary frames and the second initial frame, the second initial frame and one of the secondary frames, and one of the secondary frames and the second end of the second segment; and

associating the first and the second tertiary frames with tertiary sample positions in the first sample frame order.

9. The system of claim 1, wherein the one or more processors are further configured by machine-readable instructions to:

determine an interval between a frame position in the sequence of frames of a frame determined for a given iteration and another frame position in the sequence of frames determined for an iteration immediately preceding the given iteration;

determine whether the interval is at least equal to or less than a threshold interval such that the video is analyzed and the first feature is determined responsive to the interval being at least equal to or less than the threshold interval.

10. The system of claim 1, wherein the secondary sample positions come after the primary sample positions and before any other sample positions in the first sample frame order such that the first secondary frames and the second secondary frames associated with the secondary sample positions are analyzed after the first initial frame and the second initial frame associated with the primary sample positions and before any other frames associated with the other sample positions.

11. A method for determining a sample frame order for analyzing a video, the method being implemented in a computer system that includes one or more processors configured by machine-readable instructions, the method comprising:

obtaining one or more electronic files that define multiple frames of the video, the multiple frames of the video being ordered in a sequence from a beginning to an end;

determining segments of the video based on an analysis of the multiple frames of the video, such that a first segment having a first beginning and a first end, and a second segment having a second beginning and a second end, are determined;

determining a first sample frame order for analyzing the video based on multiple iterations performed on individual ones of the segments of the video, wherein determining the first sample frame order includes;

determining, for a first iteration of the first segment, a first initial frame based on a function of frame position in the sequence of frames between the first beginning of the first segment and the first end of the first segment, the first initial frame having a first initial frame position in the sequence of frames;

determining, for the first iteration of the second segment, a second initial frame based on a function of frame position in the sequence of frames between the second beginning of the second segment and the second end of the second segment, the second initial frame having a second initial frame position in the sequence of frames;

associating the first initial frame and the second initial frame with primary sample positions in a first sample frame order that is different from the sequence of frames;

determining, for a second iteration of the first segment, first secondary frames based on a function of frame position in the sequence of frames between the first beginning of the first segment and the first initial frame position, and between the first initial frame position and the first end of the first segment;

determining, for a second iteration of the second segment, second secondary frames based on a function of frame position in the sequence of frames between the second beginning of the second segment and the second initial frame position, and between the second initial frame position and the second end of the second segment; and

associating the first secondary frames and the second secondary frames with secondary sample positions in the first sample frame order;

analyzing the video according to the first sample frame order to determine one or more features of the video;

determining a first feature of the video based on a feature analysis of the frames in the video performed on the frames in the first sample frame order

identifying one or more frames from the multiple frames based on the first feature; and

outputting the one or more frames identified.

12. The method of claim 11, wherein the first initial frame is a first middle frame of the first segment such that determining the first initial frame based on the function of frame position in the sequence of frames between the first beginning of the first segment and the first end of the first segment includes determining a frame in the sequence of frames that is an equal number of frames from both the first beginning of the first segment and the first end of the first segment, and

wherein the second initial frame is a second middle frame of the second segment such that determining the second initial frame based on the function of frame position in the sequence of frames between the second beginning of the second segment and the second end of the second segment includes determining a frame in the sequence of frames that is an equal number of frames from both the second beginning of the second segment and the second end of the second segment.

13. The method of claim 11, wherein the first secondary frames have first secondary frame positions in the sequence of frames, the first secondary frame positions located at, or adjacent to, midpoints between:

the first beginning of the first segment and the first initial frame position, and the first initial frame position and the first end of the first segment, and

wherein the second secondary frames have second secondary frame positions in the sequence of frames, the second secondary frame positions located at, or adjacent to, midpoints between;

the second beginning of the second segment and the second initial frame position, and the second initial frame position and the second end of the second segment.

14. The method of claim 11, wherein the first feature includes one or more of a goodness score, a motion metric, a shape metric, a face metric, and/or a persistence metric.

15. The method of claim 11, wherein the multiple iterations include a quantity of iterations beyond the first iteration and the second iteration, and wherein determining the first sample frame order further includes determining, for individual ones of the quantity of iterations of individual ones of the segments, other sets of frames to be associated with other sample positions in the first sample frame order based on a function of frame position in the sequence of frames between a position of a previously determined frame determined during an immediately preceding iteration within the multiple iterations of a segment, and one of the first beginning of the first segment, the second beginning of the second segment, the first end of the first segment, the second end of the second segment, or another position of another previously determined frame determined during a second preceding iteration that immediately preceded the immediately preceding iteration of the segment.

16. The method of claim 11, wherein the segments of the video are determined based on similarity between neighboring frames in the sequence such that individual ones of the segments of the video comprise multiple similar sequential frames and determining the segments of the video includes determining the similarity between the neighboring frames in the sequence.

17. The method of claim 11, wherein the primary sample positions in the first sample frame order represent a first depth pass of the video and the secondary sample positions in the first sample frame order represent a second depth pass of the video.

18. The method of claim 11, wherein determining the first sample frame order further includes:

determining, for a third iteration of the first segment, first tertiary frames based on a function of frame position in the sequence of frames between the first beginning of the first segment and one of the first secondary frames, one of the first secondary frames and the first initial frame, the first initial frame and one of the secondary frames, and one of the secondary frames and the first end of the first segment;

determining, for a third iteration of the second segment, second tertiary frames based on a function of frame position in the sequence of frames between the second beginning of the second segment and one of the second secondary frames, one of the second secondary frames and the second initial frame, the second initial frame and one of the secondary frames, and one of the secondary frames and the second end of the second segment; and

associating the first and the second tertiary frames with tertiary sample positions in the first sample frame order.

19. The method of claim 11, further comprising:

determining an interval between a frame position in the sequence of frames of a frame determined for a given iteration and another frame position in the sequence of frames determined for an iteration immediately preceding the given iteration;

determining whether the interval is at least equal to or less than a threshold interval such that the video is analyzed and the first feature is determined responsive to the interval being at least equal to or less than the threshold interval.

20. The method of claim 11, wherein the secondary sample positions come after the primary sample positions and before any other sample positions in the first sample frame order such that the first secondary frames and the second secondary frames associated with the secondary sample positions are analyzed after the first initial frame and the second initial frame associated with the primary sample positions and before any other frames associated with the other sample positions.