Systems and methods for accelerating video analysis

• US 10,402,656 B1
• Filed: 07/13/2017
• Issued: 09/03/2019
• Est. Priority Date: 07/13/2017
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Tse, Daniel
• Primary Examiner(s): Bost, Dwayne D
• Assistant Examiner(s): Brinich, Stephen M
• Application Number:
  US15/649,509
• Time in Patent Office: 782 Days
• Field of Search: 382224-229, 382168-172, 715723
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

Video information defining video content may be accessed. The video content may include video frames. The video frames may be grouped into similarity groups based on similarity of the video frames. The similarity groups may include a first similarity group including a first video frame and a second video frame. The first video frame may be similar to the second video frame. Representative video frames may be selected from the similarity groups. An individual representative video frame may correspond to an individual similarity group. Representative video frames may include a first representative video frame selected from and corresponding to the first similarity group. The video content may be analyzed based on the representative video frames.

Claims

1. A system that accelerates video analysis, the system comprising:

one or more physical processors configured by machine readable instruction to;

access video information defining video content, the video content including video frames;

group the video frames into similarity groups based on similarity of the video frames, the similarity groups including a first similarity group including a first video frame and a second video frame, the first video frame similar to the second video frame;

select representative video frames from the similarity groups, an individual representative video frame corresponding to an individual similarity group, wherein the representative video frames include a first representative video frame selected from and corresponding to the first similarity group, the first representative video frame being the first video frame;

analyze the video content based on analysis of the representative video frames to identify visuals within the representative video frames, the identified visuals within the representative video frames representative of visuals within the corresponding similarity groups, wherein visuals within the first similarity group is identified based on analysis of the first video frame and without analysis of the second video frame, the identified visuals within the first video frame being representative of the visuals within the first similarity group; and

determine encoding framerates for individual portions of the video content corresponding to the individual similarity groups based on the identified visuals within the corresponding representative video frames such that a first encoding framerate for a first portion of the video content corresponding to the first similarity group is determined based on the identified visuals within the first video frame, wherein the first portion of the video content corresponding to the first similarity group is encoded using the first encoding framerate.

2. The system of claim 1, wherein the one or more physical processors are, to group the video frames into the similarity groups, further configured by the machine-readable instruction to:

extract some or all of the video frames from the video content;

determine pairwise similarity of the extracted video frames; and

group the video frames into the similarity groups based on the determined pairwise similarity of the extracted video frames.

3. The system of claim 2, wherein the video content is characterized by a framerate and the video frames are extracted from the video content at the framerate.

4. The system of claim 2, wherein the video content is characterized by a first framerate and the video frames are extracted from the video content at a second framerate, the second framerate lower than the first framerate.

5. The system of claim 1, wherein the video frames are grouped into the similarity groups further based on sequentiality of the video frames, the first video frame sequential with the second video frames.

6. The system of claim 1, wherein grouping the video frames into the similarity groups includes determining the similarity based on comparisons of feature vectors of pairs of the video frames, the feature vectors including a first feature vector characterizing a local binary pattern distribution of the individual video frames and a second feature vector characterizing a color distribution of the individual video frames.

7. The system of claim 1, wherein the one or more physical processors are further configured by machine-readable instructions to downsample the video frames, the similarity of the video frames determined based on similarity of the downsampled video frames.

8. The system of claim 1, wherein the analysis of the video content includes at least one of object identification, facial identification, or activity identification of visuals within the video content.

9. The system of claim 1, wherein the video content includes a first segment, a second segment, and a third segment, the second segment located between the first segment and the third segment, and the first similarity group includes non-continuous segments of the video content such that the first similarity group includes the first segment and the third segment and does not include the second segment.

10. A method for accelerating video analysis, the method performed by a computing system including one or more physical processors, the method comprising:

accessing, by the one or more physical processors, video information defining video content, the video content including video frames;

grouping, by the one or more physical processors, the video frames into similarity groups based on similarity of the video frames, the similarity groups including a first similarity group including a first video frame and a second video frame, the first video frame similar to the second video frame;

selecting, by the one or more physical processors, representative video frames from the similarity groups, an individual representative video frame corresponding to an individual similarity group, wherein the representative video frames include a first representative video frame selected from and corresponding to the first similarity group, the first representative video frame being the first video frame;

analyzing, by the one or more physical processors, the video content based on analysis of the representative video frames to identify visuals within the representative video frames, the identified visuals within the representative video frames representative of visuals within the corresponding similarity groups, wherein visuals within the first similarity group is identified based on analysis of the first video frame and without analysis of the second video frame, the identified visuals within the first video frame being representative of the visuals within the first similarity group; and

determining, by the one or more physical processors, encoding framerates for individual portions of the video content corresponding to the individual similarity groups based on the identified visuals within the corresponding representative video frames such that a first encoding framerate for a first portion of the video content corresponding to the first similarity group is determined based on the identified visuals within the first video frame, wherein the first portion of the video content corresponding to the first similarity group is encoded using the first encoding framerate.

11. The method of claim 10, wherein grouping the video frames into the similarity groups includes:

extracting some or all of the video frames from the video content;

determining pairwise similarity of the extracted video frames; and

grouping the video frames into the similarity groups based on the determined pairwise similarity of the extracted video frames.

12. The method of claim 11, wherein the video content is characterized by a framerate and the video frames are extracted from the video content at the framerate.

13. The method of claim 11, wherein the video content is characterized by a first framerate and the video frames are extracted from the video content at a second framerate, the second framerate lower than the first framerate.

14. The method of claim 12, wherein the video frames are grouped into the similarity groups further based on sequentiality of the video frames, the first video frame sequential with the second video frames.

15. The method of claim 10, wherein grouping the video frames into the similarity groups includes determining the similarity based on comparisons of feature vectors of pairs of the video frames, the feature vectors including a first feature vector characterizing a local binary pattern distribution of the individual video frames and a second feature vector characterizing a color distribution of the individual video frames.

16. The method of claim 10, further comprising downsampling the video frames, the similarity of the video frames determined based on similarity of the downsampled video frames.

17. The method of claim 10, wherein the analysis of the video content includes at least one of object identification, facial identification, or activity identification of visuals within the video content.

18. The method of claim 10, wherein the video content includes a first segment, a second segment, and a third segment, the second segment located between the first segment and the third segment, and the first similarity group includes non-continuous segments of the video content such that the first similarity group includes the first segment and the third segment and does not include the second segment.

19. A system that accelerates video analysis, the system comprising:

one or more physical processors configured by machine readable instruction to;

access video information defining video content, the video content including video frames;

extract the video frames from the video content, wherein the video content is characterized by a first framerate and the video frames are extracted from the video content at the first framerate or a second framerate lower than the first framerate;

group the video frames into similarity groups based on similarity of the video frames, the similarity groups including a first similarity group including a first video frame and a second video frame, the first video frame similar to the second video frame, wherein the one or more physical processors are, to group the video frames into the similarity groups, further configured by the machine-readable instruction to;

extract some or all of the video frames from the video content;

determine pairwise similarity of the extracted video frames; and

group the video frames into the similarity groups based on the determined pairwise similarity of the extracted video frames;

select representative video frames from the similarity groups, an individual representative video frame corresponding to an individual similarity group, wherein the representative video frames include a first representative video frame selected from and corresponding to the first similarity group, the first representative video frame being the first video frame;

analyze the video content based on analysis of the representative video frames to identify visuals within the representative video frames, the identified visuals within the representative video frames representative of visuals within the corresponding similarity groups, wherein visuals within the first similarity group is identified based on analysis of the first video frame and without analysis of the second video frame, the identified visuals within the first video frame being representative of the visuals within the first similarity group; and

determine encoding framerates for individual portions of the video content corresponding to the individual similarity groups based on the identified visuals within the corresponding representative video frames such that a first encoding framerate for a first portion of the video content corresponding to the first similarity group is determined based on the identified visuals within the first video frame, wherein the first portion of the video content corresponding to the first similarity group is encoded using the first encoding framerate.

20. The system of claim 19, wherein the video content includes a first segment, a second segment, and a third segment, the second segment located between the first segment and the third segment, and the first similarity group includes non-continuous segments of the video content such that the first similarity group includes the first segment and the third segment and does not include the second segment.