Systems and methods for determining highlight segment sets

• US 10,002,641 B1
• Filed: 10/17/2016
• Issued: 06/19/2018
• Est. Priority Date: 10/17/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Chik, Desmond, Tse, Daniel, Wills, Jonathan
• Primary Examiner(s): Jones, Heather
• Application Number:
  US15/295,862
• Time in Patent Office: 610 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06F 2218/16 by matching signal segments
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier

Abstract

A system and/or method configured to determine highlight segment sets. Content files that define content in content segment sets may be obtained. Individual highlight segment sets of content segments may be determined from the content segment sets. Determining a first highlight segment set of content segments included in the first content segment may include iterating (a)-(c) for multiple iterations. At (a), individual content segments included in the first content segment set may be selected as a selected content segment for inclusion in the first highlight segment set. At, (b) diversity scores for content segments that are (i) included in the first content segment set and (ii) not yet selected for inclusion in the first highlight segment set may be determined. At (c), one or more of the content segments may be disqualified for inclusion in the first highlight segment set for future iterations based on the diversity scores.

Claims

1. A system configured for determining highlight segment sets, the system comprising:

one or more processors configured by machine-readable instructions to;

obtain content files that define content in content segment sets, the content segment sets including a first content segment set that includes a first content segment and a second content segment;

determine individual highlight segment sets of content segments from the content segment sets, wherein determining a first highlight segment set of content segments included in the first content segment set includes;

(a) selecting an individual content segment included in the first content segment set as a selected content segment for inclusion in the first highlight segment set based on selection criterion, the selection criterion including a first selection criterion;

(b) determining diversity scores for content segments that are (i) included in the first content segment set and (ii) not yet selected for inclusion in the first highlight segment set, the diversity scores indicating a level of similarity between the individual content segments not yet selected for inclusion in the first highlight segment set and the selected content segment selected at operation (a); and

(c) disqualifying one or more of the content segments for inclusion in the first highlight segment set for future iterations based on the diversity scores,

iterate (a)-(c) for multiple iterations to determine the first highlight segment set.

2. The system of claim 1, wherein the one or more processors are further configured by machine-readable instructions to iterate (a)-(c) until a threshold number of content segments are selected for inclusion in the first highlight segment set.

3. The system of claim 1, wherein the one or more processors are further configured by machine-readable instructions to iterate (a)-(c) until there are no content segments remaining in the first content segment set that are not disqualified or included in the first highlight segment set.

4. The system of claim 1, wherein the diversity scores are determined based on visual similarity, temporal similarity, and semantic similarity between the individual content segments not yet selected for inclusion in the first highlight segment set and the selected content segment selected at operation (a), such that for a first iteration of (a)-(c), a first diversity score is determined based on the visual similarity, the temporal similarity, and the semantic similarity between the second content segment and the first content segment.

5. The system of claim 4, wherein the diversity scores are further determined based on emotive similarity between the individual content segments not yet selected for inclusion in the first highlight segment set and the selected content segment selected at operation (a), such that for a first iteration of (a)-(c), the first diversity score is further determined based on the emotive similarity between the second content segment and the first content segment.

6. The system of claim 1, wherein the second content segment is disqualified at the first iteration based on a first diversity score determined for the second content segment being below a threshold diversity score, the first diversity score indicating the level of similarity between the second content segment and the first content segment.

7. The system of claim 1, wherein the first content segment set includes the first content segment, the second content segment, a third content segment, and a fourth content segment, such that determining diversity scores for a first iteration of (b) includes determining a first diversity score for the second content segment, determining a second diversity score for the third content segment, and determining a third diversity score for the fourth content segment,

wherein the first diversity score for the second content segment is determined based on the visual similarity, the temporal similarity, and the semantic similarity between the second content segment and the first content segment, the second diversity score for the third content segment is determined based on the visual similarity, the temporal similarity, and the semantic similarity between the third content segment and the first content segment, and wherein the third diversity score for the fourth content segment is determined based on the visual similarity, the temporal similarity, and the semantic similarity between the fourth content segment and the first content segment.

8. The system of claim 7, wherein responsive to the first diversity score and the second diversity score being below a threshold diversity score for the first iteration, the one or more processors are further configured by machine-readable instructions to disqualify the second content segment and the third content segment for inclusion in the first highlight segment set for future iterations.

9. The system of claim 1, wherein the one or more processors are configured by machine-readable instructions to:

generate a graphical representation of attributes of one or more of the second content segment, a third content segment, and/or a fourth content segment based on one or more of a first diversity score determined for the second content segment, a second diversity score determined for the third content segment, and a third diversity score determined for the fourth content segment;

wherein one or more of the content segments are disqualified for inclusion in the first highlight segment set for future iterations based on the graphical representation.

10. The system of claim 1, wherein the second content segment is disqualified during a first iteration based on a first diversity score determined for the second content segment being less than other diversity scores for other content segments in the first content segment set, wherein the first diversity score for the first iteration indicates the level of similarity between the second content segment and the first content segment.

11. The system of claim 1, wherein the first selection criterion includes a quality score.

12. The system of claim 1, wherein the first content segment set includes one or more video segments, images, and/or audio segments.

13. A method for determining highlight segment sets, the method being implemented in a computer system that includes one or more processors configured by machine-readable instructions, the method comprising:

obtaining content files that define content in content segment sets, the content segment sets including a first content segment set that includes a first content segment and a second content segment;

determining individual highlight segment sets of content segments from the content segment sets, wherein determining a first highlight segment set of content segments included in the first content segment set includes;

(a) selecting an individual content segment included in the first content segment set as a selected content segment for inclusion in the first highlight segment set based on selection criterion, the selection criterion including a first selection criterion;

(b) determining diversity scores for content segments that are (i) included in the first content segment set and (ii) not yet selected for inclusion in the first highlight segment set, the diversity scores indicating a level of similarity between the individual content segments not yet selected for inclusion in the first highlight segment set and the selected content segment selected at operation (a); and

(c) disqualifying one or more of the content segments for inclusion in the first highlight segment set for future iterations based on the diversity scores,

iterating (a)-(c) for multiple iterations to determine the first highlight segment set.

14. The method of claim 13, further comprising iterating (a)-(c) until a threshold number of content segments are selected for inclusion in the first highlight segment set.

15. The method of claim 13, further comprising iterating (a)-(c) until there are no content segments remaining in the first content segment set that are not disqualified or included in the first highlight segment set.

16. The method of claim 13, wherein the diversity scores are determined based on visual similarity, temporal similarity, and semantic similarity between the individual content segments not yet selected for inclusion in the first highlight segment set and the selected content segment selected at operation (a), such that for a first iteration of (a)-(c), a first diversity score is determined based on the visual similarity, the temporal similarity, and the semantic similarity between the second content segment and the first content segment.

17. The method of claim 13, wherein the diversity scores are further determined based on emotive similarity between the individual content segments not yet selected for inclusion in the first highlight segment set and the selected content segment selected at operation (a), such that for a first iteration of (a)-(c), a first diversity score is further determined based on the emotive similarity between the second content segment and the first content segment.

18. The method of claim 13, wherein the second content segment is disqualified at the first iteration based on a first diversity score determined for the second content segment being below a threshold diversity score, the first diversity score indicating the level of similarity between the second content segment and the first content segment.

19. The method of claim 13, wherein the first content segment set includes the first content segment, the second content segment, a third content segment, and a fourth content segment, such that determining diversity scores for a first iteration of (b) includes determining a first diversity score for the second content segment, determining a second diversity score for the third content segment, and determining a third diversity score for the fourth content segment,

wherein the first diversity score for the second content segment is determined based on the visual similarity, the temporal similarity, and the semantic similarity between the second content segment and the first content segment, the second diversity score for the third content segment is determined based on the visual similarity, the temporal similarity, and the semantic similarity between the third content segment and the first content segment, and wherein the third diversity score for the fourth content segment is determined based on the visual similarity, the temporal similarity, and the semantic similarity between the fourth content segment and the first content segment.

20. The method of claim 19, wherein responsive to the first diversity score and the second diversity score being below a threshold diversity score, disqualifying one or more the content segments includes disqualifying the second content segment and the third content segment for inclusion in the first highlight segment set for future iterations.

21. The method of claim 13, further comprising:

generating a graphical representation of attributes of one or more of the second content segment, a third content segment, and/or a fourth content segment based on one or more of a first diversity score determined for the second content segment, a second diversity score determined for the third content segment, and a third diversity score determined for the fourth content segment;

wherein one or more of the content segments are disqualified for inclusion in the first highlight segment set for future iterations based on the graphical representation.

22. The method of claim 13, wherein the second content segment is disqualified during a first iteration based on a first diversity score determined for the second content segment being less than other diversity scores for other content segments in the first content segment set, wherein the first diversity score for the first iteration indicates the level of similarity between the second content segment and the first content segment.

23. The method of claim 13, wherein the first selection criterion includes a quality score.

24. The method of claim 13, wherein the first content segment set includes one or more video segments, images, and/or audio segments.