User input-based video summarization

• US 10,541,000 B1
• Filed: 06/26/2015
• Issued: 01/21/2020
• Est. Priority Date: 06/26/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Karakotsios, Kenneth Mark, Poncini, Deon, Townsend, Matthew Alan
• Primary Examiner(s): Hunter, Mishawn N.
• Application Number:
  US14/752,735
• Time in Patent Office: 1,670 Days
• Field of Search: 386230, 386239, 386241, 386248, 386278, 386291, 386296, 725 32, 725 34, 725 38, 725 40, 725 46, 725 47, 725 60, 725 61, 715723
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 9/79 Processing of colour television signals in connection with recording
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

One or more frames of video data may depict content that is determined to likely be of interest to a user. A video segment that includes the one or more frames may be determined. Based at least partly on one or more first summarization parameters associated with the user, a first video summarization may be generated, where the first video summarization includes the first video segment and possibly other video segments associated with the video data. The first video summarization may be provided to the user. Upon receiving data that is representative of user feedback relating to the first video summarization, one or more second summarization parameters may be determined based at least partly on the data. A second video summarization of the video data may be generated based at least partly on the one or more second summarization parameters. The second video summarization may then be provided to the user.

Claims

1. A system comprising:

memory;

one or more processors; and

one or more computer-executable instructions stored in the memory and executable by the one or more processors to perform operations comprising;

receiving video data generated by a camera of a user device;

determining first summarization parameters based on data associated with a user account, the data including at least one of a search history of items searched for via a website, a browse history of items viewed via the website, or a purchase history of items purchased via the website;

generating a first video summarization of the video data based on the first summarization parameters, wherein generating the first video summarization comprises;

generating a first video segment that includes a first frame of the video data and that is of a first duration that is less than a total duration of the video data;

generating a second video segment that includes a second frame of the video data that is of a second duration that is less than the total duration of the video data; and

generating at least one of the first video segment or the second video segment based on a first directorial style associated with the first summarization parameters;

receiving, from the user device, a request to modify the first video summarization;

determining, based on the request, second summarization parameters that are different than the first summarization parameters, wherein the second summarization parameters are based at least partly on;

identifying one or more prior video summarizations previously generated in association with the user account; and

determining a second directorial style different than the first directorial style and associated with the one or more prior video summarizations;

generating a second video summarization of the video data based on the second summarization parameters, the second video summarization being different than the first video summarization; and

sending the second video summarization to the user device.

2. The system as recited in claim 1, wherein the operations further comprise determining the first frame of the video data and the second frame of the video data by:

determining a curve that includes multiple data points that comprise a first data point that is representative of a first interest value assigned to the first frame and a second data point that is representative of a second interest value assigned to the second frame;

determining that the first data point constitutes a first local maxima of the curve;

determining that the second data point constitutes a second local maxima of the curve;

selecting the first frame based on the first data point constituting the first local maxima of the curve; and

selecting the second frame based on the second data point constituting the second local maxima of the curve.

3. The system as recited in claim 1, wherein the operations further comprise:

determining that a third frame of the video data depicts a first scene;

generating a third video segment that includes the third frame and that is of a third duration that is less than the total duration of the video data;

determining that a fourth frame of the video data depicts a second scene;

generating a fourth video segment that includes the fourth frame and is of a fourth duration that is less than the total duration of the video data;

determining, based on a first interest value assigned to the third frame and a second interest value assigned to the fourth frame, that the third video segment is assigned a higher ranking than the fourth video segment; and

determining, based on the second summarization parameters, that the third video segment is to be included in the second video summarization.

4. The system as recited claim 1, wherein the operations further comprise:

determining that a third frame of the video data depicts a first scene;

generating a third video segment that includes the second frame and is of a third duration that is less than the total duration of the video data;

determining that a similarity between the second video segment and the third video segment is above a threshold value; and

determining, based on the second summarization parameters, that the third video segment is to be included in the second video summarization.

5. The system as recited in claim 1, wherein the operations further comprise:

determining that the purchase history indicates that an item of the items was purchased in association with the user account; and

generating the first video summarization in a particular directorial style that is associated with a category of items that includes the item.

6. A method comprising:

determining multiple video segments associated with video data, wherein determining the multiple video segments associated with the video data comprises determining a first video segment of the multiple video segments that is associated with at least one frame of the video data and that has a duration that is less than a total duration of the video data;

determining one or more summarization parameters based at least partly on a user account and prior to generating a video summarization of the video data, wherein determining the one or more summarization parameters includes;

identifying one or more prior video summarizations previously generated in association with the user account; and

determining one or more characteristics of a directorial style associated with the user account; and

generating, based at least partly on the one or more summarization parameters, the video summarization.

7. The method as recited in claim 6, further comprising:

assigning an interest value to a frame of the video data based at least partly on at least one of a person or a scene depicted in the frame;

determining that the interest value is above a threshold value;

determining a video segment of the multiple video segments, wherein determining the video segment comprises determining that the video segment includes the frame; and

selecting the video segment to be included in the video summarization.

8. The method as recited in claim 6, further comprising:

determining that the first video segment includes a first frame of the video data that depicts first content;

determining a second video segment of the multiple video segments, wherein determining the second video segment comprises determining that the second video segment includes a second frame of the video data that depicts second content;

assigning a first rank to the first video segment based at least partly on the first content;

assigning a second rank to the second video segment based at least partly on the second content;

determining that the first rank is greater than the second rank; and

selecting the first video segment to be included in the video summarization.

9. The method as recited in claim 6, further comprising determining that the video summarization includes at least the first video segment and a second video segment of the multiple video segments, wherein determining that the video summarization includes the at least the first video segment and the second video segment comprises determining that the first video segment includes one or more first frames of the video data that depicts first content and that the second video segment includes one or more second frames of the video data that depict second content.

10. The method as recited in claim 6, further comprising:

determining that the first video segment includes a first frame of the video data that depicts first content;

determining a second video segment of the multiple video segments, wherein determining the second video segment comprises determining that the second video segment includes a second frame of the video data that depicts second content;

determining that a similarity between the first content and the second content is greater than a threshold value;

associating the first video segment and the second video segment with a category;

ranking the first video segment, the second video segment, and additional video segments of the multiple video segments within the category;

determining that a first ranking assigned to the first video segment is greater than a second ranking assigned to the second video segment; and

selecting the first video segment to be included in the video summarization.

11. The method as recited in claim 10, further comprising:

receiving, from a user device associated with the user account, data representative of a request to at least one of modify or replace the first video segment;

selecting the second video segment to be included in a second video summarization of the video data; and

sending, to the user device, the second video summarization.

12. The method as recited in claim 6, further comprising;

receiving, from a user device associated with the user account, data representative of a request to at least one of modify or replace the video summarization, the video summarization including at least a first video segment of the multiple video segments;

determining one or more second summarization parameters, including a second directorial style that is different than the directorial style, based at least partly on the data;

generating, based at least partly on the one or more second summarization parameters, a second video segment of the multiple video segments to be included in a second video summarization of the video data; and

sending, to the user device, the second video summarization.

13. The method as recited in claim 6, further comprising determining that the one or more summarization parameters are based at least partly on at least one of input received from a user associated with the user account, default parameters, prior behavior exhibited by the user, information associated with the user account, or prior behavior exhibited by other users.

14. The method as recited in claim 6, further comprising receiving, from a user device associated with the user account, data representative of feedback relating to the video summarization, wherein the receiving the data includes at least one of:

receiving an indication of approval or disapproval of an entirety of the video summarization;

receiving an indication of approval or disapproval of one or more video segments included in the video summarization;

receiving a request to modify, remove, or replace one or more video segments included in the video summarization;

or

receiving a request to add one or more video segments of the multiple video segments to the video summarization.

15. A system comprising:

memory;

one or more processors; and

one or more computer-executable instructions stored in the memory and executable by the one or more processors to perform operations comprising;

determining that a first frame of video data depicts first content, wherein determining that the first frame of the video data depicts the first content comprises determining that the first content includes at least one of a person, an object, or a scene;

determining one or more first summarization parameters based at least partly on one or more prior video summarizations previously generated in association with a user account, the one or more first summarization parameters including a first directorial style;

generating a first video segment that includes the first frame and is of a first duration that is less than a total duration of the video data;

generating a first video summarization of the video data that includes the first video segment based at least partly on the one or more first summarization parameters, the one or more first summarization parameters including a first directorial style; and

sending, to a user device, the first video summarization.

16. The system as recited in claim 15, wherein the operations further comprise:

receiving, from the user device, data representative of a request to modify the first video summarization;

generating, based at least partly on the data, one or more second summarization parameters that are different than the first summarization parameters, the one or more second summarization parameters including a second directorial style that is different than the first directorial style;

determining that a second frame of the video data depicts second content;

generating a second video segment that includes the second frame and is of a second duration that is less than the total duration of the video data;

selecting, based at least partly on the one or more second summarization parameters, the second video segment; and

generating, based at least partly on the one or more second summarization parameters and the second video segment, a second video summarization.

17. The system as recited in claim 16, wherein the request to modify the first video summarization includes at least one of:

an additional video segment from the video data but not included in the first video summarization to be added, the additional video segment corresponding to the second frame of the video data;

at least one video segment included in the first video summarization be removed or replaced;

or

content depicted in the at least one video segment be modified.

18. The system as recited in claim 15, wherein the operations further comprise:

determining that a second frame of the video data depicts second content;

determining that a third frame of the video data depicts third content;

determining a second video segment that includes the second frame;

determining a third video segment that includes the third frame;

assigning a first rank to the second video segment based at least partly on the second content;

assigning a second rank to the third video segment based at least partly on the third content;

determining that the first rank is greater than the second rank; and

selecting the second video segment to be included in the first video summarization.

19. The system as recited in claim 18, wherein the operations further comprise determining that the one or more first summarization parameters include, in addition to the first directorial style, at least one of a preferred duration of the first video summarization, a preferred duration of video segments included within the first video summarization, or an acceptable level of entropy among at least the first video segment and the second video segment.

20. The system as recited in claim 15, wherein the operations further comprise:

assigning an interest value to the first frame based at least partly on the first content;

determining that the interest value is above a threshold value; and

selecting the first video segment to be included in the first video summarization.