IDENTIFYING EVENTS OF INTEREST WITHIN VIDEO CONTENT

• US 20100071013A1
• Filed: 11/23/2009
• Published: 03/18/2010
• Est. Priority Date: 07/21/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: AOL LLC (Apollo Global Management, Inc.)
• Inventors: Franco, Alexander, Vandermolen, Harmannus, Sommers, Michael, Wiegering, Anthony, Levine, Scott J.
• Granted Patent: US 8,667,521 B2
• Time in Patent Office: Days
• US Class Current: 725/109
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: H04H 60/33 Arrangements for monitoring the users' behaviour or opinions
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/73 using meta-information
• CPC Class Codes: H04N 21/2408 Monitoring of the upstream path of the transmission network, e.g. client requests monitoring data switching networks utilization H04L43/0876; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/25866 Management of end-user data customer care in data networks H04L41/5077
• CPC Class Codes: H04N 21/26603 for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8405 represented by keywords
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

Instant messages sent by a viewer of a video while the video is perceivable to the viewer may be monitored. An event of interest occurring in the video may be identified based on the monitored instant messages. Metadata that describes the event of interest may be generated and assigned to the video. A request for video content associated with the video may be received from a user. An indexed collection of videos that includes the video may be searched for the requested video content. Based on the search, it may be determined that the video satisfies the user'"'"'s request. The event of interest occurring in the video may be identified based on the metadata assigned to the video. User perception of a scene from the video that includes the event of interest may be enabled.

Claims

1. A method comprising:

one or more processors monitoring and aggregating instant messages sent by viewers of a video while the video is perceivable to the viewers;

using the aggregated instant messages to identify an event of interest in the video and to segment the video to generate a video scene containing the event of interest;

generating metadata corresponding to the video scene;

enabling later retrieval of the metadata by a user; and

enabling the user to access and perceive the video scene based on the metadata.

2. The method of claim 1, wherein using the aggregated instant messages to segment the video comprises using the aggregated instant messages to estimate a start time and an end time of a scene of the video in which the event of interest occurs.

3. The method of claim 2, wherein generating the metadata comprises generating metadata that specifies the start time and the end time of the scene of the video in which the event of interest occurs.

4. The method of claim 3, wherein generating the metadata that specifies the scene start time and the scene end time comprises generating metadata that includes the scene start time and the scene end time and that enables the user to selectively view the scene in the video.

5. The method of claim 1 wherein monitoring the instant messages sent by viewers comprises monitoring instant messages sent by a viewer of a video to another viewer of the video while both viewers are concurrently viewing the video.

6. The method of claim 1, wherein aggregating the instant messages comprises:

anticipating the event of interest in the video,

accessing a timeline of the video,

estimating a first window of time within the timeline of the video based on the anticipated event of interest, and

determining an aggregate number of instant messages sent by the viewers of the video during the first window of time.

7. The method of claim 6, wherein estimating the first window of time within the timeline of the video based on the anticipated event of interest comprises:

estimating a duration of time that viewers are typically expected to discuss, through instant messaging, the anticipated event of interest that occurs in the video while perceiving the video;

selecting a particular point in the timeline of the video; and

estimating the first window of time to be a window of time that begins at the particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging, the anticipated event of interest.

8. The method of claim 6, wherein using the aggregated instant messages to identify the event of interest comprises:

accessing a threshold,

determining whether the aggregate number of instant messages sent by the viewers during the first window of time meets the threshold,

if the aggregate number of instant messages sent during the first window of time does not meet the threshold, concluding that the first window of time does not correspond to occurrence of an event of interest in the video and estimating a second window of time in the timeline of the video based on the anticipated event of interest, and

if the aggregate number of instant messages sent during the first window of time meets the threshold, concluding that the first window of time corresponds to occurrence of an event of interest in the video and calculating, based on the first window of time, a start time and an end time of a scene of the video in which the event of interest occurs.

9. The method of claim 8, wherein the second window of time occurs subsequent to the first window of time within the timeline of the video.

10. The method of claim 8, wherein estimating the first window of time comprises:

estimating a duration of time that viewers are typically expected to discuss, through instant messaging, the anticipated event of interest that occurs in the video while perceiving the video;

selecting a particular point in the timeline of the video; and

estimating the first window of time to be a window of time that begins at the particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging, the anticipated event of interest.

11. The method of claim 10,

wherein determining whether the aggregate number of instant messages sent during the first window of time meets the threshold comprises determining that the aggregate number of instant messages sent during the first window of time meets the threshold, and

wherein calculating, based on the first window of time, a start time and an end time of a scene of the video in which the event of interest occurs comprises:

determining an amount of time associated with the anticipated event of interest, and

calculating, based on the particular point in the timeline of the video and based on the determined amount of time associated with the anticipated event of interest, the start time and the end time of the scene in the video in which the event of interest occurs.

12. The method of claim 11, wherein calculating the scene start time comprises:

starting from the selected particular point in the timeline of the video, rewinding the video by the amount of time associated with the anticipated event of interest; and

marking the time at which the rewinding stops as the scene start time.

13. The method of claim 11, wherein calculating the scene end time comprises:

starting from the selected particular point in the timeline of the video, fast forwarding the video by a predetermined amount of time; and

marking the time at which the fast forwarding stops as the scene end time.

14. The method of claim 10, wherein estimating the second window of time comprises:

selecting a second particular point in the timeline of the video; and

estimating the second window of time to be a window of time that begins at the second particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging, the anticipated event of interest.

15. The method of claim 14,

wherein determining whether the aggregate number of instant messages sent by the viewers of the video during the first window of time meets the threshold comprises determining that the aggregate number of instant messages sent by the viewers of the video during the first window of time does not meet the threshold, and

wherein aggregating the instant messages further comprises determining an aggregate number of instant messages sent by the viewers of the video during the second window of time.

16. The method of claim 15, wherein using the aggregated instant messages to identify the event of interest further comprises:

determining whether the aggregate number of instant messages sent by the viewers of the video during the second window of time meets the threshold,

if the aggregate number of instant messages sent by the viewers during the second window of time does not meet the threshold, concluding that the second window of time does not correspond to occurrence of an event of interest in the video and estimating a third window of time in the timeline of the video based on the anticipated event of interest, and

if the aggregate number of instant messages sent by the viewers during the second window of time meets the threshold, concluding that the second window of time corresponds to occurrence of an event of interest in the video and calculating, based on the second window of time, a start time and an end time of a scene of the video in which the event of interest occurs.

17. The method of claim 6 wherein determining an aggregate number of the instant messages that are received during the first window of time comprises:

receiving an indication the video has started and an identification of the viewers of the video,

initializing a video clock,

receiving instant messages sent by the viewers and assigning a time T0 to each instant message, the time T0 being a time of receipt of the instant message as indicated by the video clock,

determining an instant message input probability curve for each instant message that has been assigned a time T0 that falls within the first window of time, and

combining the instant message input probability curves for the instant messages that have been assigned a time T0 that falls within the window of time.

18. The method of claim 17 wherein determining the instant message input probability curve for an instant message that has been assigned a time T0 that falls within the first window of time comprises:

determining a network delay,

moving backwards from the time T0 by the network delay to a time T-1,

determining an amount of typing time for the identified instant message,

moving backwards from T-1 by the amount of typing time to a time T-2, and

determining an instant message input probability curve, wherein the instant message input probability curve comprises a probability distribution curve centered at T-2 that reflects a time at which a sender of the instant message began inputting the instant message.

19. The method of claim 1 wherein monitoring the instant messages sent by viewers comprises monitoring instant messages sent by a viewer of the video to a user who is not viewing the video.

20. The method of claim 1 wherein monitoring the instant messages sent by viewers comprises monitoring the instant messages sent by a viewer subsequent to or concurrent with the occurrence of the event of interest in the video.

21. The method of claim 1 wherein monitoring the instant messages comprises monitoring instant messages that include a frame identifier that identifies a frame of the video.

22. The method of claim 21 wherein monitoring instant messages that include a frame identifier comprises monitoring instant messages that include a frame identifier that identifies the frame displayed to a viewer of the video at the time that the viewer initiates inputting message text to be subsequently sent as an instant message by a client system.

23. The method of claim 21 wherein monitoring instant messages that include a frame identifier comprises monitoring instant messages that include a frame identifier that identifies the frame displayed to a viewer of the video at the time that the viewer requests delivery of a message previously inputted by the viewer, the message being delivered as an instant message from a client system across a data network in response to the request.

24. The method of claim 1 wherein monitoring the instant messages sent by viewers comprises monitoring instant messages that include a video identifier that identifies the video.

25. The method of claim 24 wherein monitoring instant messages sent by viewers comprises receiving instant messages sent by a viewer.

26. The method of claim 25,

further comprising determining a category for the video identifier and filtering, based on the determined category, the monitored instant messages to identify instant messages having content relating to the video, and

wherein aggregating instant messages sent by viewers comprises aggregating the identified instant messages having content relating to the video.

27. The method of claim 26 wherein receiving instant messages sent by a viewer comprises receiving text instant messages and filtering the received instant messages based on the determined category comprises:

determining one or more keywords associated with the category;

analyzing text of a particular received instant message;

determining whether the analyzed text includes the one or more keywords;

storing the text of the particular received instant message and a frame identifier that is associated with the particular received instant message conditioned on the analyzed text including the one or more keywords; and

discarding the particular received instant message conditioned on the analyzed text not including the one or more keywords.

28. A computer program product stored on a computer-readable storage medium that includes instructions that, when executed by a processor, cause the processor to:

monitor and aggregate instant messages sent by viewers of a video while the video is perceivable to the viewers;

use the aggregated instant messages to identify an event of interest in the video and to segment the video to generate a video scene containing the event of interest;

generate metadata corresponding to the video scene;

enable later retrieval of the metadata by a user; and

enable the user to access and perceive the video scene based on the metadata.

29. A system comprising:

means for monitoring and aggregating instant messages sent by viewers of a video while the video is perceivable to the viewers;

means for using the aggregated instant messages to identify an event of interest in the video and to segment the video to generate a video scene containing the event of interest;

means for generating metadata corresponding to the video scene;

means for enabling later retrieval of the metadata by a user; and

means for enabling the user to access and perceive the video scene based on the metadata.