Identifying events of interest within video content

• US 7,624,416 B1
• Filed: 10/04/2006
• Issued: 11/24/2009
• Est. Priority Date: 07/21/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: AOL LLC (Apollo Global Management, Inc.)
• Inventors: Wiegering, Anthony, Vandermolen, Harmannus, Franco, Alexander, Levine, Scott J., Sommers, Michael
• Primary Examiner(s): Miller, John W
• Assistant Examiner(s): McGahey, Christopher S
• Application Number:
  US11/538,620
• Time in Patent Office: 1,147 Days
• Field of Search: 725 1-153, 709201-253
• US Class Current: 725/109
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: H04H 60/33 Arrangements for monitoring the users' behaviour or opinions
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/73 using meta-information
• CPC Class Codes: H04N 21/2408 Monitoring of the upstream path of the transmission network, e.g. client requests monitoring data switching networks utilization H04L43/0876; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/25866 Management of end-user data customer care in data networks H04L41/5077
• CPC Class Codes: H04N 21/26603 for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8405 represented by keywords
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

Instant messages sent by a viewer of a video while the video is perceivable to the viewer may be monitored. An event of interest occurring in the video may be identified based on the monitored instant messages. Metadata that describes the event of interest may be generated and assigned to the video. A request for video content associated with the video may be received from a user. An indexed collection of videos that includes the video may be searched for the requested video content. Based on the search, it may be determined that the video satisfies the user'"'"'s request. The event of interest occurring in the video may be identified based on the metadata assigned to the video. User perception of a scene from the video that includes the event of interest may be enabled.

Claims

1. A method comprising:

one or more processors monitoring instant messages sent by viewers of a video while the video is perceivable to the viewers;

filtering the monitored instant messages to identify instant messages having message content relating to the video;

estimating a duration of time that viewers are typically expected to discuss, through instant messaging, an event of interest that occurs in the video while perceiving the video;

determining a video category corresponding to the video;

based on the determined video category, anticipating an event of interest within the video;

determining an amount of time associated with the anticipated event of interest;

accessing a timeline of the video;

selecting a particular point in the timeline of the video;

determining a window of time that begins at the particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging the event of interest;

determining an aggregate number of the identified instant messages that are received during the window of time;

accessing a threshold;

comparing the aggregate number to the threshold;

conditioned on the aggregate number of the identified instant messages meeting the threshold, determining that the window of time corresponds to occurrence of an event of interest in the video, and calculating, based on the particular point in the timeline of the video and based on the determined amount of time associated with the anticipated event of interest, a start time and an end time of a scene of the video in which the event of interest occurs;

generating metadata that specifies the start time and the end time of the scene of the video in which the event of interest occurs; and

assigning the metadata to the video.

2. The method of claim 1 wherein monitoring the instant messages sent by a viewer comprises monitoring instant messages sent by a viewer of a video to another viewer of the video while both viewers are concurrently viewing the video.

3. The method of claim 1 wherein monitoring the instant messages sent by a viewer comprises monitoring instant messages sent by a viewer of the video to a user who is not viewing the video.

4. The method of claim 1 wherein monitoring the instant messages sent by a viewer comprises monitoring the instant messages sent by a viewer subsequent to or concurrent with the occurrence of the event of interest in the video.

5. The method of claim 1 wherein monitoring the instant messages comprises monitoring instant messages that include a frame identifier that identifies a frame of the video.

6. The method of claim 5 wherein monitoring instant messages that include a frame identifier comprises monitoring instant messages that include a frame identifier that identifies the frame displayed to a viewer of the video at the time that the viewer initiates inputting message text to be subsequently sent as an instant message by a client system.

7. The method of claim 5 wherein monitoring instant messages that include a frame identifier comprises monitoring instant messages that include a frame identifier that identifies the frame displayed to a viewer of the video at the time that the viewer requests delivery of a message previously inputted by the viewer, the message being delivered as an instant message from a client system across a data network in response to the request.

8. The method of claim 1 wherein monitoring the instant messages sent by a viewer comprises monitoring instant messages that include a video identifier that identifies the video.

9. The method of claim 1 wherein monitoring instant messages sent by a viewer comprises receiving instant messages sent by a viewer.

10. The method of claim 9 wherein filtering the received instant messages based on the video identifier comprises determining a category for the video identifier and filtering based on the determined category.

11. The method of claim 10 wherein receiving instant messages sent by a viewer comprises receiving text instant messages and filtering the received instant messages based on the determined category comprises:

determining one or more keywords associated with the category;

analyzing text of a particular received instant message;

determining whether the analyzed text includes the one or more keywords;

storing the text of the particular received instant message and a frame identifier that is associated with the particular received instant message conditioned on the analyzed text including the one or more keywords; and

discarding the particular received instant message conditioned on the analyzed text not including the one or more keywords.

12. The method of claim 1 further comprising storing the aggregate number of identified instant messages that are received during the window of time with an indication of the particular point.

13. The method of claim 1 wherein generating metadata that specifies the start time and the end time of the scene comprises

including the scene start time and the scene end time in the metadata associated with the video, the metadata enabling the user to selectively view the scene in the video.

14. The method of claim 1 wherein calculating the scene start time comprises:

starting from the selected particular point in the timeline of the video, rewinding the video by the amount of time associated with the anticipated event of interest; and

marking the time at which the rewinding stops as the scene start time.

15. The method of claim 1 wherein calculating the scene end time comprises:

starting from the selected particular point in the timeline of the video, fast forwarding the video by a predetermined amount of time; and

marking the time at which the fast forwarding stops as the scene end time.

16. The method of claim 1 wherein determining an aggregate number of the identified instant messages that are received during the window of time comprises:

receiving an indication the video has started and an identification of the viewers of the video;

initializing a video clock;

receiving instant messages sent by the viewers and assigning a time T0 to each instant message, the time T0 being a time of receipt of the instant message as indicated by the video clock;

determining an instant message input probability curve for each identified instant message that has been assigned a time T0 that falls within the window of time; and

combining the instant message input probability curves for the identified instant messages that have been assigned a time T0 that falls within the window of time.

17. The method of claim 16 wherein determining the instant message probability input curve for an identified instant message that has been assigned a time T0 that falls within the window of time comprises:

determining a network delay;

moving backwards from the time T0 by the network delay to a time T−

1;

determining an amount of typing time for the identified instant message;

moving backwards from T−

1 by the amount of typing time to a time T−

2; and

determining an instant message probability input curve, wherein the instant message probability input curve comprises a probability distribution curve centered at T−

2 that reflects a time at which a sender of the instant message began inputting the instant message.

18. The method of claim 1 wherein the instant messages are text-based messages.

19. The method of claim 1 further comprising:

receiving from a user a request for video content associated with the video;

searching an indexed collection of videos, the collection including the video, for the requested video content;

based on the search, determining that the video satisfies the user'"'"'s request;

identifying the event of interest occurring in the video based on the metadata assigned to the video; and

enabling user perception of the scene from the video that includes the event of interest.

20. The method of claim 19 wherein searching an indexed collection of videos for the requested video content comprises:

accessing a collection of videos that have been indexed based on metadata associated with the videos, the metadata including keywords describing the content included within the videos and categories to which the video content belongs;

searching the indexed collection for the video content requested by the user; and

identifying a video from within the collection that satisfies the user'"'"'s request for video content and includes the event of interest.

43. The method of claim 1, further comprising:

selecting a second particular point in the timeline of the video that is subsequent to the particular point;

determining a second window of time that begins at the second particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging, an event of interest;

determining a second aggregate number of the identified instant messages that are received during the second window of time;

comparing the second aggregate number to the threshold;

determining that the second aggregate number of the identified instant messages does not meet the threshold; and

concluding that the second window of time does not correspond to occurrence of an event of interest in the video.

44. The method of claim 43, wherein selecting a second particular point in the timeline of the video comprises:

accessing a slide time increment, and

selecting the second particular point as a point in time corresponding to the particular point fast forwarded by the slide time increment.

45. The method of claim 44, wherein the slide time increment has a smaller duration than the second window of time.

46. The method of claim 44, further comprising:

selecting a third particular point in the timeline of the video that is subsequent to the second particular point;

determining a third window of time that begins at the third particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging, an event of interest;

determining a third aggregate number of the identified instant messages that are received during the third window of time;

comparing the third aggregate number to the threshold;

determining that the third aggregate number of the identified instant messages does not meet the threshold; and

concluding that the third window of time does not correspond to occurrence of an event of interest in the video.

47. The method of claim 46, wherein selecting a third particular point in the timeline of the video comprises:

accessing the slide time increment, and

selecting the third particular point as a point in time corresponding to the second particular point fast forwarded by the slide time increment.

48. The method of claim 1, wherein the amount of time associated with the anticipated event of interest comprises an amount of time reflective of a duration of time between a time when the anticipated event of interest begins and a time when the anticipated event of interest ends.

21. A computer program product stored on a computer-readable storage medium that includes instructions that, when executed by a processor, cause the processor to:

monitor instant messages sent by viewers of a video while the video is perceivable to the viewers;

filter the monitored instant messages to identify instant messages having message content relating to the video;

estimate a duration of time that viewers are typically expected to discuss, through instant messaging, an event of interest that occurs in the video while perceiving the video;

determine a video category corresponding to the video;

based on the determined video category anticipate an event of interest within the video;

determine an amount of time associated with the anticipated event of interest;

access a timeline of the video;

select a particular point in the timeline of the video;

determine a window of time that begins at the particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging the event of interest;

determine an aggregate number of the identified instant messages that are received during the window of time;

access a threshold;

compare the aggregate number to the threshold;

conditioned on the aggregate number of the identified instant messages meeting the threshold, determine that the window of time corresponds to occurrence of an event of interest in the video, and calculate, based on the particular point in the timeline of the video and based on the determined amount of time associated with the anticipated event of interest, a start time and an end time of a scene of the video in which the event of interest occurs;

generate metadata that specifies the start time and the end time of the scene of the video in which the event of interest occurs; and

assign the metadata to the video.

22. The computer program product of claim 21 wherein the instructions that cause the processor to monitor the instant messages sent by a viewer comprise instructions that, when executed, cause the processor to monitor instant messages sent by a viewer of a video to another viewer of the video while both viewers are concurrently viewing the video.

23. The computer program product of claim 21 wherein the instructions that cause the processor to monitor the instant messages sent by a viewer comprise instructions that, when executed, cause the processor to monitor instant messages sent by a viewer of the video to a user who is not viewing the video.

24. The computer program product of claim 21 wherein the instructions that cause the processor to monitor the instant messages sent by a viewer comprise instructions that, when executed, cause the processor to monitor the instant messages sent by a viewer subsequent to or concurrent with the occurrence of the event of interest in the video.

25. The computer program product of claim 21 wherein the instructions that cause the processor to monitor the instant messages comprise instructions that, when executed, cause the processor to monitor instant messages that include a frame identifier that identifies a frame of the video.

26. The computer program product of claim 25 wherein the instructions that cause the processor to monitor instant messages that include a frame identifier comprise instructions that, when executed, cause the processor to monitor instant messages that include a frame identifier that identifies the frame displayed to a viewer of the video at the time that the viewer initiates inputting message text to be subsequently sent as an instant message by a client system.

27. The computer program product of claim 25 wherein the instructions that cause the processor to monitor instant messages that include a frame identifier comprise instructions that, when executed, cause the processor to monitor instant messages that include a frame identifier that identifies the frame displayed to a viewer of the video at the time that the viewer requests delivery of a message previously inputted by the viewer, the message being delivered as an instant message from a client system across a data network in response to the request.

28. The computer program product of claim 21 wherein the instructions that cause the processor to monitor the instant messages sent by a viewer comprise instructions that, when executed, cause the processor to monitor instant messages that include a video identifier that identifies the video.

29. The computer program product of claim 21 wherein the instructions that cause the processor to monitor instant messages sent by a viewer comprise instructions that, when executed, cause the processor to receive instant messages sent by a viewer.

30. The computer program product of claim 29 wherein the instructions that cause the processor to filter the received instant messages based on the video identifier comprise instructions that, when executed, cause the processor to determine a category for the video identifier and filtering based on the determined category.

31. The computer program product of claim 30 wherein the instructions that cause the processor to receive instant messages sent by a viewer comprise instructions that, when executed, cause the processor to receive text instant messages and the instructions that cause the processor to filter the received instant messages based on the determined category comprise instructions that, when executed, cause the processor to:

determine one or more keywords associated with the category;

analyze text of a particular received instant message;

determine whether the analyzed text includes the one or more keywords;

store the text of the particular received instant message and a frame identifier that is associated with the particular received instant message conditioned on the analyzed text including the one or more keywords; and

discard the particular received instant message conditioned on the analyzed text not including the one or more keywords.

32. The computer program product of claim 21 further comprising instructions that, when executed, cause the processor to store the aggregate number of identified instant messages that are received during the window of time with an indication of the particular point.

33. The computer program product of claim 21 wherein the instructions that cause the processor to generate metadata that describes the event of interest comprise instructions that, when executed, cause the processor to

include the scene start time and the scene end time in the metadata associated with the video, the metadata enabling the user to selectively view the scene in the video.

34. The computer program product of claim 21 wherein the instructions that cause the processor to calculate a scene start time comprise instructions that, when executed, cause the processor to,

starting from the selected particular point in the timeline of the video rewind the video by the amount of time associated with the anticipated event of interest;

and

mark the time at which the rewinding stops as the scene start time.

35. The computer program product of claim 21 wherein the instructions that cause the processor to calculate a scene end time comprise instructions that, when executed, cause the processor to:

starting from the selected particular point into the timeline of the video, fast forward the video by a predetermined amount of time; and

mark the time at which the fast forwarding stops as the scene end time.

36. The computer program product of claim 21 wherein the instructions that cause the processor to determine an aggregate number of the identified instant messages that are received during the window of time comprise instructions that, when executed, cause the processor to:

receive an indication the video has started and an identification of the viewers of the video;

initialize a video clock;

receive instant messages sent by the viewers and assigning a time T0 to each instant message, the time T0 being a time of receipt of the instant message as indicated by the video clock;

determine an instant message input probability curve for each identified instant message that has been assigned a time T0 that falls within the window of time; and

combine the instant message input probability curves for the identified instant messages that have been assigned a time T0 that falls within the window of time.

37. The computer program product of claim 36 wherein the instructions that cause the processor to determine the instant message probability input curve for an identified instant message that has been assigned a time T0 that falls within the window of time comprise instructions that, when executed, cause the processor to:

determine a network delay;

move backwards from the time T0 by the network delay to a time T−

1;

determine an amount of typing time for the identified instant message;

move backwards from T−

1 by the amount of typing time to a time T−

2; and

determine an instant message probability input curve, wherein the instant message probability input curve comprises a probability distribution curve over an interval of time centered at T−

2 that reflects a time at which a sender of the instant message began inputting the instant message.

38. The computer program product of claim 21 wherein the instant messages are text-based messages.

39. The computer program product of claim 21 further comprising instructions that, when executed, cause the processor to:

receive from a user a request for video content associated with the video;

search an indexed collection of videos, the collection including the video, for the requested video content;

based on the search, determine that the video satisfies the user'"'"'s request;

identify the event of interest occurring in the video based on the metadata assigned to the video; and

enable user perception of the scene from the video that includes the event of interest.

40. The computer program product of claim 39 wherein the instructions that cause the processor to search an indexed collection of videos for the requested video content comprise instructions that, when executed, cause the processor to:

access a collection of videos that have been indexed based on metadata associated with the videos, the metadata including keywords describing the content included within the videos and categories to which the video content belongs;

search the indexed collection for the video content requested by the user; and

identify a video from within the collection that satisfies the user'"'"'s request for video content and includes the event of interest.

49. The computer program product of claim 21, wherein the instructions further comprise instructions that, when executed by a processor, cause the processor to:

select a second particular point in the timeline of the video that is subsequent to the particular point;

determine a second window of time that begins at the second particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging, an event of interest;

determine a second aggregate number of the identified instant messages that are received during the second window of time;

compare the second aggregate number to the threshold;

determine that the second aggregate number of the identified instant messages does not meet the threshold; and

conclude that the second window of time does not correspond to occurrence of an event of interest in the video.

50. The computer program product of claim 49, wherein the instructions that cause the processor to select a second particular point in the timeline of the video comprise instructions that cause the processor to:

access a slide time increment, and

select the second particular point as a point in time corresponding to the particular point fast forwarded by the slide time increment.

51. The computer program product of claim 50, wherein the slide time increment has a smaller duration than the second window of time.

52. The computer program product of claim 50, wherein the instructions further comprise instructions that, when executed by a processor, cause the processor to:

select a third particular point in the timeline of the video that is subsequent to the second particular point;

determine a third window of time that begins at the third particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging, an event of interest;

determine a third aggregate number of the identified instant messages that are received during the third window of time;

compare the third aggregate number to the threshold;

determine that the third aggregate number of the identified instant messages does not meet the threshold; and

conclude that the third window of time does not correspond to occurrence of an event of interest in the video.

53. The computer program product of claim 52, wherein the instructions that cause the processor to select a third particular point in the timeline of the video comprise instructions that cause the processor to

access the slide time increment, and

select the third particular point as a point in time corresponding to the second particular point fast forwarded by the slide time increment.

54. The computer program product of claim 21, wherein the amount of time associated with the anticipated event of interest comprises an amount of time reflective of a duration of time between a time when the anticipated event of interest begins and a time when the anticipated event of interest ends.

41. A system comprising:

means for monitoring instant messages sent by viewers of a video while the video is perceivable to the viewers;

means for filtering the monitored instant messages to identify instant messages having message content relating to the video;

means for estimating a duration of time that viewers are typically expected to discuss, through instant messaging, an event of interest that occurs in the video while perceiving the video;

means for determining a video category corresponding to the video;

means for anticipating, based on the determined video category an event, of interest within the video;

means for determining an amount of time associated with the anticipated event of interest;

means for accessing a timeline of the video;

means for selecting a particular point in the timeline of the video;

means for determining a window of time that begins at the particular point and has a duration corresponding to the estimated duration of time that viewers are typically expected to discuss, through instant messaging, the event of interest;

means for determining an aggregate number of the identified instant messages that are received during the window of time;

means for accessing a threshold;

means for comparing the aggregate number to the threshold;

means for, conditioned on the aggregate number of the identified instant messages meeting the threshold, determining that the window of time corresponds to occurrence of an event of interest in the video, and calculating, based on the particular point in the timeline of the video and based on the determined amount of time associated with the anticipated event of interest, a start time and an end time of a scene of the video in which the event of interest occurs;

means for generating metadata that specifies the start time and the end time of the scene of the video in which the event of interest occurs; and

means for assigning the metadata to the video.

42. The system of claim 41 further comprising:

means for receiving from a user a request for video content associated with the video;

means for searching an indexed collection of videos, the collection including the video, for the requested video content;

based on the search, means for determining that the video satisfies the user'"'"'s request;

means for identifying the event of interest occurring in the video based on the metadata assigned to the video; and

means for enabling user perception of the scene from the video that includes the event of interest.