Intelligent determination of replays based on event identification

• US 9,484,065 B2
• Filed: 10/15/2010
• Issued: 11/01/2016
• Est. Priority Date: 10/15/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: McDowell, Brian, Paulus, Owen, Jurenka, David, Alvarado, Ivann
• Primary Examiner(s): Tran, William
• Application Number:
  US12/905,880
• Publication Number:
  US 20120093481A1
• Time in Patent Office: 2,209 Days
• Field of Search: 386/241
• US Class Current: 1/1
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/322 used signal is digitally coded
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/8455 involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

A system for intelligently determining replay locations in a multimedia content stream based on identifying events in the multimedia content stream is provided. In one embodiment, events in the multimedia content stream are identified by analyzing information in the multimedia content stream, in real time. In another embodiment, events in the multimedia content stream are identified by analyzing the viewing behavior and an emotional response from users viewing the multimedia content, in real time. One or more replay locations in the multimedia content stream are determined based on the events identified in the multimedia content stream. The multimedia content stream with the replay locations are displayed to a user via a user interface in the user'"'"'s processing device.

Claims

1. A method for marking one or more content segments from a distributed multimedia content stream as replayable segments of interest, the method comprising:

identifying a multimedia content stream that has been distributed to, and has had at least part of its content played to a plurality of users who are not all local to one another and who are respectively monitored by respective ones of remote from one another and network connected capture devices;

receiving by way of a network and from the remote-from-one-another capture devices of a plurality of the monitored users, feedback information related to the identified multimedia content stream, the received feedback information indicating one or more of determined emotional responses and one or more of determined stream presentation control actions of the monitored users and timings of behaviors associated with those determined emotional responses and with those stream presentation control actions of the monitored users, wherein the stream presentation control actions of each respectively monitored user control how the identified multimedia content stream is presented to the respectively monitored user;

based on the received feedback information received way of the network and from the remote-from-one-another capture devices of the monitored users, using the indicated timings of the received feedback information to identify one or more segments in the content of the identified multimedia content stream corresponding to the one or more of the determined emotional responses and the one or more of the determined stream presentation control actions of the plurality of the monitored users; and

based on the identifying of the one or more segments, generating a marked multimedia content stream having content of the identified multimedia content stream, the marked stream including marked and replayable content whose respective marking denotes the corresponding content as having at least one content segment corresponding to the detected one or more of the emotional responses and the one or more of the determined stream presentation control actions of the plurality of the monitored users;

wherein;

the capture devices are configured to detect behaviors of their respective users and to determine therefrom a corresponding one or more of the emotional responses and stream control actions of their monitored users;

the capture devices are configured to respond to the stream presentation control actions of their monitored users by controlling at least one of time of playing to their respective users of content portions of the distributed and at least partly played multimedia content stream, including by means of play time control through rewinding and replaying of content, fast forwarding through content and stopping or pausing a playing of content;

the capture devices are further configured to provide identifying information about the respective content portion played to the respective users at the time of their respectively detected behaviors; and

the identifying of the one or more segments based on the received feedback information comprises;

analyzing received user-feedback information related to play time control by the one or more users to thereby identify for the marking of the replayable content, that content which is related to one or more of a pause, forward, rewind, jump or a stop control behaviors detected by the capture devices.

2. The method of claim 1, wherein in addition to the identifying of the one or more segments based on the received feedback information, the method comprises identifying within the content of the obtained multimedia content stream at least one of imagery and audio present therein which can be designated as a within-stream event of possible interest and wherein the identifying of one of imagery and audio comprises:

detecting at least one of an audio cue or a visual cue in the obtained multimedia content stream to identify the corresponding event of possible interest.

3. The method of claim 2, wherein the audio cue or the visual cue comprises at least one of an applause, a cheer, a remark, a score update or a commentary update in the multimedia content stream.

4. The method of claim 1, wherein among the behaviors the capture devices are configured to detect for thereby determining one or more emotional responses of the one or more users to whom the multimedia content is played, are:

one or more of facial expressions, vocal responses, gestures, and movements performed by the respectively monitored users.

5. The method of claim 1, wherein the generating of the marked multimedia content stream comprises:

determining the time of occurrence within the marked multimedia content stream of a point of interest where the point of interest corresponds to a content segment associated with at least one of the detected emotional responses and detected stream presentation control actions, and

determining the duration of the point of interest or the type of the point of interest.

6. The method of claim 1, wherein the generating of the marked multimedia content stream comprises:

embedding into the marked multimedia content stream as metadata, information about the marked and replayable content.

7. The method of claim 6, wherein the embedded metadata information comprises:

information about one or more events of interest identified in the multimedia content stream, the time of occurrence of the one or more events, a point of interest related to the one or more events, the time of occurrence of the point of interest, a start time of a respective marked and replayable content portion and the duration of time to replay the respective marked and replayable content portion.

8. One or more processor readable storage devices having processor readable code embodied on said one or more processor readable storage devices, the processor readable code being configured to be read by the one or more processors, executed by the one or more processors and thus cause the one or more processors to perform a method comprising:

identifying a multimedia content stream that has been distributed to, and has had at least part of its content played to a plurality of users who are not all local to one another and who are respectively monitored by respective ones of remote from one another and network connected capture devices;

receiving by way of a network and from the remote from one another capture devices of a plurality of the monitored users, respective feedback information related to the identified multimedia content stream, the received feedback information indicating one or more of determined emotional responses and indicating one or more of determined stream presentation control actions of the respective monitored users and timings of the behaviors associated with those determined emotional responses and with those stream presentation control actions, wherein the latter stream presentation control actions of each respectively monitored user control how the identified multimedia content stream is presented to the respectively monitored user;

based on the received feedback information received way of the network and from the remote from one another capture devices of the monitored users, identifying one or more segments in the content of the identified multimedia content stream which correspond to the one or more determined emotional responses and to the one or more determined stream presentation control actions of the plurality of the monitored users; and

based on the identifying of the one or more segments, generating a marked multimedia content stream, the marked stream having content of the identified multimedia content stream including marked and replayable content portions whose respective marking denotes the corresponding content as having at least one content segment corresponding to the determined emotional responses and the determined stream presentation control actions of the plurality of the monitored users;

wherein;

the capture devices are configured to detect behaviors of their respective users and to determine therefrom a corresponding one or more of the emotional responses and stream control actions of their monitored users;

the capture devices are configured to respond to the stream presentation control actions of their monitored users by controlling at least one of time of playing to their respective users of content portions of the distributed and at least partly played multimedia content stream, including by means of play time control through rewinding and replaying of content, fast forwarding through content and stopping or pausing a playing of content;

the capture devices are further configured to provide identifying information about the respective content portion played to the respective users at the time of their respectively detected behaviors; and

the identifying of the one or more segments based on the received feedback information comprises;

analyzing received user-feedback information related to play time control by the one or more users to thereby identify for the marking of the replayable content, that content which is related to one or more of a pause, forward, rewind, jump or a stop control behaviors detected by the capture devices.

9. One or more processor readable storage devices of claim 8, wherein the performed method further comprises:

receiving an input from the one or more users to replay the marked multimedia content stream; and

replaying one or more of the marked and replayable content portions whose respective markings denote the content portions as having at least one content segment corresponding to the detected emotional responses and stream presentation control actions.

10. One or more processor readable storage devices of claim 8, wherein the step in the performed method of identifying the one or more segments comprises:

determining if the one or more emotional responses and stream presentation control actions attributed to the monitored users are those from at least a predetermined threshold percentage of the monitored users.

11. An apparatus, comprising:

a computing device, including one or more processors and a memory accessible by the one or more processors, the computing device being communicatively coupled over a network to a plurality of respective capture devices, at least two of the capture devices being remote from one another;

the one or more processors of the computing device being configured to identify and obtain a multimedia content stream that has been distributed to, and has had at least part of its content played to a plurality of users who are monitored by respective ones of the capture devices;

the one or more processors of the computing device being further configured to identify, based on received feedback information from the capture devices whose users have had at least part of the content played to them, one or more segments in the content of the identified multimedia content stream which correspond to one or more determined emotional responses and to one or more determined stream presentation control actions of two or more monitored users, wherein the latter stream presentation control actions control how the identified multimedia content stream is respectively presented at respectively remote presentation locations to the respective and there monitored users; and

the one or more processors of the computing device being further configured to generate a marked multimedia content stream having content of the identified multimedia content stream including marked and replayable content portions whose respective markings denote the content portions as having at least one content segment corresponding to the detected emotional responses and to the detected stream presentation control actions of the two or more monitored users;

wherein;

the capture devices are configured to detect behaviors of their respective users and to determine therefrom a corresponding one or more of the emotional responses and stream presentation control actions of their monitored users;

the capture devices are configured to respond to the stream presentation control actions of their monitored users by controlling at least one of time of playing to their respective users of content portions of the distributed and at least partly played multimedia content stream, including by means of play time control through rewinding and replaying of content, fast forwarding through content and stopping or pausing a playing of content;

the capture devices are further configured to provide identifying information about the respective content portion played to the respective users at the time of their respectively detected behaviors; and

the identifying of the one or more segments based on the received feedback information comprises analyzing received user-feedback information related to play time control by the one or more users to thereby identify for the marking of the replayable content, that content which is related to one or more of a pause, forward, rewind, jump or a stop control behaviors detected by the capture devices.