Context relevant interactive television

• US 9,866,915 B2
• Filed: 05/18/2012
• Issued: 01/09/2018
• Est. Priority Date: 11/28/2011
• Status: Expired due to Fees

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Excalibur IP, LLC (Acacia Research Corporation)
• Inventors: Urrabazo, Roger, Rice, Dave, La, John, Glidden, Todd R., Raman, Vishwas
• Primary Examiner(s): Saltarelli, Dominic D
• Application Number:
  US13/475,910
• Publication Number:
  US 20130139209A1
• Time in Patent Office: 2,062 Days
• Field of Search: 725 32- 36, 725 18, 725 87
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739

Abstract

Methods, systems and computer readable medium with program instructions for providing contextually relevant information for multimedia content streaming through a rendering device, includes detecting a selection of the multimedia content for rendering on the rendering device. An audio signal of the multimedia content is retrieved and analyzed to generate an audio fingerprint. The generated audio fingerprint is used to identify the multimedia content that is selected for rendering. Contextually relevant content for a portion of the multimedia content being rendered on the rendering device is dynamically retrieved from one or more sources and mapped to data associated with the portion of the multimedia content. The mapping results in the rendering of the relevant content during the rendering of the portion of the multimedia content and provides additional information that is contextually relevant to the portion of the multimedia content that is currently being rendered on the rendering device.

Claims

1. A method for providing contextually relevant information for multimedia content received at a rendering device, comprising:

detecting a selection of the multimedia content for rendering on the rendering device;

retrieving an audio signal of the multimedia content, the audio signal of a particular segment of the multimedia content is analyzed to generate an audio fingerprint, wherein the audio fingerprint is defined by a vector of data points;

identifying the multimedia content by matching the vector defining the audio fingerprint of the particular segment with a corresponding vector of the multimedia content;

identifying a plurality of trigger events that are likely to occur within the currently rendering multimedia content by analyzing at least part of the multimedia content, the analyzing is based on accumulated status leading up to a current context of a portion of the multimedia content currently being rendered on the rendering device;

dynamically retrieving, in substantial real time, content that is contextually relevant to specific ones of the plurality of trigger events that are likely to occur, from one or more content sources; and

mapping at least a portion of the content retrieved from the one or more sources to a specific portion of the multimedia content currently being rendered, the mapping causes the retrieved content to be provided for rendering when a particular one of the plurality of trigger events associated with the retrieved content occurs within the specific portion of the multimedia content, wherein the content retrieved from the one or more sources is additional information that is contextually relevant to the particular trigger event occurring in the specific portion of the multimedia content that is currently rendering on the rendering device, wherein method operations are performed by a processor.

2. The method of claim 1, wherein identifying the multimedia content further includes,

analyzing the particular segment to identify acoustic modulations in the particular segment, the analysis generating the vector of data points for the particular segment based on the acoustic modulation, the vector defining the audio fingerprint that is unique for the particular segment of the audio signal;

and

querying a content database on a server using the vector of the particular segment of audio signal to obtain content information for multimedia content that matches the audio fingerprint of the particular segment, the content information obtained from a content provider provides details related to the multimedia content that matches the audio signal.

3. The method of claim 1, wherein mapping further includes

providing a video offset at the specific portion of the multimedia content where the particular trigger event occurs based on context, wherein the video offset is defined by temporally tagging the specific portion of the multimedia content;

and

mapping the content retrieved from the one or more sources to the video offset at the specific portion of the multimedia content.

4. The method of claim 3, further includes storing the retrieved content from the one or more sources in a content repository for subsequent retrieval and rendering.

5. The method of claim 4, wherein the subsequent retrieval of content from the content repository further includes,

determining age of the content stored in the content repository for the particular trigger event;

when the age of the content exceeds a pre-defined threshold value,

automatically discarding the content from the content repository.

6. The method of claim 3, further includes,

providing an indicator at the video offset for the content from the one or more sources, the indicator indicative of presence of additional contextually relevant information for the specific portion of the multimedia content.

7. The method of claim 3, wherein the trigger event is a predefined event for the multimedia content.

8. The method of claim 1, wherein when there is no content available from the one or more sources for the portion of the multimedia content, providing a default content for the portion of the multimedia content based on the context of the multimedia content.

9. The method of claim 1, further includes,

detecting a switch in the multimedia content rendered on the rendering device;

automatically discarding content retrieved from the one or more sources and stored in a content repository for the multimedia content rendered on the rendering device; and

dynamically retrieving content from the one or more sources for newly switched multimedia content based on context and audio fingerprint generated for the newly switched multimedia content.

10. A method for providing contextually relevant information for multimedia content received at a rendering device, comprising:

detecting a selection of the multimedia content for rendering on the rendering device, wherein the multimedia content selected is a pre-recorded content;

retrieving an audio signal of the multimedia content, the audio signal of a particular segment of the multimedia content is analyzed to generate an audio fingerprint, wherein the audio fingerprint is defined by a vector of data points;

identifying the multimedia content by matching the vector defining the audio fingerprint of the particular segment with a corresponding vector of the multimedia content;

identifying a plurality of trigger events that have occurred within the multimedia content currently rendering on the rendering device;

dynamically retrieving, in substantial real time, content that is contextually relevant to a specific portion of the identified multimedia content currently rendering on the rendering device, from a content repository, wherein the content repository includes content collected from a plurality of sources for the specific ones of the plurality of trigger events that have occurred within the specific portion of the multimedia content and at least a portion of the content mapped to the specific portion of the multimedia content corresponding to at least a particular one of the plurality of trigger events, the mapping performed based on analysis of at least part of the multimedia content, wherein the analysis identifies accumulated status leading up to a current context of the specific portion of the multimedia content currently rendered on the rendering device in which the particular trigger event occurred, the mapping of the retrieved content is performed in substantial real time based on the current context of the multimedia content so as to present recent content for the specific portion of the multimedia content; and

providing the content retrieved from the content repository for rendering during the occurrence of the particular trigger event within the specific portion of the multimedia content, wherein the content retrieved from the content repository is additional information that is contextually relevant to the specific portion of the multimedia content that is currently rendered on the rendering device,

wherein method operations are performed by a processor.

11. The method of claim 10, wherein identifying the multimedia content further includes,

analyzing the particular segment to identify acoustic modulations in the particular segment, the analysis generating the vector of data points for the particular segment based on the acoustic modulation, the vector defining the audio fingerprint that is unique for the particular segment of the audio signal;

and

querying a content database on a server using the vector of the particular segment of audio signal to obtain content information for multimedia content that matches the audio fingerprint of the particular segment, the content information obtained from a content provider provides details related to the multimedia content matching the audio signal.

12. The method of claim 10, wherein mapping further includes,

identifying the particular trigger event using a video offset defined by a temporal tag at the specific portion of the multimedia content that is currently rendered on the rendering device;

and

retrieving the content mapped to the specific portion of the multimedia content from the content repository based on the video offset.

13. The method of claim 12, further includes,

dynamically retrieving additional content related to the particular trigger event from one or more sources;

adding the additional content to the content retrieved for the particular trigger event from the content repository to generate composite content; and

rendering the composite content on the rendering device during the rendering of the specific portion of the multimedia content in which the particular trigger event occurs, the composite content providing updated information related to the particular trigger event at the specific portion of the multimedia content.

14. The method of claim 10, wherein retrieving content from the content repository further includes,

analyzing the content to determine age of the content;

when the age of the content exceeds a pre-defined threshold value,

updating the content associated with the specific portion of the multimedia content in the content repository, the content for updating retrieved from the one or more sources based on the context of the multimedia content, the age of the content in the content repository is updated to reflect time of the update.

15. The method of claim 10, wherein the content retrieved from the plurality of sources includes any one or combination of video, audio, image, or promotional media related to the particular trigger event.

16. A method for providing contextually relevant information for multimedia content received at a rendering device, comprising:

detecting a selection of the multimedia content for rendering on the rendering device, the multimedia content selected is a pre-recorded content;

retrieving an audio signal of the multimedia content, the audio signal of a particular segment of the multimedia content is analyzed to generate an audio fingerprint, wherein the audio fingerprint is defined by a vector of data points;

identifying the multimedia content using the audio fingerprint, the multimedia content identified by matching the vector for the particular segment with a corresponding vector of the multimedia content;

identifying a plurality of trigger events that have occurred within the multimedia content currently rendering on the rendering device;

retrieving, in substantial real time, content that is contextually relevant to specific portions of the identified multimedia content currently being rendered on the rendering device, from a content repository and dynamically from one or more sources, wherein the content repository includes content collected from a plurality of sources for the specific ones of the plurality of trigger events that have occurred during prior rendering of the multimedia content and at least a portion of the content retrieved from the content repository is mapped to a specific portion of the multimedia content corresponding to at least a particular one of the plurality of trigger events, the mapping based on analysis of at least part of the multimedia content, the analysis identifying accumulated status leading up to a current context of the specific portion of the multimedia content currently being rendered in which the particular trigger event occurred, the mapping of the retrieved content is performed in substantial real time based on the current context of the multimedia content so as to present recent content for the specific portion of the multimedia content; and

providing the retrieved content for rendering during current rendition of the specific portion of the multimedia content in which the particular trigger event occurs, wherein the retrieved content is additional information that is contextually relevant to the particular trigger event occurring in the specific portion of the multimedia content that is currently rendering on the rendering device,

wherein method operations are performed by a processor.

17. The method of claim 16, further includes,

mapping the dynamically retrieved content from the one or more sources to data associated with the particular trigger event occurring within the specific portion of the multimedia content currently rendering on the rendering device, the mapping results in updating the content stored in the content repository for the particular trigger event occurring within the portion of the multimedia content with the retrieved content.