Systems and methods for augmenting audio/visual broadcasts with annotations to assist with perception and interpretation of broadcast content

• US 8,201,080 B2
• Filed: 05/24/2006
• Issued: 06/12/2012
• Est. Priority Date: 05/24/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Basson, Sara H., Faisman, Alexander, Kanevsky, Dimitri
• Primary Examiner(s): Rutledge, Amelia
• Assistant Examiner(s): Tran, Quoc A
• Application Number:
  US11/439,806
• Publication Number:
  US 20070277092A1
• Time in Patent Office: 2,211 Days
• Field of Search: 715/230, 715/233
• US Class Current: 715/230
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

Systems and methods are provided for augmenting audio/visual broadcasts (e.g., movies) with non-textual annotations that are designed to assist an individual to interpret or otherwise understand the content of the audio/visual broadcast. For example, the non-textual annotations can be in the form of visual cues, audio cues, tactile cues, olfactory cues, and/or other types of non-textual cues that assist persons with certain types of sensory deficiencies or cognitive disabilities to understand and interpret the content of a broadcast.

Claims

1. A method for annotating a content data stream, comprising:

receiving a content data stream;

annotating a portion of an image representing a person in the content data stream with metadata that symbolically describes emotional information conveyed by the person, wherein the portion is a body part of the person; and

outputting the content data stream augmented with non-textual annotations of the portion of the image representing the person having a requested modality for the interpretation or cognition of the annotated content, wherein the emotion information is conveyed by both the portion of the image and the non-textual annotations individually,

wherein the non-textual annotations are configured to be detected and processed by a personal annotation output device receiving the outputted content data stream to output the portion of the image and output the non-textual annotations in the requested modality for the interpretation or cognition of the annotated content, wherein the requested modality is transmitted by the personal annotation output device.

2. The method of claim 1, wherein annotating comprises:

identifying a plurality of segments of the content data stream that are representative of a type of information to be annotated; and

labeling said identified segments with said metadata that symbolically represents said type of information.

3. The method of claim 2, wherein identifying is performed automatically.

4. The method of claim 1, further comprising rendering the non-textual annotations using a personal annotation output device which generates and outputs the non-textual annotations to an individual using said device.

5. The method of claim 1, further comprising rendering the non-textual annotations as visual annotations.

6. The method of claim 1, wherein the metadata symbolically represents an emotional level of the emotional information.

7. The method of claim 6, wherein the symbolic representations of emotion are rendered as non-textual annotations in the form of colored objects, wherein different emotions are symbolically represented by different colors.

8. The method of claim 7, wherein the symbolic representations of emotional level of the emotional information are rendered as non-textual annotations in the form of color intensity of the color associated with the emotion.

9. A method for annotating a content data stream, the method comprising:

receiving, by a processor, a content data stream;

annotating, by a processor, content of the content data stream with metadata that symbolically describes information present in the content;

outputting, by a processor, the content of the content data stream in a modality that is directly perceptible by an individual; and

processing, by a processor, the annotated content to output a metadata stream in synchronization with the output content data stream, wherein the metadata data stream is output in a form that is not directly perceptible by an individual, but can be detected and processed using a personal annotation output device to output non-textual annotations in a modality requested for the interpretation or cognition of the annotated content, wherein the requested modality is specified by a receiver of the outputted content data stream,

wherein annotating the content of the content data stream with metadata that symbolically describes information present in the content further comprises;

identifying a segment of the content, wherein the content is an audio/visual data stream;

identifying a region of the segment to be annotated, wherein the region is a portion of an image of the segment; and

annotating the region of the segment of the content data stream with metadata that symbolically represents emotional informational content of the region, wherein the metadata is associated with the region and is not associated with another portion of the image of the segment.

10. The method of claim 9, further comprising:

transmitting the metadata stream to the personal annotation output device;

rendering the metadata stream by the personal annotation output device to generate non-textual annotations in one or more modalities supported by said device;

outputting the non-textual annotations to a user of said device.

11. The method of claim 9, further comprising transmitting a request for personalized metadata for the content data stream, which can be rendered for a target modality to non-textually annotate the content data stream.

12. The method of claim 11, further comprising generating the personalized metadata stream in real time when the content data stream is output for viewing.

13. The method of claim 10, further comprising automatically identifying a type of non-textual annotation required by the user of said personal annotation output device.