System and method for inserting a description of images into audio recordings

• US 7,996,227 B2
• Filed: 10/03/2007
• Issued: 08/09/2011
• Est. Priority Date: 11/09/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Zhang, Yu, Boyle, Peter C.
• Primary Examiner(s): ALBERTALLI, BRIAN LOUIS
• Application Number:
  US11/866,495
• Publication Number:
  US 20080114601A1
• Time in Patent Office: 1,406 Days
• Field of Search: None
• US Class Current: 704/270
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G11B 27/105 of operating discs

Abstract

There is disclosed a system and method for interpreting and describing graphic images. In an embodiment, the method of inserting a description of an image into an audio recording includes: interpreting an image and producing a word description of the image including at least one image keyword; parsing an audio recording into a plurality of audio clips, and producing a transcription of each audio clip, each audio clip transcription including at least one audio keyword; calculating a similarity distance between the at least one image keyword and the at least one audio keyword of each audio clip; and selecting the audio clip transcription having a shortest similarity distance to the at least one image keyword as a location to insert the word description of the image. The word description of the image can then be appended to the selected audio clip to produce an augmented audio recording including the interpreted word description of the image.

Claims

1. A method of inserting a description of an image into an audio recording, comprising:

interpreting, using a computer device, an image, including;

applying a template to the image to extract file content;

constructing a logical structure from the image;

populating the logical structure with the extracted file content; and

producing a word description of the image including at least one image keyword from the interpreted image;

parsing, using the computer device, an audio recording into a plurality of audio clips;

producing a transcription of each audio clip;

extracting a plurality of noun phrases from the transcription;

calculating an importance value for each noun phrase and identifying at least one audio keyword based upon the importance value;

calculating, using the computer device, a similarity distance between the at least one image keyword and the at least one audio keyword of each audio clip; and

selecting, using the computer device, the audio clip transcription having a shortest similarity distance to the at least one image keyword as a location to insert the word description of the image.

2. The method of claim 1, further comprising:

appending the word description of the image to the selected audio clip to produce an augmented audio recording including at least one interpreted word description of an image.

3. The method of claim 1, further comprising:

providing at least one template to interpret the image, the at least one template including at least one image interpretation component to produce a word description of the image.

4. The method of claim 3, further comprising:

providing at least one of optical character recognition technology, edge finding technology, color edge finding technology, curve finding technology, shape finding technology, and contrast finding technology as an image interpretation component in the at least one template.

5. The method of claim 1, further comprising:

parsing the audio recording into a plurality of audio clips of substantially a same length, and

adjusting the length of each audio clip to end at a natural pause in speech.

6. The method of claim 1, further comprising:

calculating the similarity distance between the image and an audio clip by calculating the similarity distance between at least one image keyword of an image and the at least one audio keyword of an audio clip.

7. The method of claim 6, further comprising:

obtaining the similarity distance between the at least one image keyword and the at least one audio keyword by calculating a path length between the at least one image keyword and the at least one audio keyword in a hierarchical semantic electronic dictionary.

8. A system for inserting a description of an image into an audio recording, comprising:

a computer hardware device, including;

an interpreting system for interpreting an image, including;

applying a template to the image to extract file content;

constructing a logical structure from the image;

populating the logical structure with the extracted file content; and

producing a word description of the image including at least one image keyword from the interpreted image;

a parsing system for parsing an audio recording into a plurality of audio clips for producing a transcription of each audio clip and for extracting a plurality of noun phrases from the transcription;

a calculating system for calculating an importance value for each noun phrase and identifying at least one audio keyword based upon the importance value; and

calculating a similarity distance between the at least one image keyword and the at least one audio keyword of each audio clip; and

a selecting system for selecting the audio clip transcription having a shortest similarity distance to the at least one image keyword as a location to insert the word description of the image.

9. The system of claim 8, further comprising:

an appending system for appending the word description of the image to the selected audio clip to produce an augmented audio recording including at least one interpreted word description of an image.

10. The system of claim 8, further comprising:

at least one template to interpret the image, the at least one template including at least one image interpretation component to produce a word description of the image.

11. The system of claim 10, further comprising:

at least one of optical character recognition technology, edge finding technology, color edge finding technology, curve finding technology, shape finding technology, and contrast finding technology as an image interpretation component in the at least one template.

12. The system of claim 8, wherein the parsing system is configured to parse the audio recording into a plurality of audio clips of substantially a same length, and to adjust the length of each audio clip to end at a natural pause in speech.

13. The system of claim 8, wherein the calculating system is configured to calculate the similarity distance between the image and an audio clip by calculating the similarity distance between at least one image keyword of an image and the at least one audio keyword of an audio clip.

14. The system of claim 13, wherein the calculating system is configured to calculate the similarity distance between the at least one image keyword and the at least one audio keyword based a path length between the at least one image keyword and the at least one audio keyword in a hierarchical semantic electronic dictionary.

15. A program product stored on a non-transitory computer readable medium, which when executed, inserts a description of an image into an audio recording, the non-transitory computer readable medium comprising program code for:

interpreting an image, including;

applying a template to the image to extract file content;

constructing a logical structure from the image;

populating the logical structure with the extracted file content; and

producing a word description of the image including at least one image keyword from the interpreted image;

parsing an audio recording into a plurality of audio clips;

producing a transcription of each audio clip;

extracting a plurality of noun phrases from the transcription;

calculating an importance value for each noun phrase and identifying at least one audio keyword based upon the importance value;

calculating a similarity distance between the at least one image keyword and the at least one audio keyword of each audio clip; and

selecting the audio clip transcription having a shortest similarity distance to the at least one image keyword as a location to insert the word description of the image.

16. The program product of claim 15, further comprising program code for:

appending the word description of the image to the selected audio clip to produce an augmented audio recording including at least one interpreted word description of an image.

17. The program product of claim 15, further comprising program code for:

providing at least one template to interpret the image, the at least one template including at least one image interpretation component to produce a word description of the image.

18. The program product of claim 17, further comprising code for:

providing at least one of optical character recognition technology, edge finding technology, color edge finding technology, curve finding technology, shape finding technology, and contrast finding technology as an image interpretation component in the at least one template.

19. The program product of claim 15, further comprising program code for:

parsing the audio recording into a plurality of audio clips of substantially a same length, and adjusting the length of each audio clip to end at a natural pause in speech.

20. The program product of claim 15, further comprising program code for:

calculating the similarity distance between the image and an audio clip by calculating the similarity distance between at least one image keyword of an image and the at least one audio keyword of an audio clip.

21. The program product of claim 20, further comprising program code for:

obtaining the similarity distance between the at least one image keyword and the at least one audio keyword by calculating a path length between the at least one image keyword and the at least one audio keyword in a hierarchical semantic electronic dictionary.