Tool for automatically mapping multimedia annotations to ontologies

• US 20030061028A1
• Filed: 09/21/2001
• Published: 03/27/2003
• Est. Priority Date: 09/21/2001
• Status: Abandoned Application

• Current Assignee: Knumi, Inc.
• Original Assignee: Knumi, Inc.
• Inventors: Dey, Jayanta K., Sivasankaran, Rajendran M.
• Application Number:
  US09/956,889
• Publication Number:
  US 20030061028A1
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70

Abstract

A tool for learning to relate annotations and transcript of a multimedia sequence to nodes in a formally or semi-formally represented ontology covering a broad range of possible multimedia documents. The device includes learning data preparation that involves certain special techniques for deriving data from the past mappings of annotations to nodes in an ontology, building inverted indices maintaining certain special statistics and a retriever that exploits these special statistics to rank the relevance of the nodes in an ontology for a given a set of new annotations.

Claims

1. An interactive multimedia delivery system dynamically linking contextual information with multimedia documents, said system retrieving said contextual information by searching an ontology and one or more databases over a network, said ontology comprising one or more nodes, said system comprising:

a. a learning data preparation component accessing mappings of annotations in said ontology and fusing annotations mapped in each of said nodes to form learning instances;

b. an intelligent inverted index creating a data structure based on the following calculated statistics for said learning instances;

term frequency (tf), inverse document frequency (idf), and contribution frequency (cf);

c. a retriever receiving a request for new annotations associated with multimedia documents, said retriever utilizing said inverted index to retrieve and rank most relevant nodes for said received new annotations, said ranking determined based upon a weight, wt_ij, contributed to a particular node in said ontology by the occurrence of a word i in a learning instance j;

d. an information retriever extracting information related to said requested annotations from said most relevant nodes and said one or more databases over said network, and

e. a contextual information linker linking multimedia content with said extracted information.

2. An interactive multimedia delivery system dynamically linking contextual information with multimedia documents, as per claim 1, wherein said weight wt_ij is given by:

wt_ij=(0.4+0.6×

Normalized_—

tf _ijxidf_j)×

wt_—

cf

3. An interactive multimedia delivery system dynamically linking contextual information with multimedia documents, as per claim 1, wherein said multimedia documents comprises audio, text, graphics, video documents.

4. An interactive multimedia delivery system dynamically linking contextual information with multimedia documents, as per claim 1, wherein said annotations are accessible via any of the following devices:

an interactive television, a computer, a portable computer, a handheld device, or a telephone.

5. An interactive multimedia delivery system dynamically linking contextual information with multimedia documents, as per claim 1, wherein said network is any of the following:

wide area network (WAN), local area network (LAN), wireless network, the telephony network, or the Internet.

6. An interactive multimedia delivery system dynamically linking contextual information with multimedia documents, as per claim 1, said learning data preparation further comprising:

a tokenizer, which tokenizes said learning instances;

a stemmer which stems said tokenized learning instances, and

a stop-word-remover, which removes stop words from said stemmed tokenized learning instances.

7. A method for searching an ontology of mapped multimedia annotations for appropriate annotations for one more multimedia documents, said ontology comprising one or more nodes, said method comprising the steps of:

a. receiving a request for searching and extracting one or more annotations related to said multimedia documents from said ontology;

b. identifying nodes in said ontology that are relevant to said multimedia documents, said nodes further comprising fused learning instances formed by fusing annotations in each of said nodes, said identification based upon using special statistics including term frequency, inverse document frequency and contribution frequency;

c. extracting information from said identified relevant nodes, and

d. dynamically linking said extracted information with said multimedia documents.

8. A method for searching an ontology of mapped multimedia annotations for appropriate annotations for one more multimedia documents, as per claim 7, wherein said multimedia documents comprises audio, text, graphics, video documents.

9. A method for searching an ontology of mapped multimedia annotations for appropriate annotations for one more multimedia documents, as per claim 7, wherein said annotations are accessible via any of the following devices:

an interactive television, a computer, a portable computer, a handheld device.

10. A method for searching an ontology of mapped multimedia annotations for appropriate annotations for one more multimedia documents, as per claim 7, said method further comprising:

tokenizing said learning instances;

stemming said tokenized learning instances, and

removing stop words from said stemmed tokenized learning instances.

11. A method for retrieving contextual information by searching an ontology and one or more databases, said method comprising:

receiving a request for contextual information;

retrieving from an ontology, with automatically mapped annotations, said requested contextual information using information retrieval statistics;

retrieving said requested contextual information from one or more databases, and

rendering an integrated presentation comprising audio, video, or graphics and said retrieved contextual information.

12. A method for retrieving contextual information by searching an ontology and one or more databases, as per claim 11, wherein said information retrieval statistics include calculating the following parameters:

13. A method for retrieving contextual information by searching an ontology and one or more databases, as per claim 11, wherein said information retrieval statistic further comprises calculating a weight contributed by a particular category in said ontology by a occurrence of word i in a learning vector j, said weight given by:

wt_ij=(0.4+0.6×

Normalized_—

tf _ijxidf_j)×

wt_—

cf

14. A method for retrieving contextual information by searching an ontology and one or more databases, as per claim 11, wherein said weight further depends on a contribution frequency, said contribution frequency given by the number of annotations (that comprises said learning instance) in which said word i appears.

15. A method for retrieving contextual information by searching an ontology and one or more databases, as per claim 11, wherein said annotations are retrieved from any of the following sources:

text documents, message boards, chat rooms, product descriptions, and multimedia documents comprising audio, video, images, and graphics in various formats.

16. A method for retrieving contextual information by searching an ontology and one or more databases, as per claim 11, wherein said annotations are viewable via any of the following devices:

an interactive television, a computer, or a handheld device, connected to the Internet, a cable system, or a wireless network.

17. A method for retrieving contextual information by searching an ontology and one or more databases, as per claim 11, wherein said databases are located on a network.

18. A method for retrieving contextual information by searching an ontology and one or more databases, as per claim 17, wherein said network is any of the following:

local area network (LAN), wide area network (WAN), wireless network, world wide web (WWW), or Internet.

19. A system for retrieving contextual information by searching for a selected multimedia representation, said system comprising:

a server, said server receiving requests for contextual information for a selected multimedia representation;

one or more databases associated with said server,

wherein said server retrieves both from its own ontology, said ontology having automatically mapped annotations, and from said one or more databases said requested contextual information, and renders said retrieved information as an integrated presentation comprising said multimedia and said retrieved contextual information.

20. A system for retrieving contextual information by searching for a selected multimedia representation, as per claim 19, wherein said information retrieval statistics includes calculating the following parameters:

$1)<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{Normalized\_tf}}_{\mathrm{ij}}=0.4+0.6\times <br><br>\frac{\log <br><br>\left({\mathrm{tf}}_{\mathrm{ij}}+0.5\right)}{\log <br><br>\left({\mathrm{max\_tf}}_j+1\right)}<br><br><br><br>2)<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{idf}}_i=\frac{\log <br><br>\left(\frac{N}{{\mathrm{df}}_i}\right)}{\log <br><br>\left(N\right)}<br><br><br><br>3)<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{wt\_cf}=\left(0.5+\frac{\mathrm{cf}}{\mathrm{tc}}\right)<br><br>\left(1.0-\frac{0.5}{1+0.05<br><br>t<br><br>\text{\hspace{1em}<br><br>}<br><br>c^2}\right)$wt_ij=(0.4+0.6×

Normalized_—

tf _ijxidf_j)×

wt_—

cf

21. A system for retrieving contextual information by searching for a selected multimedia representation, as per claim 19, wherein said information retrieval statistic further comprises calculating a weight contributed by a particular category in said ontology by a occurrence of word i in a learning vector j, said weight given by:

wt_ij=(0.4+0.6×

Normalized_—

tf _ijxidf_j)×

wt_—

cf

22. A system for retrieving contextual information by searching for a selected multimedia representation, as per claim 21, wherein said weight further depends on a contribution frequency, said contribution frequency given by the number of annotations (that comprises said learning instance) in which said word i appears.

23. A system for retrieving contextual information by searching for a selected multimedia representation, as per claim 19, wherein said contextual information are retrieved from any of the following sources:

text documents, message boards, chat rooms, product descriptions, and multimedia documents comprising audio, video, images, and graphics in various formats.

24. A system for retrieving contextual information by searching for a selected multimedia representation, as per claim 19, wherein said contextual information is accessible via any of the following devices:

an interactive television, a computer, or a handheld device, connected to the Internet, a cable system, or a wireless network.

25. A system for retrieving contextual information by searching for a selected multimedia representation, as per claim 19, wherein said databases are located on a network.

26. A system for retrieving contextual information by searching for a selected multimedia representation, as per claim 25, wherein said network is any of the following:

local area network (LAN), wide area network (WAN), wireless network, world wide web (WWW), or Internet.

27. A method for automatically mapping annotations to ontologies, said method comprising the steps of:

extracting annotations from a multimedia document segment;

mapping said extracted multimedia document segment to an appropriate node in said ontology;

comparing to other related content mapped to said appropriate node, and

integrating said related content with said extracted multimedia document segment.

28. A method for automatically mapping annotations to ontologies, as per claim 27, wherein pre-certification of said related content is required before said integration step.

29. A method for automatically mapping annotations to ontologies, as per claim 27, wherein said step of integration is accomplished via dynamic content linking.

30. A method for automatically mapping annotations to ontologies, as per claim 27, wherein said annotations are retrieved from any of the following sources:

text documents, message boards, chat rooms, product descriptions, and multimedia documents comprising audio, video, images, and graphics in various formats.

31. A method for automatically mapping annotations to ontologies, as per claim 27, wherein said annotations are accessible via any of the following devices:

an interactive television, a computer, or a handheld device, connected to the Internet, a cable system, or a wireless network.

32. An article of manufacture comprising a computer usable medium having computer readable program code embodied therein which searches an ontology of mapped multimedia annotations for appropriate annotations for one more multimedia documents, said ontology comprising one or more nodes, said article comprising:

a. computer readable program code receiving a request for searching and extracting one or more annotations related to said multimedia documents from said ontology;

b. computer readable program code identifying nodes in said ontology that are relevant to said multimedia documents, said nodes further comprising fused learning instances formed by fusing annotations in each of said nodes, said identification based upon using special statistics including term frequency, inverse document frequency and contribution frequency;

c. computer readable program code extracting information from said identified relevant nodes, and

d. computer readable program code dynamically linking said extracted information with said multimedia documents.