DATABASE FOR MIXED MEDIA DOCUMENT SYSTEM

• US 20070050411A1
• Filed: 07/31/2006
• Published: 03/01/2007
• Est. Priority Date: 08/23/2005
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Piersol, Kurt, Lee, Dar-Shyang, Hull, Jonathan
• Granted Patent: US 9,405,751 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system includes a content-based retrieval database configured with an index table to represent two-dimensional geometric relationships between objects extracted from a printed document in a way that allows look-up using a text-based index. A ranked set of document, page and location hypotheses can be computed given data from the index table. The techniques effectively transform features detected in an image patch into textual terms (or other searchable features) that represent both the features themselves and the geometric relationship between them. A storage facility can be used to store additional characteristics about each document image patch.

Claims

1. A database system for providing mixed media documents, comprising:

an index table for receiving an electronic description of a feature extracted from a paper document, and associating the paper document and a location within that document of the feature with a mixed media document that combines printed and digital media; and

an accumulator module for receiving query terms that capture two-dimensional relationships between objects in a target document, and computing at least one mixed media document and location hypotheses potentially responsive to the query terms, based on data from the index table.

2. The system of claim 1 further comprising:

a storage facility for storing additional characteristics associated with the target document.

3. The system of claim 2 wherein the storage facility is a relational database.

4. The system of claim 2 wherein the additional characteristics include one or more actions including at least one of retrieval of textual information, retrieval of graphical information, executing a process, executing a command, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document, and displaying a document.

5. The system of claim 1 wherein the index table is capable of receiving electronic descriptions of features extracted from a plurality of paper documents.

6. The system of claim 1 wherein the paper document includes multiple pages, and the index table is further configured for associating paper documents, pages, and x-y feature locations within those pages with one or more mixed media documents.

7. The system of claim 1 wherein the computing at least one mixed media document and location hypotheses carried out by the accumulator module includes computing a ranked set of mixed media document, page, and location hypotheses based on data from the index table.

8. The system of claim 1 wherein the receiving query terms that capture two-dimensional relationships between objects in a target document carried out by the accumulator module includes receiving a set of horizontally and vertically adjacent word-pairs extracted from the target document.

9. The system of claim 1 wherein the index table comprises:

an inverted term index table, where each unique term in the inverted index table points to a list of records, and each record identifies a candidate region on a page within a mixed media document.

10. The system of claim 9 wherein the computing at least one mixed media document and location hypotheses carried out by the accumulator module includes examining every record indexed by a key corresponding to the query terms, and identifying a region most consistent with all query terms.

11. The system of claim 10 wherein if the identified region has a matching score that satisfies matching criteria, the accumulator module confirms the corresponding mixed media document and location hypothesis.

12. The system of claim 9 wherein the index table further comprises:

a document index table that includes relevant information for each mixed media document, the information including at least one of print resolution, print date, paper size, shadow file name, and page image location.

13. The system of claim 1 wherein the electronic description received by the index table is computed by a feature extraction module that associates extracted features with inner-document location data of those features.

14. The system of claim 1 wherein the target document is an image of a paper document or a patch of that paper document.

15. A computer implemented method for providing mixed media documents, comprising:

receiving, at an index table, an electronic description of a feature extracted from a paper document, the index table for associating the paper document and a location within that document of the feature with a mixed media document that combines printed and digital media;

receiving query terms that capture two-dimensional relationships between objects in a target document; and

computing at least one mixed media document and location hypotheses potentially responsive to the query terms, based on data from the index table.

16. The method of claim 15 further comprising:

storing additional characteristics associated with the target document.

17. The method of claim 16 wherein the additional characteristics include one or more actions including at least one of retrieval of textual information, retrieval of graphical information, executing a process, executing a command, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document, and displaying a document.

18. The method of claim 15 wherein receiving, at an index table, an electronic description of a feature extracted from a paper document, includes:

receiving electronic descriptions of features extracted from a plurality of paper documents.

19. The method of claim 15 wherein computing at least one mixed media document and location hypotheses includes:

computing a ranked set of mixed media document, page, and location hypotheses based on data from the index table.

20. The method of claim 15 wherein receiving query terms that capture two-dimensional relationships between objects in a target document includes:

receiving a set of horizontally and vertically adjacent word-pairs extracted from the target document.

21. The method of claim 15 wherein the index table comprises an inverted term index table, where each unique term in the inverted index table points to a list of records, and each record identifies a candidate region on a page within a mixed media document, and computing at least one mixed media document and location hypotheses includes:

examining every record indexed by a key corresponding to the query terms; and

identifying a region most consistent with all query terms.

22. The method of claim 21 wherein if the identified region has a matching score that satisfies matching criteria, the method further includes:

confirming the corresponding mixed media document and location hypothesis.

23. The method of claim 15 wherein the target document is an image of a paper document or a patch of that paper document.

24. A machine-readable medium encoded with instructions, that when executed by one or more processors, cause the processor to carry out a process for providing mixed media documents, the process comprising:

receiving, at an index table, an electronic description of a feature extracted from a paper document, the index table for associating the paper document and a location within that document of the feature with a mixed media document that combines printed and digital media;

receiving query terms that capture two-dimensional relationships between objects in a target document; and

computing at least one mixed media document and location hypotheses potentially responsive to the query terms, based on data from the index table.

25. The machine-readable medium of claim 24, the process further comprising:

storing additional characteristics associated with the target document, the additional characteristics including one or more actions, including at least one of retrieval of textual information, retrieval of graphical information, executing a process, executing a command, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document, and displaying a document.

26. The machine-readable medium of claim 24 wherein computing at least one mixed media document and location hypotheses includes:

computing a ranked set of mixed media document, page, and location hypotheses based on data from the index table.

27. The machine-readable medium of claim 24 wherein receiving query terms that capture two-dimensional relationships between objects in a target document includes:

receiving a set of horizontally and vertically adjacent word-pairs extracted from the target document.

28. The machine-readable medium of claim 24 wherein the index table comprises an inverted term index table, where each unique term in the inverted index table points to a list of records, and each record identifies a candidate region on a page within a mixed media document, and computing at least one mixed media document and location hypotheses includes:

examining every record indexed by a key corresponding to the query terms; and

identifying a region most consistent with all query terms.

29. The machine-readable medium of claim 28 wherein if the identified region has a matching score that satisfies matching criteria, the process further includes:

confirming the corresponding mixed media document and location hypothesis.

30. The machine-readable medium of claim 24 where the target document is an image of a paper document or a patch of that paper document.