Mixed Media Reality Recognition Using Multiple Specialized Indexes

• US 20090070302A1
• Filed: 09/29/2008
• Published: 03/12/2009
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Moraleda, Jorge
• Granted Patent: US 8,369,655 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

An MMR system for searching across multiple indexes comprises a plurality of mobile devices, a pre-processing server or MMR gateway, and an MMR matching unit, and may include an MMR publisher. The MMR matching unit receives an image query from the pre-processing server or MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The MMR matching unit includes a segmenter for segmenting received images by content type, a distributor for distributing the images to corresponding content type index tables, and an integrator for integrating recognition results. The result is returned to the mobile device via the pre-processing server or MMR gateway. The present invention also includes a number of novel methods including a method for processing content-type specific image queries and for processing queries across multiple indexes.

Claims

1. A dispatcher apparatus having:

a first input to receive an image query;

a segmenter coupled to the first input for segmenting an image into one or more content-type specific image queries;

a distributor coupled to the segmenter for submitting the one or more content-type specific queries to one or more coupled corresponding content type index tables for recognition;

a second input coupled for receiving recognition results from the one or more corresponding content type index tables;

an integrator for integrating the recognition results into an integrated result; and

an output coupled to the second input for transmitting the integrated result.

2. The dispatcher apparatus of claim 1, wherein the image content types are selected from the group of black text on white background, black and white natural images, color natural images, black and white diagrams, color diagrams, headings, and color text.

3. The dispatcher apparatus of claim 1, wherein information regarding the image content type is received with the image query.

4. The dispatcher apparatus of claim 1, wherein the segmenter is further configured for determining image content type information from the received image query.

5. The dispatcher apparatus of claim 1, wherein integrating the recognition results comprises consideration of at least one factor selected from a competence factor associated with the recognition results, context information for the recognition results, surface area of the image segment, prior probability of correctness for the one or more corresponding content type index tables, and level of agreement between recognition results received from the one or more corresponding content type index tables.

6. The dispatcher apparatus of claim 1, wherein the integrated result includes a matched document, page, and location on the page.

7. A method of processing content-type specific image queries, comprising:

receiving an image query;

segmenting the image query by image content type to produce one or more content-type specific image queries;

submitting the one or more content-type specific queries to one or more corresponding content type index tables for recognition;

receiving recognition results from the one or more corresponding content type index tables;

integrating the recognition results into an integrated result; and

transmitting the integrated result.

8. The method of claim 7, wherein information regarding the image content type is received with the image query.

9. The method of claim 7, further comprising determining image content type information from the received image query.

10. The method of claim 7, wherein the image content types are selected from black text on white background, black and white natural images, color natural images, black and white diagrams, color diagrams, headings, and color text.

11. The method of claim 7, further comprising determining indexed content types included in the content type index tables.

12. The method of claim 7, wherein the integrated result includes a matched document, page, and location on the page.

13. A method of processing image queries across multiple index tables, comprising:

receiving an image query and associated recognition parameters;

correlating the received recognition parameters with indexed parameters for each of a plurality of index tables to formulate a strategy for submission of the image query to a subset of the plurality of index tables; and

receiving recognition results from the subset.

14. The method of claim 13 wherein the formulated strategy is parallel submission to the subset, further comprising after receiving the recognition results, integrating the recognition results into an integrated result.

15. The method of claim 13, wherein the integrated result includes a matched document, page, and location on the page.

16. The method of claim 13, wherein integrating the recognition results comprises consideration of at least one factor selected from a competence factor associated with the recognition results, context information for the recognition results, surface area of the image segment, prior probability of correctness for the one or more corresponding content type index tables, and level of agreement between recognition results received from the one or more corresponding content type index tables.

17. The method of claim 13 wherein the formulated strategy is serial submission to the subset, further comprising establishing an order for submission of the image query to the index tables in the subset.

18. The method of claim 17, wherein submission of the image query according to the established order further comprises:

submitting the image query to a first of the subset of the plurality of index tables for recognition; and

responsive to unsuccessful recognition by the first of the subset of the plurality of index tables, submitting the image query to a second of the subset of the plurality of index tables for recognition.

19. The method of claim 13, further comprising, responsive to unsuccessful recognition by the subset of the plurality of index tables, storing the received image query.

20. The method of claim 13, further comprising learning the submission strategy associated with the received recognition parameters.