Mixed media reality recognition using multiple specialized indexes

• US 8,369,655 B2
• Filed: 09/29/2008
• Issued: 02/05/2013
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Moraleda, Jorge, Erol, Berna, Hull, Jonathan J.
• Primary Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US12/240,590
• Publication Number:
  US 20090070302A1
• Time in Patent Office: 1,590 Days
• Field of Search: 382/305, 707/E17.014, 707/E17.019
• US Class Current: 382/305
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

An MMR system for searching across multiple indexes comprises a plurality of mobile devices, a pre-processing server or MMR gateway, and an MMR matching unit, and may include an MMR publisher. The MMR matching unit receives an image query from the pre-processing server or MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The MMR matching unit includes a segmenter for segmenting received images by content type, a distributor for distributing the images to corresponding content type index tables, and an integrator for integrating recognition results. The result is returned to the mobile device via the pre-processing server or MMR gateway. The present invention also includes a number of novel methods including a method for processing content-type specific image queries and for processing queries across multiple indexes.

Claims

1. A dispatcher apparatus having:

one or more processors;

a segmenter stored on a memory and executable by the one or more processors, the segmenter for receiving an image query including an image and segmenting the image into one or more content-type specific queries;

a distributor stored on the memory and executable by the one or more processors, the distributor coupled to the segmenter for submitting the one or more content-type specific queries to one or more coupled corresponding content-type index tables for recognition; and

an integrator stored on the memory and executable by the one or more processors, the integrator for receiving recognition results from the one or more corresponding content-type index tables, integrating the recognition results into an integrated result based on a level of agreement between the recognition results and transmitting the integrated result, wherein the integrated result includes a page and a location on the page corresponding to the image query.

2. The dispatcher apparatus of claim 1, wherein the one or more content-types are selected from a group of black text on white background, black and white natural images, color natural images, black and white diagrams, color diagrams, headings, and color text.

3. The dispatcher apparatus of claim 1, wherein information regarding the one or more content-types is received with the image query.

4. The dispatcher apparatus of claim 1, wherein the segmenter is further configured for determining information regarding the one or more content-types from the received image query.

5. The dispatcher apparatus of claim 1, wherein integrating the recognition results comprises consideration of at least one factor selected from a competence factor associated with the recognition results, context information for the recognition results, surface area of an image segment, and prior probability of correctness for the one or more corresponding content-type index tables.

6. The dispatcher apparatus of claim 1, wherein the integrated result includes a determination that a same recognition result was produced by the content-type index tables.

7. A computer-implemented method of processing content-type specific queries, comprising:

receiving an image query including an image;

segmenting, with one or more processors, the image into one or more content-type specific queries;

submitting the one or more content-type specific queries to one or more corresponding content-type index tables for recognition;

receiving recognition results from the one or more corresponding content-type index tables;

integrating, with the one or more processors, the recognition results into an integrated result based on a level of agreement between the recognition results, the integrated result including a page and a location on the page corresponding to the image query; and

transmitting the integrated result.

8. The method of claim 7, wherein information regarding the one or more content-types is received with the image query.

9. The method of claim 7, further comprising determining information regarding the one or more content-types from the received image query.

10. The method of claim 7, wherein the one or more content-types are selected from black text on white background, black and white natural images, color natural images, black and white diagrams, color diagrams, headings, and color text.

11. The method of claim 7, further comprising determining the one or more content-types included in the one or more corresponding content-type index tables.

12. The method of claim 7, wherein the integrated result includes a determination that a same recognition result was produced by the content-type index tables.

13. A computer program product comprising a non-transitory machine-readable medium including a non-transitory computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive an image query including an image;

segment the image into one or more content-type specific queries;

submit the one or more content-type specific queries to one or more corresponding content-type index tables for recognition;

receive recognition results from the one or more corresponding content-type index tables;

integrate the recognition results into an integrated result based on a level of agreement between the recognition results, the integrated result including a page and a location on the page corresponding to the image query; and

transmit the integrated result.

14. The computer program product of claim 13, wherein information regarding the one or more content-types is received with the image query.

15. The computer program product of claim 13, further causing the computer to determine information regarding the one or more content-types from the received image query.

16. The computer program product of claim 13, wherein the one or more content-types are selected from black text on white background, black and white natural images, color natural images, black and white diagrams, color diagrams, headings, and color text.

17. The computer program product of claim 13, further causing the computer to determine the one or more content-types included in the one or more corresponding content-type index tables.

18. The computer program product of claim 13, wherein the integrated result includes a determination that a same recognition result was produced by the content-type index tables.