ARCHITECTURE FOR MIXED MEDIA REALITY RETRIEVAL OF LOCATIONS AND REGISTRATION OF IMAGES

• US 20090070415A1
• Filed: 09/15/2008
• Published: 03/12/2009
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Furukawa, Takahiro, Graham, Jamey, Hull, Jonathan J., Erol, Berna, Takahashi, Sadao, Kishi, Hidenobu, Moraleda, Jorge
• Granted Patent: US 8,825,682 B2
• Time in Patent Office: Days
• US Class Current: 709/203
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06K 15/00 Arrangements for producing a permanent visual presentation of the output data , e.g. computer output printersprinting or plotting combined with another operation, e.g. with conveying, G06K17/00
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32765 Initiating a communication
• CPC Class Codes: H04N 1/32776 using an interactive, user-operated device, e.g. a computer terminal, mobile telephone H04N1/32782, H04N1/32784 take precedence

Abstract

A MMR system for publishing comprises a plurality of mobile devices, an MMR gateway, an MMR matching unit and an MMR publisher. The mobile devices send retrieval requests including image queries and other contextual information. The MMR gateway processes the retrieval request from the mobile devices and then generates an image query that is passed on to the MMR matching unit. The MMR matching unit receives an image query from the MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The MMR matching unit includes an image registration unit that receives new content from the MMR publisher and updates the index table of the MMR matching unit. A method for automatically registering images and other data with the MMR matching unit, a method for dynamic load balancing and a method for image-feature-based queue ordering are also included.

Claims

1. A system for receiving a retrieval request and generating a recognition result, the system comprising:

a gateway having a plurality of inputs for receiving the retrieval request, the gateway processing the retrieval request to generate an image query and a recognition parameter, the gateway having a plurality of outputs for sending recognition results; and

a matching unit coupled to receive the image query and the recognition parameter from the gateway, the matching unit analyzing and comparing the image query and recognition parameter to an index table to generate the recognition result, the matching unit coupled to send the recognition result to the gateway.

2. The system of claim 1 further comprising:

a publisher for generating at least one image, the publisher coupled to provide the image to the matching unit; and

wherein the matching unit includes an image registration unit for automatically extracting image features from the image and updating an index table, the image registration unit coupled to the publisher to receive images.

3. The system of claim 1 wherein:

the retrieval request includes one or more of an image query, other contextual information and metadata;

the image query is an image in any format or one or more features of an image; and

the recognition results include one or more from the group of a Boolean value (true/false), a page ID, a location on the page, an action, a document, an action defined within a portable document file, an address, information capable of being rendered on a mobile device, binary data, menus with additional actions, raster images, and image features.

4. The system of claim 1 further comprising a mobile device having plug-in module, the mobile device capable of capturing an image, the mobile device adapted for communication with the gateway, and the plug-in module adapted to process images, present a captured image on the mobile device, receive user selection of the captured image, send the captured image to the mobile device.

5. The system of claim 1 further comprising a hot spot database having additional information related to the recognition results, the hot spot database is accessed using the recognition results to retrieve additional information, and the additional information is output by the gateway along with the recognition results.

6. The system of claim 1 wherein the matching unit comprises:

an acquisition unit having a plurality of recognition units, each recognition coupled to a corresponding index table, the recognition units performing recognition on the image query and retrieving the recognition results from the index tables; and

a dispatcher for receiving image queries and recognition parameters and sending them to specific recognition units of the acquisition unit.

7. The system of claim 6 wherein a plurality of recognition units and corresponding index tables include:

a first recognition unit performing an invisible junction algorithm, a second recognition unit performing a brick wall coding algorithm, and a third recognition unit performing a path coding algorithm.

8. The system of claim 6 wherein a plurality of recognition units and corresponding index tables include:

a first recognition unit and index table for a first group of images and a second recognition unit and index table for a second group of images, wherein the first group of images is different than the second group of images.

9. The system of claim 6 wherein the matching unit comprises an image registration module for updating the index tables of the acquisition unit, the image registration module adapted to receive images and page numbers, extract features and add the features and page numbers to the index tables of the acquisition unit.

10. The system of claim 9 wherein the matching unit includes a dynamic load balancer for dynamically adjust the index tables, the dynamic load balancer coupled to monitor and evaluate the image query sent from the dispatcher and recognition results output by the acquisition unit to measure usage, and generating a control signal for the image registration module to modify configuration of the index tables to match usage.

11. The system of claim 6 wherein the dispatcher includes an image feature order unit having an input for receiving the image query and an output for an ordering signal, the image feature order unit analyzing an input image query and predicting a time required to recognize the input image query by analyzing its image features, the ordering signal indicating a recognition time.

12. The system of claim 6 wherein at least one recognition unit includes a feature extractor coupled to a retriever, the feature extractor coupled having an input for receiving an image query, and producing image features from the image query and sending them to the retriever, the retriever comparing the image features to the index to generate the recognition results.

13. A method for dynamic load balancing in a mixed media reality system, the method comprising:

monitoring for image queries and recognition results for a pre-define period;

generating a load balancing control signal specifying a configuration for index tables of an acquisition unit; and

loading a first set of index tables and a second set of index tables with content according to the load balancing control signal.

14. The method of claim 13 wherein the image queries and recognition results are transferred between a dispatcher and an acquisition unit.

15. The method of claim 13 wherein the generating a load balancing control signal includes determining a distribution of images successfully found and in which index tables.

16. The method of claim 13 further comprising:

recognizing an input image query using the first set of index tables;

determining whether recognition was successful;

if recognition was not successful, recognizing the input image query using the second set of index tables.

17. The method of claim 16 wherein the first set of index tables correspond to content for a current day and the second set of index tables correspond to content from other past days.

18. A method for performing image feature-based ordering in the distribution of image queries by a dispatcher to a acquisition unit having a plurality of recognition units, the method comprising:

receiving an image query;

analyzing image features in the image query generating a predicted time by measuring a time required to recognize the image features;

creating correlations between image features and predicted times;

measuring an actual time required by the acquisition unit to recognize the image query;

adjusting the correlations by the actual time;

using the adjusted correlations to reorder and assign image queries to recognition units of the acquisition unit.

19. The method of claim 18 wherein the image features using in analyzing image features in the image query are different from image features using in performing recognition.

20. The method of claim 18 wherein analyzing image features in the image query includes analyzing several different feature sets and measuring correlations over time.

21. The method of claim 20 further comprising selecting and using a feature set that provides a best predictor and has a lowest computational cost from the several different feature sets.