Architecture for mixed media reality retrieval of locations and registration of images

• US 8,825,682 B2
• Filed: 09/15/2008
• Issued: 09/02/2014
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Kishi, Hidenobu, Takahashi, Sadao, Furukawa, Takahiro, Erol, Berna, Hull, Jonathan J., Moraleda, Jorge, Graham, Jamey
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US12/210,511
• Publication Number:
  US 20090070415A1
• Time in Patent Office: 2,178 Days
• Field of Search: 707758-760, 707/915, 707/696, 707/711, 707/715, 707/736, 707/741, 707/769, 707/830, 382/190, 382/305, 382171-172, 382/180, 382/219, 358 117- 118, 709/203, 709/206, 715209-210, 715/762, 715/864
• US Class Current: 707/758
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06K 15/00 Arrangements for producing a permanent visual presentation of the output data , e.g. computer output printersprinting or plotting combined with another operation, e.g. with conveying, G06K17/00
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32765 Initiating a communication
• CPC Class Codes: H04N 1/32776 using an interactive, user-operated device, e.g. a computer terminal, mobile telephone H04N1/32782, H04N1/32784 take precedence

Abstract

A MMR system for publishing comprises a plurality of mobile devices, an MMR gateway, an MMR matching unit and an MMR publisher. The mobile devices send retrieval requests including image queries and other contextual information. The MMR gateway processes the retrieval request from the mobile devices and then generates an image query that is passed on to the MMR matching unit. The MMR matching unit receives an image query from the MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The MMR matching unit includes an image registration unit that receives new content from the MMR publisher and updates the index table of the MMR matching unit. A method for automatically registering images and other data with the MMR matching unit, a method for dynamic load balancing and a method for image-feature-based queue ordering are also included.

Claims

1. A system for receiving a retrieval request and generating a recognition result, the system comprising:

a processor;

a gateway having a plurality of inputs for receiving the retrieval request that includes at least an image portion and contextual information, the gateway processing the retrieval request to generate an image query and a recognition parameter, the gateway having a plurality of outputs for sending recognition results;

a matching unit stored on a computer-readable storage medium and executable by the processor, the matching unit coupled to receive the image query and the recognition parameter from the gateway, the matching unit analyzing and comparing the image query and recognition parameter to an index table to generate the recognition result including a page identification number and a location on the recognition result, the matching unit coupled to send the recognition result to the gateway; and

a hotspot database coupled to the matching unit, the hotspot database for retrieving hotspot information corresponding to the page and the location identified in the recognition result.

2. The system of claim 1 further comprising:

a publisher that generates at least one image, the publisher coupled to provide the image to the matching unit; and

wherein the matching unit includes an image registration unit that automatically extracts image features from the image and that updates an index table, the image registration unit coupled to the publisher to receive images.

3. The system of claim 1 wherein:

the contextual information includes one or more from the group of device information, placement information, date, location, weather, personal profile and retrieval history;

the image query is an image in any format or one or more features of the image; and

the recognition results include one or more from the group of a Boolean value (true/false), an action, a document, the action defined within a portable document file, an address, information capable of being rendered on a mobile device, binary data, menus with additional actions, raster images, and image features.

4. The system of claim 1 further comprising a mobile device having plug-in module, the mobile device capable of capturing an image, the mobile device adapted for communication with the gateway, and the plug-in module adapted to process images, present a captured image on the mobile device, receive user selection of the captured image, and send the captured image to the gateway.

5. The system of claim 1, wherein the hotspot information related to the recognition result also includes links associated with the recognition result.

6. The system of claim 1 wherein the matching unit comprises:

an acquisition unit having a plurality of recognition units, each recognition unit coupled to a corresponding index table, the recognition units performing recognition on the image query and retrieving the recognition results from the index tables; and

a dispatcher that receives image queries and recognition parameters, sending them to specific recognition units of the acquisition unit.

7. The system of claim 6 wherein a plurality of recognition units and corresponding index tables include:

a first recognition unit performing an invisible junction algorithm, a second recognition unit performing a brick wall coding algorithm, and a third recognition unit performing a path coding algorithm.

8. The system of claim 6 wherein a plurality of recognition units and corresponding index tables include:

a first recognition unit and index table for a first group of images and a second recognition unit and index table for a second group of images, wherein the first group of images is different than the second group of images.

9. The system of claim 6 wherein the matching unit comprises an image registration module that updates the index tables of the acquisition unit, the image registration module adapted to receive images and page numbers, extract features and add the features and page numbers to the index tables of the acquisition unit.

10. The system of claim 9 wherein the matching unit includes a dynamic load balancer that dynamically adjusts the index tables, the dynamic load balancer coupled to monitor and evaluate the image query sent from the dispatcher and recognition results output by the acquisition unit to measure usage, and generating a control signal for the image registration module to modify configuration of the index tables to match usage.

11. The system of claim 6 wherein the dispatcher includes an image feature order unit having an input for receiving the image query and an output for an ordering signal, the image feature order unit analyzing an input image query and predicting a time required to recognize the input image query by analyzing its image features, the ordering signal indicating a recognition time.

12. The system of claim 6 wherein at least one recognition unit includes a feature extractor coupled to a retriever, the feature extractor having an input that receives an image query, produces image features from the image query and sends them to the retriever, the retriever comparing the image features to the index to generate the recognition results.