Mixed Media Reality Recognition With Image Tracking

• US 20090092287A1
• Filed: 10/07/2008
• Published: 04/09/2009
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Moraleda, Jorge
• Granted Patent: US 9,063,952 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

An MMR system integrating image tracking and recognition comprises a plurality of mobile devices, a pre-processing server or MMR gateway, and an MMR matching unit, and may include an MMR publisher. The MMR matching unit receives an image query from the pre-processing server or MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page, and the location on the page. Image tracking information also is provided for determining relative locations of images within a document page. The mobile device includes an image tracker for providing at least a portion of the image tracking information. The present invention also includes methods for image tracking-assisted recognition, recognition of multiple images using a single image query, and improved image tracking using MMR recognition.

Claims

1. A method of image tracking-assisted recognition, comprising:

receiving an image;

receiving image tracking information indicating a location of the received image relative to a location associated with a previously received image for which recognition has been performed;

in response to the image tracking information indicating that a document page includes the locations of the received image and the previously received image, retrieving a recognition result associated with the previously received image.

2. The method of claim 1, wherein no image query is submitted corresponding to the received image.

3. The method of claim 1, further comprising:

prior to retrieving the recognition result associated with the previously received image;

submitting an image query for the received image;

receiving a recognition result associated with the image query for the received image; and

verifying the tracking image information indicating that the document page includes the locations of the received image and the previously received image by comparing the recognition result associated with the received image to the recognition result associated with the previously received image.

4. The method of claim 3, further comprising:

calculating image tracking information from the recognition results; and

in response to an inconsistency between the calculated image tracking information and the received image tracking information, updating the received image tracking information.

5. A method of recognition of a plurality of received images using a single image query, comprising:

receiving a plurality of images corresponding to a document page;

receiving image tracking information indicating relative locations of the received images on the document page;

submitting the single image query corresponding to the plurality of received images; and

receiving a recognition result associated with the single image query.

6. The method of claim 5, wherein the recognition result includes a matched document, page, and location on the page.

7. The method of claim 5, wherein submitting the single image query corresponding to the plurality of received images further comprises:

combining the plurality of images corresponding to the document page into a merged image; and

forming the single image query based upon the merged image.

8. The method of claim 7, wherein the image tracking information comprises instructions for the combining.

9. The method of claim 7, wherein the merged image is a combined image stitched together from the plurality of images.

10. The method of claim 7, wherein the merged image is a super-resolution image synthesized from the plurality of images.

11. The method of claim 7, wherein the combining occurs on a mobile device.

12. The method of claim 7, wherein the combining occurs on a server.

13. The method of claim 12, wherein the image tracking information is provided by the server, and a mobile device provides sequence and relative timing information for the plurality of images.

14. The method of claim 5, wherein submitting the single image query corresponding to the plurality of received images further comprises:

extracting a set of features from each of the plurality of images corresponding to the document page;

combining the extracted sets of features to form a spatially-consistent superset of features; and

forming the single image query based upon the spatially-consistent superset of features.

15. The method of claim 14, wherein features extracted from images for the same location on the document page are combined into consensus features.

16. The method of claim 15, further comprising excluding from the superset extracted sets of features determined to be inconsistent with the consensus features.

17. The method of claim 5, further comprising cropping the recognition result based upon the image tracking information indicating the relative locations of the received images on the document page.

18. The method of claim 5, wherein the received plurality of images are from a video stream.

19. A method of improved image tracking using MMR, comprising:

receiving an image;

receiving image tracking information indicating a location of the received image on a document page relative to a location of a previously received image on the document page;

submitting an image query corresponding to the received image;

receiving a recognition result associated with the image query;

determining from the recognition result an absolute location of the received image on the document page; and

updating the image tracking information to reflect the absolute location of the received image on the document page.

20. The method of claim 19, wherein submitting the image query corresponding to the received image further comprises:

retrieving a recognition result associated with the previously received image;

submitting a document ID for the recognition result associated with the previously received image and the image tracking information.