Mixed media reality recognition with image tracking

• US 9,063,952 B2
• Filed: 10/07/2008
• Issued: 06/23/2015
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Moraleda, Jorge, Erol, Berna, Hull, Jonathan J.
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): BROMELL, ALEXANDRIA Y
• Application Number:
  US12/247,205
• Publication Number:
  US 20090092287A1
• Time in Patent Office: 2,450 Days
• Field of Search: 707/915, 382/103, 382/181
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

An MMR system integrating image tracking and recognition comprises a plurality of mobile devices, a pre-processing server or MMR gateway, and an MMR matching unit, and may include an MMR publisher. The MMR matching unit receives an image query from the pre-processing server or MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page, and the location on the page. Image tracking information also is provided for determining relative locations of images within a document page. The mobile device includes an image tracker for providing at least a portion of the image tracking information. The present invention also includes methods for image tracking-assisted recognition, recognition of multiple images using a single image query, and improved image tracking using MMR recognition.

Claims

1. A method of image tracking-assisted recognition, the method comprising:

receiving a first image depicting at least a first portion of a media;

determining a first recognition result comprising an electronic document that matches the first image and an absolute location of the first image on the electronic document;

receiving a second image depicting at least a second portion of the media;

generating first image tracking information responsive to receiving the second image, the first image tracking information indicating a relative location of the second image to the absolute location of the first image on the electronic document;

responsive to the first image tracking information indicating that the electronic document includes the relative location of the second image to the absolute location of the first image, retrieving a first recognition result, the first recognition result corresponding to both the first image and the second image;

determining a second recognition result using the second image; and

verifying the first image tracking information indicating that the electronic document includes the relative location of the second image to the absolute location of the first image by comparing the second recognition result to the first recognition result.

2. The method of claim 1, further comprising retrieving subsequent recognition results periodically for a set of consecutive images corresponding to a video stream.

3. The method of claim 1, further comprising:

calculating second image tracking information from the first recognition result and the second recognition result; and

in response to an inconsistency between the second image tracking information and the first image tracking information, updating the first image tracking information.

4. A method of improved image tracking, the method comprising:

receiving a first image depicting at least a first portion of a media;

determining a first recognition result comprising an electronic document that matches the first image and a first absolute location of the first image on the electronic document;

receiving a second image depicting at least a second portion of the media;

generating image tracking information responsive to receiving the second image, the image tracking information indicating a relative location of the second image to the first absolute location of the first image on the electronic document;

submitting an image query corresponding to the second image;

receiving a second recognition result associated with the image query, the second recognition result comprising an identifier and a second absolute location of the second image on the electronic document;

determining the second absolute location of the second image on the electronic document from the second recognition result; and

updating the image tracking information to replace the relative location of the second image with the second absolute location of the second image.

5. The method of claim 4, wherein submitting the image query further comprises:

retrieving the first recognition result associated with the first image; and

submitting a document identifier for the first recognition result and the image tracking information as the image query.

6. The method of claim 4, wherein updating the image tracking information is used to determine an absolute camera position to correct for cumulative drift.

7. A system of image tracking-assisted recognition, the system comprising:

one or more processors;

a central scheduler;

a tracking manager, stored on a memory and executable by the one or more processors, the tracking manager for;

receiving a first image depicting at least a first portion of a media, for determining a first recognition result comprising an electronic document that matches the first image and an absolute location of the first image on the electronic document;

receiving a second image depicting at least a second portion of the media;

generating first image tracking information responsive to receiving the second image, the first image tracking information indicating a relative location of the second image to the absolute location of the first image on the electronic document and responsive to the first image tracking information indicating that the electronic document includes the relative location of the second image to the absolute location of the first image;

retrieving a first recognition result, the first recognition result corresponding to both the first image and the second image;

determining a second recognition result using the second image; and

verifying the first image tracking information indicating that the electronic document includes the relative location of the second image to the absolute location of the first image by comparing the second recognition result to the first recognition result; and

a database coupled to the tracking manager, the database for storing the first recognition result.

8. The system of claim 7, wherein the tracking manager retrieves subsequent recognition results periodically for a set of consecutive images corresponding to a video stream.

9. The system of claim 7:

wherein the tracking manager calculates second image tracking information from the first recognition result and the second recognition result and in response to an inconsistency between the second image tracking information and the first image tracking information, updates the first image tracking information.

10. A system of improved image tracking, the system comprising:

one or more processors;

a tracking manager, stored on a memory and executable by the one or more processors, the tracking manager for receiving a first image depicting at least a first portion of a media, for determining a first recognition result comprising an electronic document that matches the first image and a first absolute location of the first image on the electronic document, for receiving a second image depicting at least a second portion of the media, for generating image tracking information responsive to receiving the second image, the image tracking information indicating a relative location of the second image to the first absolute location of the first image on the electronic document, for submitting an image query corresponding to the second image, for receiving a second recognition result associated with an image query, the second recognition result comprising an identifier and a second absolute location of the second image on the electronic document, for determining a second absolute location of the second image on the electronic document from the second recognition result and for updating the image tracking information to replace the relative location of the second image with the second absolute location of the second image; and

a database coupled to the tracking manager, the database for storing the first recognition result.

11. The system of claim 10:

wherein the tracking manager retrieves the first recognition result associated with the first image and submits a document identifier for the first recognition result and the image tracking information as the image query.

12. The system of claim 10, wherein updating the image tracking information is used to determine an absolute camera position to correct for cumulative drift.