Mixed media reality recognition with image tracking

• US 9,972,108 B2
• Filed: 06/22/2015
• Issued: 05/15/2018
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Moraleda, Jorge, Erol, Berna, Hull, Jonathan J.
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): BROMELL, ALEXANDRIA Y
• Application Number:
  US14/746,669
• Publication Number:
  US 20150287228A1
• Time in Patent Office: 1,058 Days
• Field of Search: 382116, 382124, 382165, 382182, 382181, 707915, 707913, 707914
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

An MMR system integrating image tracking and recognition comprises a plurality of mobile devices, a pre-processing server or MMR gateway, and an MMR matching unit, and may include an MMR publisher. The MMR matching unit receives an image query from the pre-processing server or MMR gateway and sends it to one or more of the recognition units to identify a recognition result. Image tracking information also is provided for determining relative locations of images to each other. The mobile device includes an image tracker for providing at least a portion of the image tracking information. The disclosure also includes methods for image tracking-assisted recognition, recognition of multiple images using a single image query, and improved image tracking using MMR recognition.

Claims

1. A method comprising:

receiving a first image and a second image;

receiving image tracking information identifying a position, a rotation, and a scale of the first image relative to the second image, wherein the image tracking information includes sequence and relative timing information for the first image and the second image;

determining a first set of image features of the first image and a second set of image features of the second image;

combining the first set of image features and the second set of image features into a superset of image features using the image tracking information including the position, the rotation and the scale of the first image relative to the second image;

forming a single image query corresponding to the first image and the second image based on the superset of image features;

submitting the single image query corresponding to the first image and the second image; and

receiving a recognition result matching the single image query.

2. The method of claim 1, wherein the image tracking information comprises instructions for the combining.

3. The method of claim 1, further comprising:

combining the first image and the second image into a merged image based on the superset of image features.

4. The method of claim 3, wherein the merged image is a combined image stitched together from the first image and the second image.

5. The method of claim 3, wherein the merged image is a super-resolution image synthesized from the first image and the second image.

6. The method of claim 1, wherein the first set of image features and the second set of image features for the same location are combined into consensus features.

7. The method of claim 6, further comprising excluding from the superset of image features, features that are determined to be inconsistent with the consensus features.

8. The method of claim 1, further comprising cropping the recognition result based upon the image tracking information indicating the relative locations of the first image and the second image to each other.

9. The method of claim 1, wherein the first image and the second image are from a video stream.

10. A system comprising:

one or more processors; and

a memory coupled with the one or more processors, the memory storing instructions, which when executed, cause the one or more processors to;

receive a first image and a second image;

receive image tracking information identifying a position, a rotation, and a scale of the first image relative to the second image, wherein the image tracking information includes sequence and relative timing information for the first image and the second image;

determine a first set of image features of the first image and a second set of image features of the second image;

combine the first set of image features and the second set of image features into a superset of image features using the image tracking information including the position, the rotation and the scale of the first image relative to the second image;

form a single image query corresponding to the first image and the second image based on the superset of image features;

submit the single image query corresponding to the first image and the second image; and

receive a recognition result matching the single image query.

11. The system of claim 10, wherein the instructions further cause the one or more processors to:

combine the first image and the second image into a merged image based on the superset of image features.

12. The system of claim 11, wherein the merged image is a combined image stitched together from the first image and the second image.

13. The system of claim 11, wherein the merged image is a super-resolution image synthesized from the first image and the second image.

14. A computer program product comprising a non-transitory computer useable medium including a computer readable program, wherein the computer readable program, when executed on a computer, causes the computer to:

receive a first image and a second image;

receive image tracking information identifying a position, a rotation, and a scale of the first image relative to the second image, wherein the image tracking information includes sequence and relative timing information for the first image and the second image;

determine a first set of image features of the first image and a second set of image features of the second image;

combine the first set of image features and the second set of image features into a superset of image features using the image tracking information including the position, the rotation and the scale of the first image relative to the second image;

form a single image query corresponding to the first image and the second image based on the superset of image features;

submit the single image query corresponding to the first image and the second image; and

receive a recognition result matching the single image query.

15. The computer program product of claim 14, wherein the computer readable program further causes the computer to:

combine the first image and the second image into a merged image based on the superset of image features.

16. The computer program product of claim 15, wherein the merged image is a combined image stitched together from the first image and the second image.