Mixed Media Reality Recognition with Image Tracking

• US 20150287228A1
• Filed: 06/22/2015
• Published: 10/08/2015
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Moraleda, Jorge, Erol, Berna, Hull, Jonathan J.
• Granted Patent: US 9,972,108 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

An MMR system integrating image tracking and recognition comprises a plurality of mobile devices, a pre-processing server or MMR gateway, and an MMR matching unit, and may include an MMR publisher. The MMR matching unit receives an image query from the pre-processing server or MMR gateway and sends it to one or more of the recognition units to identify a recognition result. Image tracking information also is provided for determining relative locations of images to each other. The mobile device includes an image tracker for providing at least a portion of the image tracking information. The disclosure also includes methods for image tracking-assisted recognition, recognition of multiple images using a single image query, and improved image tracking using MMR recognition.

Claims

1. A method comprising:

receiving a plurality of images;

combining the plurality of images into a merged image;

forming a single image query based upon the merged image;

submitting the single image query corresponding to the plurality of received images; and

receiving a recognition result associated with the single image query.

2. The method of claim 1, wherein combining the plurality of images into a merged image comprises:

receiving image tracking information indicating relative locations of the received images to each other; and

combining the plurality of images based on the image tracking information.

3. The method of claim 2, wherein the image tracking information comprises instructions for the combining.

4. The method of claim 1, wherein the merged image is a combined image stitched together from the plurality of images.

5. The method of claim 1, wherein the merged image is a super-resolution image synthesized from the plurality of images.

6. The method of claim 2, wherein the image tracking information includes sequence and relative timing information for the plurality of images.

7. The method of claim 1, wherein submitting the single image query corresponding to the plurality of received images further comprises:

extracting a set of features from each of the plurality of images;

combining the extracted sets of features to form a spatially-consistent superset of features; and

forming the single image query based upon the spatially-consistent superset of features.

8. The method of claim 7, wherein features extracted from images for the same location are combined into consensus features.

9. The method of claim 7, further comprising excluding from the superset extracted sets of features determined to be inconsistent with the consensus features.

10. The method of claim 2, further comprising cropping the recognition result based upon the image tracking information indicating the relative locations of the received images.

11. The method of claim 1, wherein the received plurality of images are from a video stream.

12. A system comprising:

one or more processors; and

a memory coupled with the one or more processors, the memory storing instructions, which when executed, cause the one or more processors to;

receive a plurality of images;

combine the plurality of images into a merged image;

form a single image query based upon the merged image;

submit the single image query corresponding to the plurality of received images; and

receive a recognition result associated with the single image query.

13. The system of claim 12, wherein to combine the plurality of images into a merged image, the instructions cause the one or more processors to:

receive image tracking information indicating relative locations of the received images to each other; and

combine the plurality of images based on the image tracking information.

14. The system of claim 12, wherein the merged image is a super-resolution image synthesized from the plurality of images.

15. The system of claim 12, wherein the merged image is a combined image stitched together from the plurality of images.

16. The system of claim 12, wherein to submit the single image query corresponding to the plurality of received images, the instructions cause the one or more processors to:

extract a set of features from each of the plurality of images;

combine the extracted sets of features to form a spatially-consistent superset of features; and

form the single image query based upon the spatially-consistent superset of features.

17. A computer program product comprising a non-transitory computer useable medium including a computer readable program, wherein the computer readable program, when executed on a computer, causes the computer to:

receive a plurality of images;

combine the plurality of images into a merged image;

form a single image query based upon the merged image;

submit the single image query corresponding to the plurality of received images; and

receive a recognition result associated with the single image query.

18. The computer program product of claim 17, wherein to combine the plurality of images into a merged image, the computer readable program causes the computer to:

receive image tracking information indicating relative locations of the received images to each other; and

combine the plurality of images based on the image tracking information.

19. The computer program product of claim 17, wherein the merged image is a super-resolution image synthesized from the plurality of images.

20. The computer program product of claim 17, wherein the merged image is a combined image stitched together from the plurality of images.