Method and system for position-based image matching in a mixed media environment

• US 8,332,401 B2
• Filed: 07/31/2006
• Issued: 12/11/2012
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Hart, Peter E., Lee, Dar-Shyang, Piersol, Kurt
• Primary Examiner(s): Wilson, Kimberly
• Assistant Examiner(s): Liu, Hexing
• Application Number:
  US11/461,294
• Publication Number:
  US 20060262962A1
• Time in Patent Office: 2,325 Days
• Field of Search: 707/749, 707/736, 707/738
• US Class Current: 707/736
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system provides for position-based image matching.

Claims

1. A method of image matching, comprising:

receiving from a capture device a first image of at least part of a first media type at a first position, wherein the first media type is a paper document;

matching a first feature, a second feature and a geometric relationship between the first feature and the second feature against a collection of document pages in a database;

in response to positively matching the first feature, the second feature and the geometric relationship, generating a list of document pages within which matches to the first feature, the second feature and the geometric relationship are located;

retrieving a third feature around an image patch from a first page in the list of document pages that, if present in the first media type, would result in identifying the first page in the list of document pages that best matches the first media type;

determining a second position for the capture device based on the third feature around the image patch from the first page;

sending an instruction to the capture device for moving the capture device to the second position to capture a second image that includes the third feature;

receiving from the capture device the second image that includes the third feature;

tracking the second position of the capture device over the second image based on the first position of the first image;

matching the second image and the second position of the capture device against the first page in the list of document pages; and

in response to positively matching the second image and the second position of the capture device, returning the first page within which matches to the first and second image are located and a second media type that is associated with the first page, the second media type comprising at least one of a data structure, a command, text, audio, video, a third image, a digital photograph, web link text, an application file, updated information or a service.

2. The method of claim 1, further comprising determining an x-y location of the first feature and the second feature.

3. The method of claim 1, wherein the first feature and the second feature each comprise one or more of words, characters and blocks.

4. The method of claim 1, wherein the geometric relationship includes at least one of an angle and a distance between the first feature and the second feature.

5. The method of claim 1, further comprising:

maintaining a history of tracking the first and the second positions of the capture device over the first media type.

6. The method of claim 1, further comprising:

predicting the second position of the capture device over the first media type.

7. The method of claim 1, further comprising:

performing an action in response to positively matching the second image and the second position of the capture device, wherein the action comprises at least one of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information or presenting information.

8. A system for image matching, comprising:

a processor;

a position tracking module coupled to the processor and operable to receive a sequence of images from a capture device of a first media type, wherein the first media type is a paper document and the sequence of images comprises a first image at a first position of the capture device and a second image at a second position of the capture device, the position tracking module operable to track the second position of the capture device over the second image based on the first position of the first image;

a classification module coupled to the processor to determine a list of matching document pages within which matches to a first feature, a second feature and a geometric relationship between the first feature and the second feature in the first image are located, to retrieve a third feature around an image patch from a first page in the list of matching document pages that, if present in the first media type, would result in identifying the first page in the list of document pages that best matches the first media type, to determine the second position for the capture device based on the third feature around the image patch from the first page, to match the second image and the second position of the capture device against the first page in the list of document pages;

a database coupled to the classification module and operable to store a collection of document pages and return the first page within which matches to the first and second images are located and a second media type that is associated with the first page, the second media type comprising at least one of a data structure, a command, text, audio, video, a third image, a digital photograph, web link text, an application file, updated information or a service; and

a control structure coupled to the processor and operable to send an instruction to the capture device for moving the capture device to the second position to capture the second image that includes the third feature.

9. The system of claim 8, further comprising a feature extraction module coupled to the processor to determine an x-y location of the first feature and the second feature.

10. The system of claim 8, wherein the first feature and the second feature each comprise one or more of words, characters and blocks.

11. The system of claim 8, wherein the geometric relationship includes at least one of an angle and a distance between the first feature and the second feature.

12. The system of claim 8, wherein the capture device comprises at least one of a cellular camera phone, a Personal Digital Assistant (PDA) device, a digital camera, a barcode reader, a radio frequency identification (RFID) reader, a computer peripheral, a web camera or a video card.

13. The system of claim 8, wherein the position tracking module is further operable to:

maintain a history of tracking the first and the second position of the capture device over the first media type.

14. The system of claim 8, wherein the position tracking module is further operable to:

predict the second position of the image capture device over the first media type.

15. A method of providing interaction between a first media type and a second media type, comprising:

receiving from a capture device a sequence of images of at least parts of a first media type, wherein the first media type is a paper document and the sequence of images comprises a first image at a first position and a second image at a second position;

matching a first feature, a second feature and a geometric relationship between the first feature and the second feature from the first image against a collection of document pages in a database;

in response to positively matching the first feature, the second feature and the geometric relationship, generating a list of document pages within which matches to the first feature and the geometric relationship are located;

retrieving a third feature around an image patch from a first page of the list of matching document pages that, if present in the first media type, would result in identifying the first page in the list of document pages that best matches the first media type;

determining the second position for the capture device based on the third feature around the image patch from the first page;

sending an instruction to the capture device for moving the capture device to the second position to capture the second image that includes the third feature;

tracking the second position of the capture device over the second image based on the first position of the first image; and

in response to positively matching the second image and the second position of the capture device against the first page in the list of document pages, performing an action associated with the second media type, the second media type comprising at least one of a data structure, a command, text, audio, video, a third image, a digital photograph, web link text, an application file, updated information or a service.

16. The method of claim 15, further comprising determining an x-y location of the first feature and the second feature.

17. The method of claim 15, wherein the geometric relationship includes at least one of an angle and a distance between the first feature and the second feature.

18. The method of claim 15, wherein performing the action based on the identification, the second image and the second position of the image capture device further comprises:

at least one of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information or presenting information.

19. The method of claim 15, further comprising:

maintaining a history of tracking the first and the second position of the capture device over the first media type.

20. The method of claim 15, further comprising:

predicting the second position of the capture device over the first media type based on the captured sequence of images.