METHOD AND SYSTEM FOR IMAGE MATCHING IN A MIXED MEDIA ENVIRONMENT

• US 20060262352A1
• Filed: 07/31/2006
• Published: 11/23/2006
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Piersol, Kurt, Hart, Peter E., Hull, Jonathan J., Erol, Berna, Lee, Dar-Shyang
• Granted Patent: US 8,600,989 B2
• Time in Patent Office: Days
• US Class Current: 358/1.180
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system provides for image matching portions of a document.

Claims

1. A computer-implemented method of image matching, comprising:

capturing with a capture device an image of at least part of a first media type;

matching the image against a collection of document pages in a database; and

in response to positively matching the image, returning at least one location within at least one of the document pages where the image is located.

2. The computer-implemented method of claim 1, wherein the at least one location is specified by x, y coordinates for the at least one of the document pages.

3. The computer-implemented method of claim 1, further comprising:

returning a confidence value associated with the returned at least one location.

4. The computer-implemented method of claim 1, further comprising:

associating the returned at least one location with a second media type.

5. The computer-implemented method of claim 4, further comprising:

providing the second media type to the capture device.

6. The computer-implemented method of claim 4, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

7. The computer-implemented method of claim 1, wherein the first media type is a paper document.

8. A system for image matching, comprising:

a capture device operable to capture an image of at least part of a first media type;

a feature extraction module operable to convert the captured image into a symbolic representation; and

a classification module operable to convert the symbolic representation into an identification of at least one document page and a location within the at least one document page where the image occurs.

9. The system of claim 8, further comprising:

a database operable to store a collection of document pages, the database being in operable communication with the classification module.

10. The system of claim 8, wherein the capture device comprises the feature extraction module.

11. The system of claim 8, further comprising:

a quality assessment module operable to assess content of the captured image, wherein the assessment is dependent on at least one of a need and a capability of the system.

12. The system of claim 11, wherein the assessment is dependent on whether the content of the captured image contains text.

13. The system of claim 11, wherein the assessment is dependent on a sharpness of text in the captured image.

14. The system of claim 11, wherein the assessment is dependent on whether the captured image contains a non-document image.

15. The system of claim 11, wherein the assessment is dependent on a functionality of the feature extraction module.

16. The system of claim 11, wherein an operation of the capture device is adjustable dependent on the assessment.

17. The system of claim 8, further comprising:

an image processing module operable to modify content of the captured image.

18. The system of claim 8, further comprising:

a position tracking module operable to track movement of the capture device over the first media type.

19. The system of claim 8, wherein the first media type is a paper document.

20. The system of claim 8, further comprising:

a control structure operable to control information and command flow through the system.

21. The system of claim 8, wherein the identification is associated with a second media type.

22. The system of claim 21, wherein the capture device is further operable to at least one of receive and output the second media type.

23. The system of claim 21, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

24. The system of claim 8, wherein the capture device comprises one selected from the group consisting of a cellular camera phone, a Personal Digital Assistant (PDA) device, a digital camera, a barcode reader, a radio frequency identification (RFID) reader, a computer peripheral, a web camera, and a video card.

25. A computer-implemented method of interacting a first media type with a second media type, comprising:

capturing with a capture device an image of at least part of the first media type;

validating content of the captured image as being reliably processable;

converting the captured image into a symbolic representation in response to the validation;

converting the symbolic representation into an identification of at least one document page and a location therein where the image occurs; and

providing the second media type to the capture device dependent on the identification.

26. The computer-implemented method of claim 25, wherein the first media type is a paper document.

27. The computer-implemented method of claim 25, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

28. The computer-implemented method of claim 25, further comprising:

associating a confidence value with the identification.

29. The computer-implemented method of claim 25, further comprising:

performing an action dependent on the identification, wherein the action comprises at least one selected from the group consisting of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, and presenting information.

30. The computer-implemented method of claim 25, further comprising:

maintaining a collection of document pages comprising the at least one document page.

31. A computer-readable medium having processor-executable instructions stored therein, the instructions comprising instructions to:

receive an image captured by a capture device, the image being of at least part of a first media type;

match a representation of the image against a collection of document pages stored in a database; and

determine a location within at least one of the document pages where the image is located.

32. The computer-readable medium of claim 31, further comprising instructions to:

assess a quality of the received image; and

send a command to the capture device dependent on the quality assessment.

33. The computer-readable medium of claim 31, further comprising instructions to:

modify the image prior to being handled by the instructions to match the image against the collection of document pages.

34. The computer-readable medium of claim 31, further comprising instructions to:

communicate a second media type to the capture device in response to determining the location where the image is located.

35. The computer-readable medium of claim 31, wherein the first media type is a paper document.

36. The computer-readable medium of claim 31, wherein the second media type comprises least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

37. The computer-readable medium of claim 31, further comprising instructions to:

extract features from the image; and

form the representation of the image in response to the feature extraction.