Method And System For Document Fingerprint Matching In A Mixed Media Environment

• US 20060285172A1
• Filed: 07/31/2006
• Published: 12/21/2006
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Piersol, Kurt, Hart, Peter E., Hull, Jonathan J., Erol, Berna, Lee, Dar-Shyang
• Granted Patent: US 8,335,789 B2
• Time in Patent Office: Days
• US Class Current: 358/448
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system provides for document fingerprint matching.

Claims

1. A method of image matching, comprising:

capturing an image of at least part of a first media type;

extracting features from the captured image, wherein the extracted features comprise a multi-dimensional arrangement of content in the captured image; and

matching the extracted features against a collection of document pages to identify a location of the captured image in at least one of the document pages.

2. The method of claim 1, wherein the multi-dimensional arrangement comprises a horizontal arrangement of at least one of words and characters in the captured image and a vertical arrangement of at least one of words and characters in the captured image.

3. The method of claim 1, wherein the first media type is a paper document.

4. The method of claim 1, further comprising:

identifying a second media type dependent on the location.

5. The method of claim 4, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

6. The method of claim 1, wherein extracting features comprises:

determining a first length of a first word in the captured image; and

associating the first length with a second length of a second word one of below and above the first word.

7. The method of claim 1, wherein extracting features comprises:

representing content within a first word region with at least one of a first value; and

representing a space between the first word region and a second word region with at least one of a second value.

8. The method of claim 1, wherein extracting features comprises:

identifying a first feature point in the captured image;

identifying a second feature point in the captured image; and

determining a relationship between the first feature point and the second feature point.

9. The method of claim 1, wherein the relationship is at least one of distance and angle.

10. The method of claim 1, wherein the extracting features comprises:

determining a length of a first word region in the captured image;

dividing the first word region into segments based on the length;

determining presence of a second word region in the captured image at least one of below and above the first word region; and

associating a value to the first word region based on an overlap of the segments with the second word region.

11. The method of claim 1, wherein the extracting features comprises:

horizontally classifying content in the captured image;

vertically classifying content in the captured image; and

combining the horizontal and vertical classification to identify the location of the captured image in the at least one of the document pages.

12. A system for image matching, comprising:

a capture device operable to capture an image of at least part of a first media type;

a feature extraction module operable to extract features from the captured image, wherein the extracted features comprise a multi-dimensional arrangement of content in the captured image; and

a database operable to store a collection of document pages,

wherein the feature extraction module is further operable to convert the multi-dimensional arrangement into a symbolic representation matchable against the collection of document pages to identify a location of the captured image in at least one of the document pages.

13. The system of claim 12, wherein the captured device comprises the feature extraction module.

14. The system of claim 12, wherein the database comprises the feature extraction module.

15. The system of claim 12, wherein the first media type is a paper document.

16. The system of claim 12, wherein a second media type is communicated to the capture device based on identification of the location.

17. The system of claim 16, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

18. The system of claim 12, wherein the feature extraction module is further operable to:

determine a first length of a first word in the captured image; and

associate the first length with a second length of a second word one of below and above the first word.

19. The system of claim 12, wherein the feature extraction module is further operable to:

represent content within a first word region with at least one of a first value; and

represent a space between the first word region and a second word region with at least one of a second value.

20. The system of claim 12, wherein the feature extraction module is further operable to:

identify a first feature point in the captured image;

identify a second feature point in the captured image; and

determine a relationship between the first feature point and the second feature point.

21. The system of claim 20, wherein the relationship is at least one of distance and angle.

22. The system of claim 12, wherein the feature extraction module is further operable to:

determine a length of a first word region in the captured image;

divide the first word region into segments based on the length;

determine presence of a second word region in the captured image at least one of below and above the first word region; and

associate a value to the first word region based on an overlap of the segments with the second word region.

23. The system of claim 12, wherein the feature extraction module is further operable to:

horizontally classify content in the captured image;

vertically classify content in the captured image; and

combine the horizontal and vertical classification to identify the location of the captured image in the at least one of the document pages.

24. The system of claim 12, wherein the capture device comprises one selected from the group consisting of a cellular camera phone, a Personal Digital Assistant (PDA) device, a digital camera, a barcode reader, a radio frequency identification (RFID) reader, a computer peripheral, a web camera, and a video card.

25. A method of interacting a first media type with a second media type, comprising:

capturing with a capture device an image of at least part of the first media type;

extracting features from the captured image, wherein the extracted features comprises a multi-dimensional arrangement of content within the captured image;

converting the multi-dimensional arrangement into a symbolic representation;

matching the symbolic representation against a collection of document pages;

identifying a location of the image in at least one of the document pages in response to positively matching the symbolic representation; and

providing a second media type based on the identification.

26. The method of claim 25, wherein the first media type is a paper document.

27. The method of claim 25, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

28. The method of claim 25, further comprising:

performing an action based on the identification, wherein the action comprises at least one selected from the group consisting of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, and presenting information.

29. The method of claim 25, wherein extracting features comprises:

determining a first length of a first word in the captured image; and

associating the first length with a second length of a second word one of below and above the first word.

30. The method of claim 25, wherein extracting features comprises:

representing content within a first word region with at least one of a first value; and

representing a space between the first word region and a second word region with at least one of a second value.

31. The method of claim 25, wherein extracting features comprises:

identifying a first feature point in the captured image;

identifying a second feature point in the captured image; and

determining a relationship between the first feature point and the second feature point.

32. The method of claim 31, wherein the relationship is at least one of distance and angle.

33. The method of claim 25, wherein the extracting features comprises:

determining a length of a first word region in the captured image;

dividing the first word region into segments based on the length;

determining presence of a second word region in the captured image at least one of below and above the first word region; and

associating a value to the first word region based on an overlap of the segments with the second word region.

34. The method of claim 25, wherein the extracting features comprises:

horizontally classifying content in the captured image;

vertically classifying content in the captured image; and

combining the horizontal and vertical classification to identify the location of the captured image in the at least one of the document pages.