Method and system for multi-tier image matching in a mixed media environment

• US 8,521,737 B2
• Filed: 07/31/2006
• Issued: 08/27/2013
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hart, Peter E., Erol, Berna, Hull, Jonathan J., Piersol, Kurt
• Primary Examiner(s): Wilson, Kimberly
• Assistant Examiner(s): Liu, Hexing
• Application Number:
  US11/461,300
• Publication Number:
  US 20060262976A1
• Time in Patent Office: 2,584 Days
• Field of Search: 707/749, 707/736, 707/738
• US Class Current: 707/736
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system provides for multi-tier image matching.

Claims

1. A computer-implemented method of image matching, the method comprising:

receiving, with one or more processors, a first image of at least part of a document in a first media type;

identifying, with the one or more processors, a culling feature in the first image by comparing the first image to a library of images, the library of images comprising at least one image that includes the culling feature;

identifying, with the one or more processors, a location of the culling feature in the first image;

selecting, with the one or more processors, a set of documents that include the culling feature and the location from a collection of stored documents indexed by the culling feature;

receiving, with the one or more processors, a second image of at least part of the document in the first media type that is distinct from the first image;

identifying, with the one or more processors, horizontal and vertical bounding boxes in the second image that include horizontally and vertically adjacent pairs of words;

determining, with the one or more processors, a subset of documents from the selected set of documents by identifying a number of overlaps of the horizontal and vertical bounding boxes and locations of centers of the overlaps in the selected set of documents, wherein each document in the subset includes at least one of the horizontally and vertically adjacent pairs of words; and

ranking, with the one or more processors, the subset of documents based on the number of overlaps of the horizontal and vertical bounding boxes and the locations of the centers of the overlaps.

2. The method of claim 1, wherein the first media type is a paper document.

3. The method of claim 1, wherein the culling feature comprises at least one selected from the group consisting of a numeric-only bar code, an alphanumeric bar code, a 2-dimensional bar code, a graphic, an image, a trademark, a color, a combination of colors, a keyword, a logo, and a phrase.

4. The method of claim 1, wherein determining the subset of documents comprises:

extracting a feature from the second image; and

comparing the extracted feature with features contained in the selected set of documents to determine the subset of documents from the selected set of documents.

5. The method of claim 1, further comprising:

responsive to determining the subset of documents, providing a second media type.

6. The method of claim 5, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

7. The method of claim 1, further comprising:

responsive to determining the subset of documents, performing an action, wherein the action comprises at least one selected from the group consisting of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, and presenting information.

8. A system for image matching, comprising:

one or more processors;

a memory coupled to the one or more processors; and

a database including stored instructions executable by the one or more processors, the database operable to receive a first image of at least part of a document in a first media type, to identify a culling feature in the first image by comparing the first image to a library of images, the library of images comprising at least one image that includes the culling feature, to identify a location of the culling feature in the first image, to select a set of documents that include the culling feature and the location from a collection of documents indexed by the culling feature and stored within the database, to receive a second image of at least part of the document in the first media type that is distinct from the first image, to identify horizontal and vertical bounding boxes in the second image that include horizontally and vertically adjacent pairs of words, to determine a subset of documents from the selected set of documents by identifying a number of overlaps of the horizontal and vertical bounding boxes and locations of centers of the overlaps in the selected set of documents, wherein each document in the subset includes at least one of the horizontally and vertically adjacent pairs of words and to rank the subset of documents based on the number of overlaps of the horizontal and vertical bounding boxes and the locations of the centers of the overlaps.

9. The system of claim 8, further comprising a capture device that comprises one selected from the group consisting of a cellular camera phone, a Personal Digital Assistant (PDA) device, a digital camera, a barcode reader, a radio frequency identification (RFID) reader, a computer peripheral, a web camera, and a video card.

10. The system of claim 8, wherein the first media type is a paper document.

11. The system of claim 8, wherein the culling feature comprises at least one selected from the group consisting of a numeric-only bar code, an alphanumeric bar code, a 2-dimensional bar code, a graphic, an image, a trademark, a color, a combination of colors, a keyword, a logo, and a phrase.

12. The system of claim 8, wherein the one or more processors are

operable to perform an action responsive to the determination of the subset of documents.

13. The system of claim 12, wherein the action comprises at least one selected from the group consisting of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, and presenting information.

14. The system of claim 8, further comprising a capture device that is operable to receive and output a second media type responsive to the determination of the subset of documents.

15. The system of claim 14, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

16. The system of claim 14, wherein the database comprises:

a base media storage operable to store the collection of documents; and

a mixed media storage operable to store the second media type.

17. The system of claim 8, further comprising a capture device operable to capture the first image and to identify the culling feature and the location in the first image.

18. A computer-implemented method of providing interaction between a first media type with a second media type, comprising:

receiving, with one or more processors, a first image of at least part of a document in the first media type;

identifying, with the one or more processors, a culling feature in the first image by comparing the first image to a library of images, the library of images comprising at least one image that includes the culling feature;

identifying, with the one or more processors, a location of the culling feature in the first image;

selecting, with the one or more processors, a set of documents that include the culling feature and the location from a document database indexed by the culling feature;

receiving, with the one or more processors, a second image of at least part of the document in the first media type that is distinct from the first image;

identifying, with the one or more processors, horizontal and vertical bounding boxes in the second image that include horizontally and vertically adjacent pairs of words;

determining, with the one or more processors, a subset of documents from the selected set of documents by identifying a number of overlaps of the horizontal and vertical bounding boxes and locations of centers of the overlaps in the selected set of documents, wherein each document in the subset includes at least one of the horizontally and vertically adjacent pairs of words;

ranking, with the one or more processors, the subset of documents based on the number of overlaps of the horizontal and vertical bounding boxes and the locations of the centers of the overlaps; and

providing, with the one or more processors, a second media type associated with a highest ranked document from the subset of documents responsive to determining the subset of documents.

19. The method of claim 18, wherein the first media type is a paper document.

20. The method of claim 18, wherein the culling feature comprises at least one selected from the group consisting of a numeric-only bar code, an alphanumeric bar code, a 2-dimensional bar code, a graphic, an image, a trademark, a color, a combination of colors, a keyword, a logo, and a phrase.

21. The method of claim 18, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

22. The method of claim 18, further comprising:

performing an action responsive to determining the subset of documents, wherein the action comprises at least one selected from the group consisting of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, and presenting information.

23. A computer program product comprising a non-transitory computer readable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive a first image of at least part of a document in a first media type;

identify a culling feature in the first image by comparing the first image to a library of images, the library of images comprising at least one image that includes the culling feature;

identify a location of the culling feature in the first image;

select a set of documents that includes the culling feature and the location from a collection of documents indexed by the culling feature and stored within a database that includes the culling feature and the location;

receive a second image of at least part of the document in the first media type that is distinct from the captured first image;

identify horizontal and vertical bounding boxes in the second image that include horizontally and vertically adjacent pairs of words;

determine a subset of documents from the selected set of documents by identifying a number of overlaps of the horizontal and vertical bounding boxes and locations of centers of the overlaps in the selected set of documents, wherein each document in the subset includes at least one of the horizontally and vertically adjacent pairs of words; and

rank the subset of documents based on the number of overlaps of the horizontal and vertical bounding boxes and the locations of the centers of the overlaps.