System and methods for creation and use of a mixed media environment

• US 7,702,673 B2
• Filed: 07/31/2006
• Issued: 04/20/2010
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Piersol, Kurt Wesley, Graham, Jamey, Hart, Peter E., Hull, Jonathan J., Erol, Berna, Lee, Dar-Shyang
• Primary Examiner(s): Alam; Shahid A
• Assistant Examiner(s): Bromell; Alexandria Y
• Application Number:
  US11/461,017
• Publication Number:
  US 20060285772A1
• Time in Patent Office: 1,359 Days
• Field of Search: 707/3, 707/602, 707/707, 707/728, 707/736, 707/793, 707/802, 348/211.2, 705/27
• US Class Current: 707/707
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 40/12 Use of codes for handling textual entities
• CPC Class Codes: G06F 40/123 Storage facilities
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system includes an MMR user, a MMR computer, a user printer that produces a printed document, a networked media server, an office portal, a service provider server, an electronic display that is electrically connected to a set-top box, a document scanner, a network, a capture device, a cellular infrastructure, wireless fidelity (Wi-Fi) technology, Bluetooth® technology, infrared (IR) technology, wired technology, and a geo location mechanism. The MMR system of the present invention provides mechanisms for forming a mixed media document that includes media of at least two types, such as printed paper as a first medium and a digital photograph, digital movie, digital audio file, or web link as a second medium. Furthermore, the MMR system of the present invention facilitates business methods that take advantage of the combination of a portable electronic device, such as a cellular camera phone, and a paper document.

Claims

1. A system for creating a mixed media environment that utilizes a first type of medium and a second type of medium, the system comprising:

a capture device that captures an image patch comprising a portion less than an entire page of a first type of print medium and of unknown location with respect to the entire page;

a mixed media processor communicatively coupled to the capture device to receive the image patch and to compare the captured image patch to a previously generated mixed media reality document data structure, the mixed media reality document data structure comprising;

a first representation of the first type of print medium comprising a matchable portion for use in pattern matching to a corresponding first location within the captured image patch;

an action or a second type of medium available for presentation with a previously stored image that is representative of the first type of print medium, wherein the action or second type of medium is related to the captured image patch; and

an index correlating the matchable portion of the first representation with the action or the second type of medium and a second location in the previously stored image; and

a media storage communicatively coupled to the mixed media processor that stores the mixed media reality document data structure.

2. The system of claim 1, further comprising a communication mechanism for coupling the capture device to the mixed media processor.

3. The system of claim 2, wherein the communication mechanism is one from the group of a network, a wired coupling, a infrared link, a Bluetooth link, a wireless communication link, a cellular communication link, a data communication link, a mobile data service, a Global System for Mobile Communications, and a public switched telephone network.

4. The system of claim 1, wherein the first type of print medium is a paper document, and the second type of medium is one from the group of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information and services.

5. The system of claim 1, wherein the mixed media processor is adapted to relate the image patch to the mixed media reality document data structure and perform a responsive action in response to a relationship between the image patch and the mixed media reality document data structure.

6. The system of claim 5, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

7. The system of claim 1, wherein the capture device includes a portable input device capable of capturing digital data or an image representing a portion of the first type of print medium.

8. The system of claim 1, wherein the capture device includes a portable output device capable of outputting the first type of print medium.

9. The system of claim 1, wherein the capture device is one from the group of a cellular camera phone;

a Personal Digital Assistant device;

a digital camera;

a barcode reader;

a radio frequency identification (RFID) reader;

a computer peripheral;

a web camera;

or a video card.

10. The system of claim 1, further comprising a memory coupled to the mixed media processor, the memory for storing instructions and/or data executable by the mixed media processor, the memory storing a content-based retrieval component that retrieves content-based information from a text or image patch.

11. The system of claim 1, further comprising a base media storage for storing the second type of medium in an original form, the base media storage coupled to the mixed media processor.

12. The system of claim 1, wherein the media storage stores a plurality of mixed media reality document data structures each having a first representation of the first type of print medium and an index, the media storage coupled to the mixed media processor.

13. The system of claim 1, further comprising an output device for outputting the second type of medium in response to a signal from the mixed media processor, the output device coupled to the mixed media processor.

14. The system of claim 1, wherein the image patch comprises a text finger print of a printed document having at least two lines that is usable for pattern matching.

15. The system of claim 1, wherein the image patch uniquely identifies a location in a printed document.

16. The system of claim 15, wherein the image patch comprises a text finger print of a printed document having a portion of a first line and a portion of an adjacent second line.

17. The system of claim 1, wherein the mixed media reality document data structure further comprises a plurality of files of the second type of medium.

18. The system of claim 1, wherein the action comprises text to be presented or one or more commands to be executed, and wherein the index is a link between the text to be presented or one or more commands to be executed and the previously stored image.

19. The system of claim 1, wherein the previously stored image that is representative of the first type of print medium is an electronic representation of a document in the first type of print medium.

20. The system of claim 1, wherein the mixed media reality document data structure has a plurality of previously stored images each having a corresponding index.

21. A capture device for a mixed media system, the capture device comprising:

an input device that captures an image patch comprising a portion less than an entire page of a first print media type and of unknown location with respect to the entire page;

a communication transceiver coupled to the input device and an output device that sends and receives information and commands to and from the mixed media system; and

a mixed media processor that compares the captured image patch of the first print media type to of one or more previously generated mixed media reality document data structures, each mixed media reality document data structure comprising;

a first representation of the first print media type comprising a matchable portion for use in pattern matching corresponding to a first location within the captured image patch;

an action or a second media type available for presentation with a previously stored image that is representative of the first print media type, wherein the action or second media type is related to the captured image patch; and

an index correlating the matchable portion of the first representation with the action or the second media type and a second location in the previously stored image.

22. The system of claim 21, wherein the first print media type is a printed document.

23. The system of claim 21, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

24. The system of claim 21, wherein the output device is further adapted for presenting a second media type, the output device coupled to the mixed media processor to receive instructions and data to be output by the output device.

25. The system of claim 24, wherein the action is outputting the second media type, and the second media type is one from the group of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information and services.

26. The system of claim 21, wherein the mixed media processor includes a content-based retrieval component that retrieves content-based information from a text or image patch, and wherein the image patch of the first print media type is a patch of text.

27. The system of claim 21, wherein the capture device includes a storage device for storing the one or more mixed media reality document data structures.

28. The system of claim 21, wherein the mixed media processor compares the image patch of the first print media type to the corresponding first representation by comparing a text finger print of a printed document associated with the first representation having a portion of a first line and a portion of an adjacent second line.

29. A device for relating a first print media type and a second media type, the device comprising:

a mixed media processor having an input and an output that compares a captured image patch of a first print media type to plurality of one or more mixed media reality document data structures, each mixed media reality document data structure comprising;

a first representation of the first print media type comprising a matchable portion for use in pattern matching to a corresponding first location within the image wherein the captured image patch comprises a portion less than an entire page of the first print media type and is of unknown location with respect to the entire page;

an action or second media type available for presentation with a previously stored image that is representative of the first print media type wherein the action or second media type is related to the captured image patch; and

an index correlating the matchable portion of the first representation with the action or the second media type and a second location in the previously stored image

the input of the mixed media processor coupled to receive the image patch; and

a storage device that stores the of one or more mixed media reality document data structures, the storage device coupled to the mixed media processor.

30. The device of claim 29, further comprising a communication mechanism for coupling a capture device to the mixed media processor.

31. The device of claim 29, wherein the first print media type is a paper document, and at least one mixed media reality document data structure includes a second media type that is one from the group of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information and services.

32. The device of claim 29, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

33. The device of claim 29, further comprising a memory coupled to the mixed media processor, the memory for storing instructions and/or data executable by the mixed media processor, the memory storing a content-based retrieval component that retrieves content-based information from a text or image patch.

34. The device of claim 29, wherein the storage device stores second media types in their original form.

35. The device of claim 29, wherein the previously stored image that is representative of the first print media type is an electronic representation of a document of the first print media type.

36. The device of claim 29, wherein the image patch comprises a text finger print of a printed document having at least two lines that is usable for pattern matching with the first representation.

37. A method for providing a mixed media environment for interacting with documents of a first print media type and a second media type, the method comprising:

creating a first mixed media reality document data structure, comprising;

a first representation a document of a first print media type comprising a matchable portion for use in pattern matching to a corresponding first location within a captured image patch of the document, wherein the image patch comprises a portion less than an entire page of the document and is of unknown location with respect to the entire page;

an action or second media type available for presentation with a previously stored image that is representative of the first print media type, wherein the action or second media type is related to the captured image patch;

and an index correlating the matchable portion of the first representation with the action or the second media type and a second location in the previously stored image, wherein creating further comprises;

generating the first representation of the document of the first print media type;

indexing the generated first representation of the document;

capturing the image patch from the document of the first print media type;

comparing the image patch to the generated first representation; and

performing the action specified by the mixed media reality document data structure if the image patch is similar to the generated first representation.

38. The method of claim 37, wherein the generating the first representation of the first print media type comprises automatically capturing a text layout of a printed document.

39. The method of claim 38, wherein the automatically capturing is done by one from a group of a printed document capture module, an operating system, a print driver, a scanner, a file transfer mechanism.

40. The method of claim 37, wherein the generated first representation is one from the group of a representation of the entire document, a representation of patch of text, a representation of single word, a representation of a section of an image, and any other representation of a matchable portion of the document.

41. The method of claim 37, wherein indexing the generated first representation includes association of one from the group of the index, an electronic representation of the document, a second media type, and an action, with the generated first representation.

42. The method of claim 37, wherein indexing the generated first representation includes one from the group of specifying x-y locations of characters of a printed document, specifying x-y locations of words of the printed document, specifying x-y locations of an image or a portion of an image in a printed document, performing an OCR operation and specifying x-y locations of characters and/or words, performing feature extraction and specifying x-y locations of characters and/or words.

43. The method of claim 37, wherein capturing the image patch comprises one from the group of scanning a patch of text, scanning a patch of text and preprocessing to determine the likelihood that an intended feature description is extracted correctly, scanning a machine-readable identifier, inputting data that identifies the document, receiving email with the document attached, receiving the document by file transfer, scanning a portion of an image, and inputting the document.

44. The method of claim 37, wherein comparing the image patch to the generated first representation further comprises performing pattern matching between the image patch and the generated first representation.

45. The method of claim 44, wherein the pattern matching is performed by comparing a document fingerprint of the image patch to a document fingerprint of the generated first representation.

46. The method of claim 45, wherein comparing the document fingerprint includes extracting features from a captured image, composing descriptors from those features, and looking up a document and patch that contain a percentage of those descriptors.

47. The method of claim 37, wherein performing an action specified by the mixed media reality document data structure comprises retrieving a predetermined action that is associated with the mixed media reality document data structure having the generated first representation.

48. The method of claim 37, wherein the action is one from the group of retrieving information, writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, and input of a command to another system.