SYSTEM AND METHODS FOR CREATION AND USE OF A MIXED MEDIA ENVIRONMENT

• US 20060285772A1
• Filed: 07/31/2006
• Published: 12/21/2006
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Piersol, Kurt, Hart, Peter E., Hull, Jonathan J., Erol, Berna, Lee, Dar-Shyang
• Granted Patent: US 7,702,673 B2
• Time in Patent Office: Days
• US Class Current: 382/305
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 40/12 Use of codes for handling textual entities
• CPC Class Codes: G06F 40/123 Storage facilities
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system includes an MMR user, a MMR computer, a user printer that produces a printed document, a networked media server, an office portal, a service provider server, an electronic display that is electrically connected to a set-top box, a document scanner, a network, a capture device, a cellular infrastructure, wireless fidelity (Wi-Fi) technology, Bluetooth® technology, infrared (IR) technology, wired technology, and a geo location mechanism. The MMR system of the present invention provides mechanisms for forming a mixed media document that includes media of at least two types, such as printed paper as a first medium and a digital photograph, digital movie, digital audio file, or web link as a second medium. Furthermore, the MMR system of the present invention facilitates business methods that take advantage of the combination of a portable electronic device, such as a cellular camera phone, and a paper document.

Claims

1. A system for creating a mixed media environment that utilizes a first type of medium and a second type of medium, the system comprising:

a capture device for capturing a first representation of the first type of medium;

a mixed media processor for relating the first representation to a mixed media document having a second representation of the first type of medium and an index; and

a media storage for a storing the mixed media document.

2. The system of claim 1, further comprising a communication mechanism for coupling the capture device to the mixed media processor.

3. The system of claim 1, where in the communication mechanism is one from the group of a network, a wired coupling, a infrared link, a Bluetooth link, a wireless communication link, a cellular communication link, a data communication link, a mobile data service, a Global System for Mobile Communications, and a public switched telephone network.

4. The system of claim 1, wherein the first type of medium is a paper document, and the second type of medium is one from the group of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information and services.

5. The system of claim 1, wherein the mixed media processor is adapted to compare the first representation with the second representation to relate the first representation to the mixed media document, and the mixed media processor performs an action in response to a relationship between the first representation to the mixed media document.

6. The system of claim 5, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

7. The system of claim 1, wherein the capture device includes a portable input device for capable of capturing digital data or an image representing a portion of the first type of medium.

8. The system of claim 1, wherein the capture device includes a portable output device capable of outputting the second type of medium.

9. The system of claim 1, wherein the capture device is one from the group of a cellular camera phone;

a Personal Digital Assistant device;

a digital camera;

a barcode reader;

a radio frequency identification (RFID) reader;

a computer peripheral;

a web camera;

or a video card.

10. The system of claim 1, further comprising a memory coupled to the mixed media processor, the memory for storing instructions and/or data executable by the mixed media processor, the memory storing a content-based retrieval component that retrieves content-based information from a text or image patch.

11. The system of claim 1, further comprising a base media storage for storing the second type of medium in their original form, the base media storage coupled to the mixed media processor.

12. The system of claim 1, further comprising a mixed media storage for a plurality of mixed media documents each having a second representation and an index, the mixed media storage coupled to the mixed media processor.

13. The system of claim 1, further comprising an output device for outputting the second type of medium in response to a signal from the mixed media processor, the output device coupled to the mixed media processor.

14. The system of claim 1, wherein the first representation is a text finger print of a printed document having at least two lines that is usable for pattern matching.

15. The system of claim 1, wherein the first representation uniquely identifies a location in a printed document and is a matchable portion of a printed document.

16. The system of claim 15, wherein the first representation is a text finger print of a printed document having a portion of a first line and a portion of an adjacent second line.

17. The system of claim 1, wherein the mixed media document further comprises a file or data structure of the second type of medium, and wherein the index is link between the file and the second representation.

19. The system of claim 1, wherein the mixed media document further comprises text to be presented or one or more commands to be executed, and wherein the index is a link between the text to be presented or one or more commands to be executed file and the second representation.

20. The system of claim 1, wherein the mixed media document further comprises an electronic representation of a document in the first type of medium corresponding to the first representation, and wherein the index is link between the electronic representation and the second representation.

21. The system of claim 1, wherein the index associates the second representation, the mixed media document and the second type of medium.

22. The system of claim 1, wherein the mixed media document has a plurality of second representations each having a corresponding index.

18. The system of claim 18, wherein the mixed media document further comprises a plurality of files of the second type of medium.

23. A capture device for a mixed media system, the capture device comprising:

an input device for capturing a representation of a first media type;

a communication transceiver coupled to the input device and the output device for sending and receiving information and commands to and from a mixed media system; and

a mixed media processor for comparing the representation to a plurality of mixed media documents each having a corresponding representation, and for performing an action specified by a matching mixed media document if the representation matches the corresponding representation of the matching mixed media document.

24. The system of claim 23, wherein the first media type is a printed document.

25. The system of claim 23, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

26. The system of claim 23, further comprising an output device for presenting a second media type, the output device coupled to the mixed media processor to receive instructions and data to be output by the output device.

27. The system of claim 26, wherein the action is outputting the second media type, and the second media type is one from the group of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information and services.

28. The system of claim 23, wherein the mixed media processor includes a content-based retrieval component that retrieves content-based information from a text or image patch, and wherein the representation of the first media type first is a patch of text.

29. The system of claim 23, wherein the capture device includes a storage device for storing a plurality of mixed media documents.

30. The system of claim 23, wherein the mixed media processor compares representation of the first media type to the corresponding representation by comparing a text finger print of a printed document having a portion of a first line and a portion of an adjacent second line.

31. A device for relating a first media type and a second media type, the device comprising:

a mixed media processor having an input and an output for comparing a first representation of a first media type to a plurality of mixed media documents each having a second representation to determine a matching mixed media document, and for performing an action specified by the matching mixed media document, the input of the mixed media processor coupled to receive the first representation; and

a storage device for storing the plurality of mixed media documents, the storage device coupled to the mixed media processor.

32. The device of claim 31, further comprising a communication mechanism for coupling a capture device to the mixed media processor.

33. The device of claim 31, wherein the first media type is a paper document, and at least one mixed media document includes a second media type that is one from the group of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information and services.

34. The device of claim 31, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

35. The device of claim 31, further comprising a memory coupled to the mixed media processor, the memory for storing instructions and/or data executable by the mixed media processor, the memory storing a content-based retrieval component that retrieves content-based information from a text or image patch.

36. The device of claim 31, wherein the storage device store second media types in their original form.

37. The device of claim 31, wherein the mixed media documents further comprise an index and a file or data structure, and wherein the index is link between the file and the second representation.

38. The device of claim 31, wherein the mixed media document further comprises an electronic representation of a document of the first media type corresponding to the first representation, and an index that links the electronic representation and the second representation.

39. The device of claim 31, wherein the first representation is a text finger print of a printed document having at least two lines that is usable for pattern matching.

40. A method for providing a mixed media environment for interacting with documents of a first media type and a second media type, the method comprising:

generating a representation of a document of a first media type;

indexing the generated representation of the document to a mixed media document;

acquiring an input representation;

comparing the input representation to the generated representation; and

performing an action specified by the mixed media document if the input representation is similar to the generated representation.

41. The method of claim 40, wherein the generating the representation of the document of the first media type comprises automatically capturing a text layout of a printed document.

42. The method of claim 41, wherein the automatically capturing is done by one from a group of a printed document capture module, an operating system, a print driver, a scanner, a file transfer mechanism.

43. The method of claim 40, wherein the generated representation is one from the group of a representation of the entire document, a representation of patch of text, a representation of single word, a representation of a section of an image, and any other representation of a matchable portion of the document.

44. The method of claim 40, wherein indexing the generated representation includes association of one from the group of an index, an electronic representation of the document, a second media type, and an action, with the generated representation.

45. The method of claim 40, wherein indexing the generated representation includes one from the group of specifying x-y locations of characters of a printed document, specifying x-y locations of words of the printed document, specifying x-y locations of an image or a portion of an image in a printed document, performing an OCR operation and specifying x-y locations of characters and/or words, performing feature extraction and specifying x-y locations of characters and/or words.

46. The method of claim 40, wherein acquiring an input representation comprises one from the group of scanning a patch of text, scanning a patch of text and preprocessing to determine the likelihood that an intended feature description is extracted correctly, scanning a machine-readable identifier, inputting data that identifies the document, receiving email with the document attached, receiving the document by file transfer, scanning a portion of an image, and inputting the document.

47. The method of claim 40, wherein comparing the input representation to the generated representation further comprises performing pattern matching between the input representation and the generated representation.

48. The method of claim 47, wherein the pattern matching is performed by comparing the document fingerprint of the input representation to the document fingerprint of the generated representation.

49. The method of claim 48, wherein comparing the document fingerprint includes extracting features from a captured image, composing descriptors from those features, and looking up a document and patch that contain a percentage of those descriptors.

50. The method of claim 40, wherein performing an action specified by the mixed media document comprises retrieving a predetermined action that is associated with the mixed media document having the matching generated representation.

51. The method of claim 40, wherein the action is one from the group of retrieving information, writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, and input of a command to another system.