Interacting with rendered documents using a multi-function mobile device, such as a mobile phone

• US 20110035662A1
• Filed: 02/18/2010
• Published: 02/10/2011
• Est. Priority Date: 02/18/2009
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: King, Martin T., Smith, Michael J., Stephens, Redwood, Sanvitale, Mark, Peterson, Jesse, Mannby, Claes-Fredrik
• Application Number:
  US12/660,151
• Publication Number:
  US 20110035662A1
• Time in Patent Office: Days
• US Class Current: 715/273
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00331 with an apparatus performing optical character recognition character recognition G06V30/10
• CPC Class Codes: H04N 1/00355 Mark-sheet input sensing record carriers G06K7/00; character recognition G06V30/10; document-oriented image-based pattern recognition G06V30/40
• CPC Class Codes: H04N 1/0036 Alphanumeric symbols
• CPC Class Codes: H04N 1/00374 on the same page as at least a part of the image
• CPC Class Codes: H04N 1/00408 Display of information to the user, e.g. menus
• CPC Class Codes: H04N 1/00469 with enlargement of a selected area of the displayed information
• CPC Class Codes: H04N 1/00689 Presence
• CPC Class Codes: H04N 1/00737 : using the scanning elements as detectors
• CPC Class Codes: H04N 1/00758 : using a prescan
• CPC Class Codes: H04N 1/00782 : Initiating operations
• CPC Class Codes: H04N 1/19594 : using a television camera or a still video camera
• CPC Class Codes: H04N 1/32144 : embedded in the image data, i.e. enclosed or integrated in the image, e.g. watermark, super-imposed logo or stamp
• CPC Class Codes: H04N 2201/0084 : Digital still camera
• CPC Class Codes: H04N 2201/0096 : Portable devices
• CPC Class Codes: H04N 2201/3249 : data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3266 : of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices

Abstract

A system and method for facilitating interactions between multi-function devices and rendered documents is described. In some examples, the systems overlays an image of a rendered document with display elements associated with actions to perform for the rendered document. In some examples, the system presents a virtual real-time image in place of a real-time of a document or other display of information.

Claims

1. A method using a mobile device for performing an action associated with a rendered document, the method comprising:

capturing a portion of the rendered document using an imaging component of the mobile device;

identifying the rendered document based on the captured portion;

presenting an image representing the captured portion of the rendered document via a display of the mobile device;

identifying one or more performable actions associated with the captured portion of the rendered document;

presenting one or more user-selectable display elements along with the presented portion of the rendered document;

wherein the presented one or more display elements are associated with identified one or more performable actions associated with the captured portion of the rendered document; and

upon receiving a selection of one of the user-selectable display elements, performing the action associated with the selected display element.

2. The method of claim 1, wherein presenting the image of the captured portion of the rendered document via a display of the mobile device includes presenting a real-time image of the rendered document captured by the imaging component of the mobile device.

3. The method of claim 1, wherein presenting the image of the captured portion of the rendered document via a display of the mobile device includes presenting a composite image of the rendered document captured by the imaging component of the mobile device;

wherein the composite image is formed of two or more previously captured images having a better image quality than an image quality of a real-time image.

4. The method of claim 1, wherein presenting the image of the captured portion of the rendered document via a display of the mobile device includes presenting an electronic counterpart of the rendered document at a location in the electronic counterpart that corresponds to the captured portion of the rendered document.

5. The method of claim 1, wherein presenting one or more user-selectable elements includes presenting one or more user-selectable elements associated with the identified rendered document.

6. The method of claim 1, wherein presenting one or more user-selectable elements includes presenting one or more user-selectable elements associated with a text fragment within the captured portion of the rendered document.

7. The method of claim 1, wherein presenting one or more user-selectable elements includes presenting two or more user-selectable elements associated with two or more distinct text fragments within the captured portion of the rendered document.

8. The method of claim 1, wherein identifying one or more performable actions associated with the captured portion of the rendered document includes identifying a markup layer that maps portions of rendered documents with associated performable actions.

9. The method of claim 1, wherein identifying one or more performable actions associated with the captured portion of the rendered document includes identifying a markup layer that maps text captured from rendered documents with associated performable actions.

10. A computer-readable medium whose contents, when executed by a mobile device, cause the mobile device to perform a method of performing an action in response to a capture of an image from a rendered document, the method comprising:

capturing an image of a rendered document using a camera of the mobile device, wherein the captured image includes one or more words;

identifying one or more actions associated with the one or more words in the captured image of the rendered document; and

presenting via a display of the mobile device a representation of the captured image along with a visual indication for each of the identified one or more actions.

11. The computer-readable medium of claim 10, further comprising:

in response to receiving user input selecting the presented indication of the identified one or more actions, performing the action associated with the selected indication.

12. The computer-readable medium of claim 10, wherein presenting the captured real-time image along with the indication associated with the identified one or more actions to be performed includes overlaying a display element onto the one or more words in the captured image associated with the identified one or more actions.

13. The computer-readable medium of claim 10, wherein presenting the captured real-time image along with the indication associated with the identified one or more actions to be performed includes presenting the one or more words in the captured image associated with the identified one or more actions in a color different than other words in the captured image.

14. The computer-readable medium of claim 10, further comprising:

identifying the rendered document using the one or more words in the captured image;

determining the identified rendered document is associated with an action to be performed; and

presenting a display element along with the capture image that represents the action to be performed associated with the identified rendered document.

15. A method performed by a mobile device for presenting a rendered document located in a field of view of an imaging component of the mobile device, the method comprising:

presenting, via a display component of the mobile device, a first view representative of the rendered document when the rendered document is at a first distance from the mobile device, wherein the first view includes an image representative of the rendered document; and

presenting, via a display component of the mobile device, a second view representative of the rendered document when the rendered document is at a second distance from the mobile device that is different from the first distance, wherein the second view includes an image representative of the rendered document along with one or more display elements not included in the first view that represent the availability of supplemental electronic content associated with the rendered document.

16. A system in a multi-function device for interacting with a paper document, the system comprising:

a capture component, wherein the capture component is programmed to take an image of a portion of the paper document using an imaging component of the multi-function device;

an identification component, wherein the identification component is programmed to identify an electronic counterpart to the paper document that includes text with from the imaged portion of the paper document; and

a document presentation component, wherein the document presentation component is programmed to present, via a display component of the multi-function device, a portion of an image derived from the identified electronic counterpart that corresponds to the imaged portion of the paper document.

17. The system of claim 16, wherein the document presentation component is programmed to present a second discrete portion of the identified electronic counterpart when the capture component takes an image of a second portion of the paper document that corresponds to the second discrete portion of the identified electronic counterpart.

18. The system of claim 16, wherein the presented portion of the electronic counterpart includes one or more display elements that represent performable actions associated with the paper document.

19. The system of claim 16, wherein the presented portion of the electronic counterpart includes one or more display elements that represent performable actions associated with the text from the imaged portion of the paper document.

20. The system of claim 16, further comprising:

a movement component, wherein the movement component is programmed to track movement of the multi-function device with respect to the paper document;

wherein the document presentation component is programmed to display movement of the presented electronic counterpart according to input received from the movement component.