Automatically capturing information, such as capturing information using a document-aware device

• US 20110025842A1
• Filed: 02/18/2010
• Published: 02/03/2011
• Est. Priority Date: 02/18/2009
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: King, Martin T., Smith, Michael J., Stephens, Redwood, Sanvitale, Mark, Peterson, Jesse, Mannby, Claes-Fredrik
• Granted Patent: US 8,638,363 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00331 with an apparatus performing optical character recognition character recognition G06V30/10
• CPC Class Codes: H04N 1/00355 Mark-sheet input sensing record carriers G06K7/00; character recognition G06V30/10; document-oriented image-based pattern recognition G06V30/40
• CPC Class Codes: H04N 1/0036 Alphanumeric symbols
• CPC Class Codes: H04N 1/00374 on the same page as at least a part of the image
• CPC Class Codes: H04N 1/00408 Display of information to the user, e.g. menus
• CPC Class Codes: H04N 1/00469 with enlargement of a selected area of the displayed information
• CPC Class Codes: H04N 1/00689 Presence
• CPC Class Codes: H04N 1/00737 : using the scanning elements as detectors
• CPC Class Codes: H04N 1/00758 : using a prescan
• CPC Class Codes: H04N 1/00782 : Initiating operations
• CPC Class Codes: H04N 1/19594 : using a television camera or a still video camera
• CPC Class Codes: H04N 1/32144 : embedded in the image data, i.e. enclosed or integrated in the image, e.g. watermark, super-imposed logo or stamp
• CPC Class Codes: H04N 2201/0084 : Digital still camera
• CPC Class Codes: H04N 2201/0096 : Portable devices
• CPC Class Codes: H04N 2201/3249 : data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3266 : of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices

Abstract

A system and method for automatically changing the operation of a mobile device in response to a presence of information is described. In some examples, the system determines an information capture device is proximate to text, automatically changes operation of the capture device to a certain mode, captures the text, and performs an action associated with the captured text.

Claims

1. A system for presenting information to a user via a mobile device, the system comprising:

a text detection component, wherein the text detection component is programmed to detect that text is within view of a camera associated with the mobile device;

a device operation component, wherein the device operation component is programmed to modify operation of the mobile device in response to receiving a signal from the text detection component that indicates text is within view of the camera associated with the mobile device;

a capture component, wherein the capture component is programmed to capture an image of the text in response to receiving a signal from the device operation component that indicates modified operation of the mobile device;

an identification component, wherein the identification component is programmed to identify the text or an object that contains the text within the captured image; and

an action component, wherein the action component is programmed to perform an action associated with the identified text or the object that contains the text.

2. The system of claim 1, wherein the device operation component is programmed to modify the operation of the mobile device from a default mode of operation to a different mode of operation that enables the mobile device to capture information proximate to the mobile device.

3. The system of claim 1, wherein the text detection component is programmed to periodically sample images received by the camera.

4. The system of claim 1, wherein the text detection component is programmed to determine that the mobile device is within a certain proximity to the object and initiate reception of images by the camera based on the determined proximity.

5. The system of claim 1, wherein the identification component is programmed to identify a rendered document that includes the captured text;

and

wherein the action component is programmed to present an electronic counterpart of the object that contains the text to a user of the mobile device.

6. The system of claim 1, further comprising:

a display component, wherein the display component presents the view of the camera to a user of the mobile device;

wherein the action component is programmed to overlay text within the presented view of the camera with graphical elements that indicate the text is associated with additional information.

7. The system of claim 1, further comprising:

a display component, wherein the display component presents the view of the camera to a user of the mobile device;

wherein the action component is programmed to present one or more user-selectable options associated with the text or the display of text with the presented view of the camera.

8. A method for operating a mobile device, the method comprising:

determining that the mobile device is proximate to a rendered document; and

automatically executing a program based on the determination, wherein the executed program enables the mobile device to interact with the rendered document.

9. The method of claim 8, wherein determining that the mobile device is proximate to the rendered document includes:

capturing an image of the rendered document using an imaging component of the mobile device; and

identifying text within the image of captured image.

10. The method of claim 8, where determining that the mobile device is proximate to the rendered document includes determining that the mobile device is within a certain proximity to the rendered document.

11. The method of claim 8, where determining that the mobile device is proximate to the rendered document includes determining that the mobile device is being held in a certain position relative to the rendered document.

12. The method of claim 8, wherein the executed program that enables the mobile device to interact with the rendered document includes an application that identifies the rendered document from text captured from the rendered document.

13. The method of claim 8, wherein the executed program that enables the mobile device to interact with the rendered document includes an application that identifies actions to perform associated with text captured from the rendered document.

14. The method of claim 8, wherein determining that the mobile device is proximate to the rendered document includes capturing an image of the rendered document using an imaging component of the mobile device and identifying text within the image of captured image, the method further comprising:

receiving an indication that electronic content is associated with the identified text; and

automatically launching an application based on the determination and the received indication.

15. The method of claim 8, wherein determining that the mobile device is proximate to the rendered document includes capturing an image of the rendered document using an imaging component of the mobile device and identifying text within the image of captured image, the method further comprising:

receiving an indication that one or more actions to be performed are associated with the identified text; and

automatically launching an application based on the determination and the received indication.

16. A computer-readable medium whose contents, when executed by a mobile device, cause the mobile device to perform a method for performing an action associated with a rendered document, the method comprising:

measuring a relative orientation between a rendered document and the mobile device;

comparing the measured relative orientation to a threshold relative orientation associated with a document capture mode of the mobile device; and

automatically transitioning the mobile device to the document capture mode when the measured relative orientation is within the threshold relative orientation associated with the document capture mode.

17. The computer-readable medium of claim 16, wherein measuring the relative orientation between the rendered document and the mobile device includes measuring the distance between an imaging component of the mobile device and the rendered document.

18. The computer-readable medium of claim 16, wherein measuring the relative orientation between the rendered document and the mobile device includes measuring the distance between a center point of a display associated with the mobile device and the rendered document.

19. The computer-readable medium of claim 16, wherein the threshold relative orientation is determined by a user of the mobile device.

20. The computer-readable medium of claim 16, wherein the threshold relative orientation is determined by performing an analysis of previous occurrences of the mobile device in document capture mode, wherein the analysis includes:

analyzing occurrences when a user of the mobile device rejected the automatic launching of the application that performs actions in response to text captures; and

analyzing occurrences when a user of the mobile device manually launched the application that performs actions in response to text captures.

21. The computer-readable medium of claim 16, wherein the threshold relative orientation is determined by performing an analysis of previous occurrences of the mobile device in document capture mode, wherein the analysis includes:

measuring distances between the mobile device and a rendered document when a previous occurrence includes a user manually launching the application that performs actions in response to text captures.

22. The computer-readable medium of claim 16, wherein the threshold relative orientation is determined by performing an analysis of previous occurrences of the mobile device in document capture mode, wherein the analysis includes:

measuring distances between the mobile device and a rendered document when a previous occurrence includes a user rejecting the automatic launching of the application that performs actions in response to text captures.

23. The computer-readable medium of claim 16, wherein measuring the relative orientation between the rendered document and the mobile device includes periodically measuring the distance;

and

wherein comparing the measured relative orientation to the threshold relative orientation associated with the document capture mode of the mobile device includes comparing two or more periodically measured distances; and

wherein automatically transitioning the mobile device to the document capture mode includes transitioning the mobile device to the document capture mode when the two or more periodically measured distances are within the threshold relative orientation associated with the document capture mode.

24. The computer-readable medium of claim 16, further comprising:

periodically measuring the relative orientation between the rendered document and the mobile device after transitioning the mobile device to the document capture mode; and

automatically exiting the document capture mode when the periodically measured relative orientation is outside of the threshold relative orientation associated with the document capture mode.

25. A method for directing operation of a mobile device in response to a presence of available information, the method comprising:

periodically capturing data using an capture component of the mobile device; and

when at least some of the data has associated actions, causing the mobile device to perform one of the associated actions with respect to the captured data.

26. The method of claim 25, wherein the performed action is displaying, via a display component of the mobile device, an indication that content is associated with at least some of the captured data.

27. The method of claim 25, wherein the performed action is storing into a database associated with a user of the mobile device an indication that at least some of the captured data is text.