Automatically capturing information such as capturing information using a document-aware device

• US 10,013,766 B2
• Filed: 07/13/2015
• Issued: 07/03/2018
• Est. Priority Date: 02/18/2009
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: King, Martin T., Stephens, Redwood, Mannby, Claes-Fredrik, Peterson, Jesse, Sanvitale, Mark, Smith, Michael J.
• Primary Examiner(s): Rao, Andy
• Application Number:
  US14/797,479
• Publication Number:
  US 20160035082A1
• Time in Patent Office: 1,086 Days
• Field of Search: 348 61, 348 63, 348161, 348158
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00331 with an apparatus performing optical character recognition character recognition G06V30/10
• CPC Class Codes: H04N 1/00355 Mark-sheet input sensing record carriers G06K7/00; character recognition G06V30/10; document-oriented image-based pattern recognition G06V30/40
• CPC Class Codes: H04N 1/0036 Alphanumeric symbols
• CPC Class Codes: H04N 1/00374 on the same page as at least a part of the image
• CPC Class Codes: H04N 1/00408 Display of information to the user, e.g. menus
• CPC Class Codes: H04N 1/00469 with enlargement of a selected area of the displayed information
• CPC Class Codes: H04N 1/00689 Presence
• CPC Class Codes: H04N 1/00737 : using the scanning elements as detectors
• CPC Class Codes: H04N 1/00758 : using a prescan
• CPC Class Codes: H04N 1/00782 : Initiating operations
• CPC Class Codes: H04N 1/19594 : using a television camera or a still video camera
• CPC Class Codes: H04N 1/32144 : embedded in the image data, i.e. enclosed or integrated in the image, e.g. watermark, super-imposed logo or stamp
• CPC Class Codes: H04N 2201/0084 : Digital still camera
• CPC Class Codes: H04N 2201/0096 : Portable devices
• CPC Class Codes: H04N 2201/3249 : data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3266 : of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices

Abstract

A system and method for automatically changing the operation of a mobile device in response to a presence of information is described. In some examples, the system determines an information capture device is proximate to text, automatically changes operation of the capture device to a certain mode, captures the text, and performs an action associated with the captured text.

Claims

1. A system for presenting information to a user via a mobile device, the system comprising:

a detection component, wherein the detection component is programmed to detect that a target object is within view of a camera associated with the mobile device;

a device operation component, wherein the device operation component is programmed to modify operation of the mobile device in response to receiving a signal from the detection component that indicates the target object is within view of the camera associated with the mobile device, the device operation component being further programmed to perform operations comprising;

determining a duration over which the camera is oriented in a direction of the target object viewed in the camera; and

determining, based on the duration, an intent of a user to obtain additional information about the target object;

a capture component, wherein the capture component is programmed to capture an image of the target object using the camera, in response to receiving a signal from the device operation component that indicates modified operation of the mobile device;

an identification component, wherein the identification component is programmed to identify the target object within the captured image; and

an action component, wherein the action component is programmed to perform an action associated with the identified target object based on the intent of the user.

2. The system of claim 1, wherein the device operation component is further programmed to perform operations comprising determining that the mobile device is within a certain proximity to the target object, wherein determining an intent of the user to obtain additional information about the target object comprises determining the intent of the user based on the proximity and the duration.

3. The system of claim 1, wherein the device operation component is further programmed to perform operations comprising obtaining, by the mobile device, an electronic document corresponding to the target object, wherein performing an action comprises presenting the electronic document at the mobile device.

4. The system of claim 3, wherein obtaining an electronic document corresponding to the target object comprises:

identifying text from the image of the target object; and

identifying the electronic document based on the identified text.

5. The system of claim 1, wherein determining an intent of the user to obtain additional information about the target object comprises determining an intent of the user to obtain additional information at a time of the capturing.

6. The system of claim 1, wherein determining an intent of the user to obtain additional information about the target object comprises determining an intent of the user to obtain additional information about the target object at a future time.

7. A computer-implemented method, comprising:

detecting that a target object is within view of a camera associated with a mobile device;

determining a duration over which the camera is oriented in a direction of the target object viewed in the camera;

determining, based on the duration, an intent of a user to obtain additional information about the target object;

capturing an image of the target object using the camera;

identifying the target object within the captured image; and

performing an action associated with the identified target object based on the intent of the user.

8. The computer-implemented method of claim 7, further comprising determining a proximity of the camera to the target object, wherein determining an intent of the user to obtain additional information about the target object comprises determining the intent of the user based on the proximity and the duration.

9. The computer-implemented method of claim 7, further comprising obtaining, by the mobile device, an electronic document corresponding to the target object, wherein performing an action comprises presenting the electronic document at the mobile device.

10. The computer-implemented method of claim 9, wherein obtaining an electronic document corresponding to the target object comprises:

identifying, by the mobile device, text from the image of the target object; and

identifying, by the mobile device, the electronic document based on the identified text.

11. The computer-implemented method of claim 7, further comprising determining a proximity of the camera to the target object, wherein determining the proximity includes capturing the image of the target object using the camera of the mobile device.

12. The computer-implemented method of claim 7, wherein determining an intent of the user to obtain additional information about the target object comprises determining an intent of the user to obtain additional information at a time of the capturing.

13. The computer-implemented method of claim 7, wherein determining an intent of the user to obtain additional information about the target object comprises determining an intent of the user to obtain additional information about the target object at a future time.

14. The computer-implemented method of claim 13, further comprising creating a timeline that includes the image of the target object.

15. A non-transitory computer-readable medium encoded with a computer program that when executed by a mobile device, causes the mobile device to perform operations comprising:

detecting that a target object is within view of a camera associated with a mobile device;

determining a duration over which the camera is oriented in a direction of the target object viewed in the camera;

determining, based on the duration, an intent of a user to obtain additional information about the target object;

capturing an image of the target object using the camera;

identifying the target object within the captured image; and

performing an action associated with the identified target object based on the intent of the user.

16. The computer-readable medium of claim 15, the operations further comprising determining a proximity of the camera to the target object, wherein determining an intent of the user to obtain additional information comprises determining the intent of the user based on the proximity and the duration.

17. The computer-readable medium of claim 15, the operations further comprising obtaining, by the mobile device, an electronic document corresponding to the target object, wherein performing an action comprises presenting the electronic document at the mobile device.

18. The computer-readable medium of claim 17, wherein obtaining an electronic document corresponding to the image comprises:

identifying, by the mobile device, text from the image of the target object; and

identifying, by the mobile device, the electronic document based on the identified text.