Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display device

• US 9,460,346 B2
• Filed: 11/14/2013
• Issued: 10/04/2016
• Est. Priority Date: 04/19/2004
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: King, Martin T., Grover, Dale L., Kushler, Clifford A., Stafford-Fraser, James Q.
• Primary Examiner(s): Hannett, James
• Application Number:
  US14/080,305
• Publication Number:
  US 20140168716A1
• Time in Patent Office: 1,055 Days
• Field of Search: 348/207.1
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 30/00 Commerce
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/36 Matching; Classification
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images
• CPC Class Codes: H04N 1/00241 using an image reading device as a local input to a computer input via network, e.g. network scanners H04N1/00225
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00488 providing an audible output to the user
• CPC Class Codes: H04N 1/0049 providing a visual indication to the user, e.g. using a lamp H04N1/00408 takes precedence
• CPC Class Codes: H04N 1/107 with manual scanning
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 2201/0041 Point to point H04N2201/0039 takes precedence
• CPC Class Codes: H04N 2201/0055 By radio
• CPC Class Codes: H04N 2201/0058 Docking-station, cradle or the like
• CPC Class Codes: H04N 2201/0081 Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0096 : Portable devices
• CPC Class Codes: H04N 2201/3249 : data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3264 : of sound signals
• CPC Class Codes: H04N 2201/3269 : of machine readable codes or marks, e.g. bar codes or glyphs

Abstract

A device for capturing rendered text is described. The device incorporates one or more visual sensors that receive visual information as a part of capturing rendered text. The visual sensors are collectively capable of capturing both text that is permanently printed on a page, and text that is displayed transitorily on a dynamic device. The device further incorporates a visual information disposition subsystem for disposing of visual information received by the visual sensors. The device further incorporates a package that bears the visual sensors and the visual information disposition subsystem, and is suitable to be held in a human hand.

Claims

1. A method comprising:

capturing, at a handheld user device that includes an integrated visual sensor and an integrated visual display, text of a document displayed transitorily on a dynamic display using the visual sensor of the handheld user device;

using textual content of the captured text to identify the document and a position in the document at which a capture is performed;

determining a set of actions appropriate in connection with the capture from the identified document at the identified position, wherein determining the set of actions comprises accessing an electronic counterpart of the identified document and analyzing markup data included in the electronic counterpart of the identified document that associates the identified position with the set of actions;

presenting, on the visual display of the handheld user device, at least a portion of the determined set of actions to a user, wherein each action of the portion of the determined set of actions is presented as a respective menu item of a menu presented on the visual display of the handheld capture device;

receiving, at the handheld capture device, user input selecting one of the presented menu items; and

in response to the received user input, performing the selected action in connection with the capture.

2. The method of claim 1, wherein the determined set of actions comprises an action to be performed in accordance with a keyword or key phrase that occurs in the identified document in proximity to the identified position.

3. The method of claim 2, wherein the action to be performed in accordance with the keyword or key phrase comprises providing information associated with a seller of a product related to the keyword or key phrase.

4. The method of claim 1, wherein the determined set of actions comprises an action for providing help information for the identified document.

14. The method of claim 1, wherein the determined set of actions comprises an action for printing the electronic counterpart of the identified document and storing a record that includes a timestamp of when the printing occurred and an identifier of a user that selected the action.

15. The method of claim 1, wherein the determined set of actions comprises an action for presenting the electronic counterpart of the identified document on the visual display of the handheld user device, navigating to the identified position, and highlighting text within the electronic counterpart of the identified document that corresponds to the captured text.

5. Non-transitory computer-readable media having contents capable of causing a computing system including one or more computers and a capture device to perform a method, the method comprising:

capturing, at a handheld user device that includes an integrated visual sensor and an integrated visual display, text of a document displayed transitorily on a dynamic display using the visual sensor of the handheld user device;

using textual content of the captured text to identify the document and a position in the document at which a capture is performed;

determining a set of actions appropriate in connection with the capture from the identified document at the identified position, wherein determining the set of actions comprises accessing an electronic counterpart of the identified document and analyzing markup data included in the electronic counterpart of the identified document that associates the identified position with the set of actions;

presenting, on the visual display of the handheld user device, at least a portion of the determined set of actions to a user, wherein each action of the portion of the determined set of actions is presented as a respective menu item of a menu presented on the visual display of the handheld capture device;

receiving, at the handheld capture device, user input selecting one of the presented menu items; and

in response to the received user input, performing the selected action in connection with the capture.

6. The non-transitory computer-readable media of claim 5, wherein the determined set of actions comprises an action to be performed in accordance with a keyword or key phrase that occurs in the identified document in proximity to the identified position.

7. The non-transitory computer-readable media of claim 6, wherein the action to be performed in accordance with the keyword or key phrase comprises providing information associated with a seller of a product related to the keyword or key phrase.

8. The non-transitory computer-readable media of claim 5, wherein the determined set of actions comprises an action for providing help information for the identified document.

16. The non-transitory computer-readable media of claim 5, wherein the determined set of actions comprises an action for printing the electronic counterpart of the identified document and storing a record that includes a timestamp of when the printing occurred and an identifier of a user that selected the action.

17. The non-transitory computer-readable media of claim 5, wherein the determined set of actions comprises an action for presenting the electronic counterpart of the identified document on the visual display of the handheld user device, navigating to the identified position, and highlighting text within the electronic counterpart of the identified document that corresponds to the captured text.

9. A system, comprising:

a handheld capture device that includes an integrated visual sensor and an integrated visual display, the handheld capture device configured to capture text of a document displayed transitorily on a dynamic display using the visual sensor; and

one or more computers having one or more memory devices on which are stored computer executable instructions operable to cause the one or more computers to perform operations comprising;

receiving, from the handheld capture device, text of the document displayed transitorily on the dynamic display and captured using the visual sensor of the handheld capture device;

using textual content of the captured text to identify the document and a position in the document at which a capture is performed;

determining a set of actions appropriate in connection with the capture from the identified document at the identified position, wherein determining the set of actions comprises accessing an electronic counterpart of the identified document and analyzing markup data included in the electronic counterpart of the identified document that associates the identified position with the set of actions;

providing, to the handheld capture device for presentation on the visual display of the handheld capture device, at least a portion of the determined set of actions, wherein each action of the portion of the determined set of actions is presented as a respective menu item of a menu presented on the visual display of the handheld capture device;

receiving, at the handheld capture device, user input selecting one of the presented menu items; and

in response to the received user input, performing the selected action in connection with the capture.

10. The system of claim 9, wherein the determined set of actions comprises an action to be performed in accordance with a keyword or key phrase that occurs in the identified document in proximity to the identified position.

11. The system of claim 10, wherein the action to be performed in accordance with the keyword or key phrase comprises providing information associated with a seller of a product related to the keyword or key phrase.

12. The system of claim 9, wherein the determined set of actions comprises an action for providing help information for the identified document.

18. The system of claim 9, wherein the determined set of actions comprises an action for printing the electronic counterpart of the identified document and storing a record that includes a timestamp of when the printing occurred and an identifier of a user that selected the action.

19. The system of claim 9, wherein the determined set of actions comprises an action for presenting the electronic counterpart of the identified document on the visual display of the handheld user device, navigating to the identified position, and highlighting text within the electronic counterpart of the identified document that corresponds to the captured text.

13. A method comprising:

receiving, at one or more computers and from a handheld user device that includes an integrated visual sensor and an integrated visual display, text of a document displayed transitorily on a dynamic display and captured using the visual sensor of the handheld user device;

using textual content of the captured text to identify the document and a position in the document at which a capture is performed;

determining a set of actions appropriate in connection with the capture from the identified document at the identified position, wherein determining the set of actions comprises accessing an electronic counterpart of the identified document and analyzing markup data included in the electronic counterpart of the identified document that associates the identified position with the set of actions;

providing, to the handheld user device for presentation on the visual display of the handheld user device, at least a portion of the determined set of actions, wherein each action of the portion of the determined set of actions is presented as a respective menu item of a menu presented on the visual display of the handheld capture device;

receiving, at the handheld capture device, user input selecting one of the presented menu items; and

in response to the received user input, performing the selected action in connection with the capture.