Adding information or functionality to a rendered document via association with an electronic counterpart

• US 8,793,162 B2
• Filed: 05/05/2010
• Issued: 07/29/2014
• Est. Priority Date: 04/01/2004
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: King, Martin T., Grover, Dale L., Kushler, Clifford A., Stafford-Fraser, James Q.
• Primary Examiner(s): Garg, Yogesh C
• Application Number:
  US12/774,612
• Publication Number:
  US 20110019020A1
• Time in Patent Office: 1,546 Days
• Field of Search: None
• US Class Current: 705/26.1
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/262 using context analysis, e.g. lexical, syntactic or semantic context
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

An action plan data structure for one or more selected rendered documents is described. The data structure contains information specifying an action to perform automatically in response to a text capture from any of the selected rendered documents.

Claims

1. A method, comprising:

receiving, at a computer, a query that comprises a text string, wherein the text string is associated with a rendered document; and

in response to the query;

identifying, by the computer, an electronic counterpart of a rendered document based on the text string;

determining, by the computer, a markup associated with at least the electronic counterpart of the rendered document, wherein the markup is configured to associate selectable options with the at least the electronic counterpart, the markup providing additional functionality associated with a portion of the rendered document that includes the text string;

determining, by the computer, a plurality of selectable options associated with a set of one or more actions specified based on the markup, wherein the set of one or more actions comprises an action associated with the electronic counterpart; and

providing the plurality of selectable options.

2. The method of claim 1, wherein the set of one or more actions comprises an action with a visual element, and wherein the method further comprises performing the action with the visual element using a display.

3. The method of claim 1, wherein the set of one or more actions comprises providing information associated with the text string.

4. The method of claim 1, further comprising determining a location where the text string was captured;

and

wherein the set of one or more actions includes one or more actions associated with the determined location.

5. The method of claim 1, further comprising determining an identity of a user that captured an image that includes the text string;

and

wherein the set of one or more actions includes one or more actions associated with the determined identity.

6. The method of claim 1, wherein the set of one or more actions comprises an action with an audio element, and wherein the method further comprises performing, by the computer and using speakers of the computer, the action with the audio element.

7. The method of claim 1, wherein the markup includes a plurality of layers, each layer being associated with a different user.

8. The method of claim 1, further comprising performing a default action in response to not receiving a user selection of a presented option.

9. The method of claim 1, wherein the markup comprises an overlay for the electronic counterpart that provides additional information about the rendered document that includes the text string.

10. A non-transitory computer-readable medium having executable instructions that, when executed by a processor, cause a computer system to perform operations comprising:

capturing an image of a rendered document, wherein the image includes text;

recognizing a text string within the captured image;

identifying an electronic counterpart of the rendered document based on the text string;

determining a markup associated with at least the electronic counterpart, wherein the markup is configured to associate selectable options with the at least the electronic counterpart, the markup providing additional functionality associated with a portion of the rendered document from which the image was captured;

determining a plurality of selectable options associated with a set of one or more actions specified based on the markup associated with the at least the electronic counterpart, wherein the set of one or more actions comprises an action associated with the electronic counterpart; and

providing a plurality of selectable options associated with the set of one or more actions.

11. The non-transitory computer-readable medium of claim 10, wherein the set of one or more actions comprises an action with a visual element, and wherein the operations further comprise performing the action with the visual element using the display of the mobile device.

12. The non-transitory computer-readable medium of claim 10, wherein the set of one or more actions comprises an action with an audio element, and wherein the operations further comprise causing speakers to perform the action with the audio element.

20. The non-transitory computer-readable medium of claim 10, wherein the markup includes a plurality of layers, each layer being associated with a different user.

13. A system, comprising:

a processor;

a memory storage apparatus in data communication with the processor, the memory storage apparatus storing instructions executable by the processor and that upon such execution cause the processor to perform operations comprising;

receiving an image of a rendered document that includes text;

recognizing a text string within the image;

identifying an electronic counterpart of the rendered document based on the text string;

determining markup associated with at least the electronic counterpart, wherein the markup is configured to associate selectable options with the at least the electronic counterpart, the markup providing additional functionality associated with a portion of the rendered document from which the image was captured;

determining a plurality of selectable options associated with a set of one or more actions specified based on the markup associated with the at least the electronic counterpart, wherein the set of one or more actions comprises an action associated with the electronic counterpart;

providing the plurality of selectable options associated with the set of one or more actions; and

performing actions that are associated with selected options.

14. The system of claim 13, wherein the

instructions upon execution cause the processor to perform further operations comprising identifying at least one action of the set of one or more actions based on a search engine result.

15. The system of claim 13, wherein the identified set of one or more actions includes an action that retrieves real-time information associated with the captured image and wherein the instructions upon execution cause the processor to perform further operations comprising providing the real time information.

16. The system of claim 13, wherein the set of one or more actions includes an action that retrieves location-based information associated with the captured image and wherein the instructions upon execution cause the processor to perform further operations comprising providing the location-based information.

17. The system of claim 13, wherein the set of one or more actions includes an action that stores contact information associated with the captured image and wherein the instructions upon execution cause the processor to perform further operations comprising providing the stored contact information.

18. The system of claim 13, wherein the markup includes a plurality of layers, each layer being associated with a different user.

19. The system of claim 13, wherein the markup comprises an overlay for the electronic counterpart that provides additional information about the rendered document that includes the text string.