Determining actions involving captured information and electronic content associated with rendered documents

• US 8,903,759 B2
• Filed: 09/21/2010
• Issued: 12/02/2014
• Est. Priority Date: 12/03/2004
• Status: Expired due to Fees

• Current Assignee: Kyocera Corporation
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: King, Martin T., Grover, Dale L., Kushler, Clifford A., Stafford-Fraser, James Q.
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Hoffler, Raheem
• Application Number:
  US12/887,473
• Publication Number:
  US 20110072395A1
• Time in Patent Office: 1,533 Days
• Field of Search: None
• US Class Current: 707/600
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/174 Form filling; Merging

Abstract

Information is captured from a rendered document with a handheld document data capture device. Electronic information associated with the rendered document is applied to determine the system'"'"'s actions and/or behaviors in response to the data capture. In some embodiments, the electronic information is markup data or an action map associated with the rendered document. In some embodiments, an electronic counterpart corresponding to the rendered document is located, and information associated with the electronic counterpart is applied to determine actions and/or behaviors available to a user of the handheld document data capture device.

Claims

1. A method in a computing system, comprising:

receiving captured text, the captured text being a text transcription of an image of a human-perceptible textual portion of a printed document captured by a handheld information capture device;

identifying an electronic counterpart to the printed document from the captured text, wherein the printed document has one or more context points, each of the context points is a visible feature of the printed document, each of the context points is associated in the electronic counterpart with an action, and each of the context points has a location within the printed document;

in response to receiving the captured text, determining a plurality of actions that would be appropriate to perform in response to the capture of the captured text based on the captured text or the electronic counterpart to the printed document;

determining a location within the printed document from which the captured text was captured;

determining distances between the location within the printed document of the captured text and the locations of one or more context points within the printed document; and

selecting an action that is associated with one or more of the context points, wherein selecting is based at least in part on the determined distances.

2. The method of claim 1, further comprising

receiving information related to the determined location within the printed document of the captured text.

3. The method of claim 2, wherein receiving information related to the determined location further comprises:

information related to at least one of a title, heading, caption, footnote, endnote, or citation.

4. The method of claim 1, wherein selecting an action that is associated with one or more of the context points further comprises:

determining a likelihood that a user of the handheld information capture device will select the action.

5. The method of claim 1, wherein a determined distance of the determined distance is at least one of a word distance, a sentence distance, a paragraph distance, a page distance, or a physical distance.

6. The method of claim 1, wherein the context points within the printed document are located in at least one or more of a title, a heading, a photo, an illustration, a caption, a footnote, an endnote, a citation, a keyword, a name, a location, a phone number, and or an address.

7. The method of claim 1, further comprising:

identifying supplemental information in the captured text; and

determining the action to be performed at least in part on the identified information.

8. The method of claim 1, wherein selecting an action that is associated with one or more of the context points further comprises:

identifying at least one keyword in the captured text;

locating a keyword in information associated with the printed document, wherein the information maps regions of the printed document to regions of the electronic counterpart to the printed document and wherein the information comprises at least one of actions or rules for determining actions; and

performing the action associated with the located keyword.

9. The method of claim 1, further comprising:

receiving markup data stored in the electronic counterpart to the printed document, wherein the markup data comprises at least one of actions or rules for determining actions, and wherein the markup data is applied to perform one or more actions that involve the captured text.

10. The method of claim 9, wherein the context information is in the form of digital information associated with the printed document comprising multimedia data, reference materials, or a link to the online discussion forum.

11. The method of claim 1, further comprising:

determining at least one category to which the printed document belongs; and

selecting an action based at least in part on the determined category.

12. The method of claim 11, wherein determining at least one category further comprises:

referring to categories previously determined for the printed document.

13. The method of claim 1, wherein receiving captured text from a printed document further comprises:

receiving information identifying a region of the printed document from which the captured text was captured; and

determining the action at least in part on the received information related to the region.

14. The method of claim 13, wherein receiving information from a printed document further comprises:

receiving information identifying a type of the identified region of the printed document.

15. The method of claim 14, wherein receiving information identifying a type of the identified region of the printed document further comprises:

identifying at least one or more of an advertising region, a product or products region, a sports region, a news region, a science region, a technology region, a hobby region, an entertainment region, an opinion region, a finance region, a business region, a classified region, an arts region, a real estate region, or a subject region.

16. The method of claim 13, wherein determining the action at least in part on the received information related to the region further comprises:

combining actions determined from more than one region identified as comprising the captured information.

17. The method of claim 13, wherein determining the action at least in part on the received information related to the region further comprises:

combining actions determined from regions of more than one related document.

18. The method of claim 13, wherein determining the action at least in part on the received information related to the region further comprises:

combining actions determined from regions of multiple documents in a document hierarchy.

19. The method of claim 18, wherein the document hierarchy further comprises:

a page within an article within a magazine, a page within a section within a newspaper, or a page within a chapter within a book within an anthology.

20. The method of claim 1, further comprising:

displaying to a user an indication of the action to be performed; and

upon the user selecting the action to be performed, performing the determined action.

21. A method in a computing system for providing a choice of actions for selection by a user, comprising:

receiving captured text, the captured text being a text transcription of an image of a human-perceptible textual portion of a printed document captured by a handheld information capture device;

identifying an electronic counterpart to the printed document from the captured text, wherein the printed document has one or more context points, each of the context points is a visible feature of the printed document, each of the context points is associated in the electronic counterpart with an action, and each of the context points has a location within the printed document;

determining a location within the printed document from which the captured text was captured;

determining distances between the location within the printed document of the captured text and the locations of one or more context points within the printed document;

identifying markup data that maps regions of the printed document to regions of the electronic counterpart to the printed document, wherein the markup data comprises at least one of actions or rules for determining actions;

determining a plurality of actions to be performed, based at least in part on the determined distances, and based at least in part on markup data associated with the received text; and

providing a menu of choices to the user that includes at least a portion of the determined plurality of actions.

22. The method of claim 21, further comprising automatically performing one or more of the determined plurality of actions.

24. The method of claim 21, wherein the plurality of actions include launching an application.

25. The method of claim 21, wherein the plurality of action include scrolling to the captured text in the electronic counterpart.

26. The method of claim 21, wherein the plurality of actions include highlighting the captured text in the electronic counterpart.

27. The method of claim 21, wherein the plurality of actions include placing a bookmark in the electronic counterpart.

28. The method of claim 21, wherein the plurality of actions comprise providing access to a product sale or promotion, access to a chat room, creation of or access to a discussion thread, or access to a bulletin board.

29. The method of claim 21, wherein the plurality of actions comprise obtaining access to the electronic counterpart.

30. The method of claim 21, wherein the plurality of actions comprise storing a copy of the electronic counterpart or a link to the electronic counterpart in a personal archive.

23. A system that performs an action in response to a capture of data from a printed document, comprising:

a hand-held device, the hand-held device comprising a data capture component that stores captured text, the captured text being a text transcription of an image of human-perceptible textual data captured from the printed document and a context component that determines context information that is current at a time of the data capture; and

an action component that receives the captured textual data and the determined context information to identify an electronic counterpart to the printed document from the captured text, wherein the printed document has one or more context points, each of the context points is a visible feature of the printed document, each of the context points is associated in the electronic counterpart with an action, and each of the context points has a location within the printed document, and that performs an action based on the received captured data and the received context information by determining a location within the printed document from which the captured text was captured, and by determining distances between the location of the captured text and context points within the printed document; and

wherein the action is performed based at least in part on the determined distances.

31. The system of claim 23, wherein the action comprises launching an application.

32. The system of claim 23, wherein the action comprises scrolling to the captured text in the electronic counterpart.

33. The system of claim 23, wherein the action comprises highlighting the captured text in the electronic counterpart.

34. The system of claim 23, wherein the action comprises placing a bookmark in the electronic counterpart.

35. The system of claim 23, wherein the action comprises providing access to a product sale or promotion, access to a chat room, creation of or access to a discussion thread, or access to a bulletin board.

36. The system of claim 23, wherein the action comprises obtaining access to the electronic counterpart.

37. The system of claim 23, wherein the action comprises storing a copy of the electronic counterpart or a link to the electronic counterpart in a personal archive.