Triggering actions in response to optically or acoustically capturing keywords from a rendered document

• US 7,596,269 B2
• Filed: 04/01/2005
• Issued: 09/29/2009
• Est. Priority Date: 02/15/2004
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Exbiblio BV
• Inventors: King, Martin T., Stafford-Fraser, James Q., Kushler, Clifford A., Grover, Dale L.
• Primary Examiner(s): Couso, Jose L
• Application Number:
  US11/097,103
• Publication Number:
  US 20060041828A1
• Time in Patent Office: 1,642 Days
• Field of Search: 382/100, 382/181, 382187-188, 382/209, 382305-306, 382312-313, 178/18.01, 178/18.03, 178/18.09, 178/19.01, 178/20.01, 235375-376, 705 26- 27, 707/10, 707/E17.095, 707/E17.107, 707/E17.112, 707/E17.113, 709/225, 358/473
• US Class Current: 382/177
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/14 Tree-structured documents parsing G06F40/205; validation G06F40/226
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06K 17/0016 : Selecting or retrieving of images by means of their associated code-marks, e.g. coded microfilm or microfiche microfilm reading G03B21/11; computers therefor G06F16/00; teaching devices G09B5/02; accessing record carriers G11B27/002; static digital stores using optical elements G11C13/04; intermediate storage of facsimile picture signals H04N1/21
• CPC Class Codes: G06Q 10/00 : Administration; Management
• CPC Class Codes: G06Q 10/10 : Office automation; Time management
• CPC Class Codes: G06Q 20/102 : Bill distribution or payments
• CPC Class Codes: G06Q 20/12 : specially adapted for electronic shopping systems
• CPC Class Codes: G06Q 20/4012 : Verifying personal identification numbers [PIN]
• CPC Class Codes: G06Q 30/00 : Commerce
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0207 : Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 30/0241 : Advertisements
• CPC Class Codes: G06Q 30/0283 : Price estimation or determination
• CPC Class Codes: G06Q 30/04 : Billing or invoicing
• CPC Class Codes: G06Q 30/06 : Buying, selling or leasing transactions
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0603 : Catalogue ordering
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0641 : Shopping interfaces
• CPC Class Codes: G06Q 30/0643 : Graphical representation of items or shoppers
• CPC Class Codes: G06Q 40/04 : Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 30/142 : using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/1444 : Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields
• CPC Class Codes: G06V 30/1448 : based on markings or identifiers characterising the document or the area
• CPC Class Codes: G06V 30/248 : involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q" G06V30/242 takes precedence
• CPC Class Codes: G06V 30/262 : using context analysis, e.g. lexical, syntactic or semantic context
• CPC Class Codes: G06V 30/40 : Document-oriented image-based pattern recognition
• CPC Class Codes: G07F 17/40 : for devices for accepting orders, advertisements, or the like
• CPC Class Codes: G07G 1/00 : Cash registers alarm indicators G07G3/00
• CPC Class Codes: H04N 1/00204 : with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 1/00244 : with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00326 : with a data reading, recognizing or recording apparatus, e.g. with a bar-code apparatus arrangements for the associated working of recording or reproducing apparatus with related apparatus G11B31/00
• CPC Class Codes: H04N 1/0036 : Alphanumeric symbols
• CPC Class Codes: H04N 1/00363 : Bar codes or the like
• CPC Class Codes: H04N 1/00374 : on the same page as at least a part of the image
• CPC Class Codes: H04N 1/00381 : Input by recognition or interpretation of visible user gestures manual input means, e.g. digitisers, writing tablets H04N1/00392; gesture input to computers G06F3/017; recognition algorithms G06V40/20
• CPC Class Codes: H04N 1/107 : with manual scanning
• CPC Class Codes: H04N 1/2166 : for mass storage, e.g. in document filing systems information retrieval G06F16/00
• CPC Class Codes: H04N 1/2187 : with image input from a plurality of different locations or from a non-central location, e.g. from one or more users
• CPC Class Codes: H04N 1/32128 : attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 1/34 : for coin-freed systems ; Pay systems
• CPC Class Codes: H04N 2201/0081 : Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/3215 : of a time or duration
• CPC Class Codes: H04N 2201/3249 : data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3253 : Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3263 : of a graphical motif or symbol, e.g. Christmas symbol, logo H04N2201/3211 takes precedence
• CPC Class Codes: H04N 2201/3266 : of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 2201/3277 : The additional information being stored in the same storage device as the image data
• CPC Class Codes: H04N 2201/3278 : Transmission
• CPC Class Codes: Y02A 90/10 : Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
• CPC Class Codes: Y10S 707/99931 : Database or file accessing
• CPC Class Codes: Y10S 707/99934 : Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99936 : Pattern matching access
• CPC Class Codes: Y10S 707/99943 : Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 : Object-oriented database structure processing

Abstract

A system for processing text captured from rendered documents is described. The system receives a sequence of one or more words optically or acoustically captured from a rendered document by a user. The system identifies among words of the sequence a word with which an action has been associated. The system then performs the associated action with respect to the user.

Claims

1. A method in a computing system, comprising:

receiving, by a processor of the computing system, a sequence of one or more words optically or acoustically captured from a rendered document by a user;

identifying, by a processor of the computing system, among the words of the sequence a word with which an action has been associated; and

performing the associated action by a processor of the computing system.

2. The method of claim 1, wherein the action is associated with the identified word by virtue of sponsorship of the action by a third party.

3. The method of claim 2, wherein performing the associated action includes presenting an advertising message to the user on behalf of the third party.

4. The method of claim 1, wherein the associated action is displaying information on a display device near the user.

5. The method of claim 1, wherein the associated action is sending an email message to the user.

6. The method of claim 1, wherein the associated action is sending an email message from the user to a recipient associated with the rendered document.

7. The method of claim 1, wherein the associated action is printing information on a printer near the user.

8. The method of claim 1, further comprising selecting an associated action for performance, wherein a plurality of actions have been associated with the identified word.

9. The method of claim 8, wherein the selecting is based at least in part upon attributes of the rendered document.

10. The method of claim 9, wherein the selecting is based at least in part upon the identity of the rendered document.

11. The method of claim 9, wherein the selecting is based at least in part upon a group of documents of which the rendered document is a member.

12. The method of claim 9, wherein the selecting is based at least in part upon the type of the rendered document.

13. The method of claim 9, wherein the selecting is based at least in part upon the source of the rendered document.

14. The method of claim 8, wherein the selecting is based at least in part upon attributes of the user.

15. The method of claim 14, wherein the selecting is based at least in part upon the identity of the user.

16. The method of claim 14, wherein the selecting is based at least in part upon the location of the user.

17. The method of claim 14, wherein the selecting is based at least in part upon a group of people of which the user is a member.

18. The method of claim 1, wherein the action was associated with the identified word after the rendered document was authored.

19. The method of claim 1, wherein the action was associated with the identified word after the rendered document was rendered.

20. The method of claim 1, wherein the action is associated with the identified word for only the rendered document.

21. The method of claim 20, wherein the action is associated with the identified word for the rendered document by an action map generated for the rendered document.

22. The method of claim 20, wherein the action is associated with the identified word for a proper subset of the rendered document by an action map generated for the rendered document.

23. The method of claim 1, wherein the action is associated with the identified word for a group of documents of which the rendered document is a member.

24. The method of claim 1, wherein the action is associated with the identified word for all documents.

34. The method of claim 1, wherein performing the associated action includes automatically performing an action without receiving feedback from the user.

35. The method of claim 1, further comprising:

presenting a menu to the user, wherein the menu includes two or more user selectable actions associated with the word;

receiving input from the user to select one of the user selectable actions; and

performing the selected action.

36. The method of claim 35, wherein the menu is presented to the user via a mobile phone associated with the user.

37. The method of claim 35, wherein the menu is presented to the user via a mobile phone that optically or acoustically captured the received sequence.

38. The method of claim 35, wherein the menu is presented to the user via a handheld capture device that optically or acoustically captured the received sequence.

39. The method of claim 35, wherein performing the associated action includes purchasing a product.

40. The method of claim 35, wherein performing the associated action includes adding a calendar item to an electronic calendar associated with the user.

41. The method of claim 1, further comprising:

determining a location of the user; and

identifying an owner of the word in the determined location;

wherein performing the associated action includes presenting advertising associated with the identified owner.

42. The method of claim 41, further comprising:

receiving a selection of the presented advertisement from the user; and

billing the owner upon receiving the selection.

25. A system for responding to the capturing of text from a rendered document, comprising:

a handheld optical and/or acoustical capture device usable by a user to capture a sequence of one or more words from a rendered document; and

a processor that identifies among the words of the sequence captured with the handheld optical and/or acoustical capture device a word with which an action has been associated, and that performs the associated action.

26. The system of claim 25 wherein the handheld optical and/or acoustical capture device includes a locator component that determines the location of the handheld optical and/or acoustical capture device, and wherein the system further comprises at least one memory whose contents specify associations between combinations each of a word and a location and an action.

43. The system of claim 25, wherein the handheld optical and/or acoustical capture device is associated with a mobile phone.

44. The system of claim 25, further comprising:

an indication component that alerts the user when the word with which an action has been associated has been identified.

27. A computer-readable medium containing a data structure mapping keywords to actions, the data structure comprising:

for each of a plurality of keywords that may be captured from rendered documents using a handheld optical and/or acoustical capture device, an entry containing information specifying an action to be performed with respect to a user who captures the keyword from a rendered document using a handheld optical and/or acoustical capture device.

28. The computer-readable medium of claim 27, wherein at least one entry of the data structure applies to all documents.

29. The computer-readable medium of claim 27, wherein at least one entry of the data structure applies to a group of documents.

30. The computer-readable medium of claim 27, wherein at least one entry of the data structure applies to a single document.

31. The computer-readable medium of claim 27, wherein a first entry of the data structure for a selected keyword applies to a group of documents, and a second entry of the data structure for the selected keyword applies to a single document, and wherein the second entry is used to determine an action to be performed with respect to a user who captures the selected keyword from the single document.

32. The computer-readable medium of claim 27, wherein at least a portion of the entries were added to the data structure in response to the payment of a sponsorship fee.

33. The computer-readable medium of claim 27, wherein at least a portion of the entries contain information specifying an action of presenting an advertising message.

45. A method in a computing system, comprising:

receiving, by a processor of the computing system, a sequence of human readable characters optically captured from a rendered document by a user;

identifying, by a processor of the computing system, among the sequence a subset of the sequence with which an action has been associated; and

performing the associated action by a processor of the computing system.

46. The method of claim 45, further comprising:

presenting a menu of user selectable choices related to actions to be performed;

wherein each of the user selectable choices is based on one of multiple distinguised subsets within the sequence.

47. The method of claim 46, wherein presenting the menu includes presenting the user selectable choices in an order dependent on a span of characters of the distinguised subset.

48. A method in a computing system of selecting an action to be performed in response to an optical capture of a sequence of characters from a rendered document, the method comprising:

identifying, by a processor of the computing system, two or more actions to be performed that are associated with a markup element that includes at least one distinguised character within the sequence of characters;

determining, by a processor of the computing system for each identified action, a number of characters shared by the markup element and the sequence of characters; and

selecting, by a processor of the computing system, an action to be performed based on a comparison of the determined number of characters for each identified action.

49. A method in a computing system, comprising:

receiving, by a processor of the computing system, a sequence of one or more words optically or acoustically captured from a rendered document by a user;

identifying, by a processor of the computing system, an action to be performed associated with a location in the rendered document proximate to the received sequence; and

performing the identified action by a processor of the computing system.

50. The method of claim 49, wherein the location partially includes the received sequence.