Capturing text from rendered documents using supplemental information

• US 7,599,580 B2
• Filed: 04/01/2005
• Issued: 10/06/2009
• Est. Priority Date: 02/15/2004
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Exbiblio BV
• Inventors: Stafford-Fraser, James Q., Kushler, Clifford A., Grover, Dale L., King, Martin T.
• Primary Examiner(s): Mariam, Daniel G
• Application Number:
  US11/096,704
• Publication Number:
  US 20060026078A1
• Time in Patent Office: 1,649 Days
• Field of Search: 382/190, 382/209, 382/229, 382/305, 382/115, 382/176, 382/181, 382/187, 382/306, 382/313, 382/314, 707/3, 707/E17.058, 715/200, 715/206, 345/179, 178/18.01, 178/18.03, 178/19.01, 178/19.05
• US Class Current: 382/306
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/14 Tree-structured documents parsing G06F40/205; validation G06F40/226
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06K 17/0016 Selecting or retrieving of images by means of their associated code-marks, e.g. coded microfilm or microfiche microfilm reading G03B21/11; computers therefor G06F16/00; teaching devices G09B5/02; accessing record carriers G11B27/002; static digital stores using optical elements G11C13/04; intermediate storage of facsimile picture signals H04N1/21
• CPC Class Codes: G06Q 10/00 : Administration; Management
• CPC Class Codes: G06Q 10/10 : Office automation; Time management
• CPC Class Codes: G06Q 20/102 : Bill distribution or payments
• CPC Class Codes: G06Q 20/12 : specially adapted for electronic shopping systems
• CPC Class Codes: G06Q 20/4012 : Verifying personal identification numbers [PIN]
• CPC Class Codes: G06Q 30/00 : Commerce
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0207 : Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 30/0241 : Advertisements
• CPC Class Codes: G06Q 30/0283 : Price estimation or determination
• CPC Class Codes: G06Q 30/04 : Billing or invoicing
• CPC Class Codes: G06Q 30/06 : Buying, selling or leasing transactions
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0603 : Catalogue ordering
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0641 : Shopping interfaces
• CPC Class Codes: G06Q 30/0643 : Graphical representation of items or shoppers
• CPC Class Codes: G06Q 40/04 : Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 30/142 : using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/1444 : Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields
• CPC Class Codes: G06V 30/1448 : based on markings or identifiers characterising the document or the area
• CPC Class Codes: G06V 30/248 : involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q" G06V30/242 takes precedence
• CPC Class Codes: G06V 30/262 : using context analysis, e.g. lexical, syntactic or semantic context
• CPC Class Codes: G06V 30/40 : Document-oriented image-based pattern recognition
• CPC Class Codes: G07F 17/40 : for devices for accepting orders, advertisements, or the like
• CPC Class Codes: G07G 1/00 : Cash registers alarm indicators G07G3/00
• CPC Class Codes: H04N 1/00204 : with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 1/00244 : with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00326 : with a data reading, recognizing or recording apparatus, e.g. with a bar-code apparatus arrangements for the associated working of recording or reproducing apparatus with related apparatus G11B31/00
• CPC Class Codes: H04N 1/0036 : Alphanumeric symbols
• CPC Class Codes: H04N 1/00363 : Bar codes or the like
• CPC Class Codes: H04N 1/00374 : on the same page as at least a part of the image
• CPC Class Codes: H04N 1/00381 : Input by recognition or interpretation of visible user gestures manual input means, e.g. digitisers, writing tablets H04N1/00392; gesture input to computers G06F3/017; recognition algorithms G06V40/20
• CPC Class Codes: H04N 1/107 : with manual scanning
• CPC Class Codes: H04N 1/2166 : for mass storage, e.g. in document filing systems information retrieval G06F16/00
• CPC Class Codes: H04N 1/2187 : with image input from a plurality of different locations or from a non-central location, e.g. from one or more users
• CPC Class Codes: H04N 1/32128 : attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 1/34 : for coin-freed systems ; Pay systems
• CPC Class Codes: H04N 2201/0081 : Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/3215 : of a time or duration
• CPC Class Codes: H04N 2201/3249 : data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3253 : Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3263 : of a graphical motif or symbol, e.g. Christmas symbol, logo H04N2201/3211 takes precedence
• CPC Class Codes: H04N 2201/3266 : of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 2201/3277 : The additional information being stored in the same storage device as the image data
• CPC Class Codes: H04N 2201/3278 : Transmission
• CPC Class Codes: Y02A 90/10 : Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
• CPC Class Codes: Y10S 707/99931 : Database or file accessing
• CPC Class Codes: Y10S 707/99934 : Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99936 : Pattern matching access
• CPC Class Codes: Y10S 707/99943 : Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 : Object-oriented database structure processing

Abstract

A system for processing a text capture operation is described. The system receives text captured from a rendered document in the text capture operation. The system also receives supplemental information distinct from the captured text. The system determines an action to perform in response to the text capture operation based upon both the captured text and the supplemental information.

Claims

1. A method in a computing system for processing a distinguished text capture operation, comprising:

receiving human-readable text captured by a user via a portable capture device from a distinguished rendered document in the distinguished text capture operation;

receiving supplemental information distinct from the captured text, said supplemental information comprising an identity associated with said user; and

automatically determining, by the computing system in response to the distinguished text capture operation and based upon both the captured text and the supplemental information, which one of a predetermined plurality of actions is likely optimal for said user.

2. The method of claim 1 wherein the received captured text is captured from at least one of a dynamic display, a voice input and a printed document.

3. The method of claim 1, further comprising:

applying optical character recognition techniques to an image captured from the distinguished rendered document to generate the captured text.

4. The method of claim 3, wherein said image captured from the distinguished rendered document depicts fewer than ten words.

5. The method of claim 1, further comprising:

applying voice recognition techniques to an audio clip of a person reading aloud from the distinguished rendered document to generate the captured text.

6. The method of claim 1 wherein the received supplemental information includes text captured in previous text capture operations by said user, and wherein the determined action is identifying a single electronic document as corresponding to the distinguished rendered document.

7. The method of claim 1 wherein the received supplemental information includes a document rendering characteristic observed in a previous text capture operation by said user and a corresponding document rendering characteristic observed in the distinguished text capture operation, and wherein the determined action is identifying different electronic documents as corresponding to the rendered documents in which the distinguished text capture operation and the previous text capture operation were performed based upon a difference between the document rendering characteristic observed in the previous text capture operation and the corresponding document rendering characteristic observed in the distinguished text capture operation.

8. The method of claim 7 wherein the document rendering characteristic observed in the previous text capture operation and the corresponding document rendering characteristic observed in the distinguished text capture operation are different fonts.

9. The method of claim 7 wherein the document rendering characteristic observed in the previous text capture operation and the corresponding document rendering characteristic observed in the distinguished text capture operation are different font sizes.

10. The method of claim 7 wherein the document rendering characteristic observed in the previous text capture operation and the corresponding document rendering characteristic observed in the distinguished text capture operation are different paper types.

11. The method of claim 1 wherein the determined action is identifying a position in a distinguished electronic document that is subsequent to a position identified in the distinguished electronic document as the site of a previous text capture operation by the distinguished user as the site of the distinguished text capture operation.

12. The method of claim 1 wherein the received supplemental information indicates a geographic location at which the distinguished capture operation was performed, and wherein the determining comprises:

identifying a plurality of electronic documents containing the received captured text; and

ranking the likelihood of the identified electronic documents corresponding to the distinguished rendered document based upon the likelihood that a rendered version of each identified electronic document would be available in the indicated geographic location.

13. The method of claim 1, further comprising using the received captured text to identify an electronic document corresponding to the distinguished rendered document, wherein the received supplemental information indicates a geographic location at which rendered copies of the identified electronic document are likely to be available, and wherein the determining comprises determining that the text capture operation was performed at the indicated geographic location.

14. The method of claim 1 wherein the distinguished text capture operation is performed using a text capture device, and wherein the received supplemental information includes information about an ambient environment in which the distinguished text capture operation was performed, and wherein the determining comprises determine a configuration of a distinguished capability of the device.

15. The method of claim 14 wherein the distinguished text capture operation involves capturing an image of the captured text from the distinguished rendered document, and wherein the received supplemental information includes an indication of a frequency of light received in the image capture.

16. The method of claim 14 wherein the distinguished text capture operation involves capturing an audio clip of a person reading aloud from the distinguished rendered document, and wherein the received supplemental information includes an indication of background sounds contained in the audio clip.

17. The method of claim 14 wherein determining a configuration of a distinguished capability of the device comprises determining to disable the distinguished capability.

18. The method of claim 1 wherein the received supplemental information includes an indication of the time of day at which the distinguished text capture operation was performed, and the determined action is identifying an electronic document as corresponding to the distinguished rendered document.

19. The method of claim 1 wherein the received supplemental information includes an indication of the day of week at which the distinguished text capture operation was performed, and wherein the determined action is identifying an electronic document as corresponding to the distinguished rendered document.

20. The method of claim 1 wherein the received supplemental information includes an indication of recent interactions of said user with electronic documents, and wherein the determined action is identifying an electronic document as corresponding to the distinguished rendered document.

21. The method of claim 20 wherein the received supplemental information identifies documents that said user has recently accessed.

22. The method of claim 20 wherein the received supplemental information identifies documents that said user has recently saved.

23. The method of claim 20 wherein the received supplemental information identifies documents that said user has recently emailed.

24. The method of claim 20 wherein the received supplemental information identifies documents that said user has recently received via email.

25. The method of claim 20 wherein the received supplemental information identifies documents that said user has recently sent via instant messaging.

26. The method of claim 20 wherein the received supplemental information identifies documents that said user has recently received via instant messaging.

27. The method of claim 20 wherein the received supplemental information identifies documents that said user has recently printed.

28. The method of claim 20 wherein the received supplemental information identifies documents that said user has recently searched.

29. The method of claim 1 wherein the received supplemental information includes an indication of a time at which the distinguished text capture operation was performed, and wherein the determining is performed at time distinct from the time at which the distinguished text capture operation was performed, and wherein the determining is performed based on the captured text, the indication of the time at which the distinguished text capture operation was performed, and the time at which the determining is performed.

30. The method of claim 1 wherein the received supplemental information includes information about the conditions under which the text capture operation was performed, and wherein the determined action is associating the information about the conditions under which the text capture operation was performed with said user.

31. The method of claim 1 wherein the received supplemental information includes information about previous text capture operations performed by users other than the distinguished user, and wherein the determined action is identifying a single electronic document as corresponding to the distinguished rendered document.

32. The method of claim 1 wherein the received supplemental information includes information about previous text capture operations performed by users other than said user and about the conditions under which the previous text capture operation was performed, and wherein the determined action is identifying a single electronic document as corresponding to the distinguished rendered document.

33. The method of claim 1 wherein the distinguished text capture operation is performed using a distinguished text capture device, and wherein the determined action is identifying data that is likely to be useful for processing future text capture operations and storing the identified data in at least one of the computing system and the distinguished text capture device.

34. The method of claim 1 wherein the distinguished rendered document contains at least one line of text, and wherein the received text is a proper subset of the text contained in a single line of the distinguished rendered document.

35. The method of claim 1 wherein the distinguished rendered document contains at least one page having text, and wherein the received text is a proper subset of the text contained in a single page of the distinguished rendered document.

36. The method of claim 1 wherein the received text is comprised of words, and wherein the distinguished text capture operation involved specific user interactions with each of the words of the received text.

37. The method of claim 36 wherein said specific user interactions comprise the user speaking each of the words of the received text.

38. The method of claim 36 wherein the user directed an optical sensor at each of the words of the received text.

39. The method of claim 1 wherein the received text is comprised of ordered words, and wherein the distinguished text capture operation involved capturing a physical phenomenon corresponding to each of the words of the received text in the order of the received text.

40. The method of claim 1 wherein the received text is comprised of ordered words, and wherein the distinguished text capture operation involved capturing physical phenomenon corresponding to each of the words of the received text in the reverse order of the received text.

41. The method of claim 1 wherein the distinguished text capture operation involved manually moving an optical sensor across the distinguished rendered document.

42. The method of claim 1 wherein the distinguished text capture operation involved capturing image data from a non-rectangular region of the distinguished rendered document.

43. The method of claim 1 wherein the received text comprises fewer than ten words.

44. A system for processing a distinguished text capture operation, comprising:

a capture component that receives human-readable text captured by a user via a portable capture device from a distinguished rendered document in the distinguished text capture operation;

an information component that receives supplemental information distinct from the captured text, said supplemental information comprising an identity associated with said user; and

an action component that automatically determines, in response to the distinguished text capture operation and based upon both the captured text and the supplemental information, which one of a predetermined plurality of actions is likely optimal for said user.

45. A method in a computing system for processing a distinguished text capture operation, comprising:

receiving human-readable text captured by a user via a portable capture device from a distinguished rendered document in the distinguished text capture operation;

receiving supplemental information distinct from the captured text, said supplemental information comprising an identity associated with said user; and

identifying an electronic document similar to the rendered document based upon the captured text and the supplemental information.

46. A computer-readable medium whose contents cause a computing system to perform a method for processing a distinguished text capture operation, the method comprising:

receiving human-readable text captured by a user via a portable capture device from a distinguished rendered document in the distinguished text capture operation;

receiving supplemental information distinct from the captured text, said supplemental information comprising an identity associated with said user; and

automatically determining in response to the distinguished text capture operation and based upon both the captured text and the supplemental information, which one of a predetermined plurality of actions is likely optimal for said user.