Capturing text from rendered documents using supplement information

• US 8,831,365 B2
• Filed: 03/11/2013
• Issued: 09/09/2014
• Est. Priority Date: 02/15/2004
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: King, Martin Towle, Grover, Dale L., Kushler, Clifford A., Stafford-Fraser, James Quentin
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US13/792,501
• Publication Number:
  US 20130201527A1
• Time in Patent Office: 547 Days
• Field of Search: 382181-182, 382/209, 382/218, 382/305, 382/317, 382/321, 705/26, 705/35, 705/40, 705/44, 705/82
• US Class Current: 382/229
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/174 : Form filling; Merging
• CPC Class Codes: G06K 17/0016 : Selecting or retrieving of images by means of their associated code-marks, e.g. coded microfilm or microfiche microfilm reading G03B21/11; computers therefor G06F16/00; teaching devices G09B5/02; accessing record carriers G11B27/002; static digital stores using optical elements G11C13/04; intermediate storage of facsimile picture signals H04N1/21
• CPC Class Codes: G06Q 10/00 : Administration; Management
• CPC Class Codes: G06Q 10/10 : Office automation; Time management
• CPC Class Codes: G06Q 20/102 : Bill distribution or payments
• CPC Class Codes: G06Q 20/12 : specially adapted for electronic shopping systems
• CPC Class Codes: G06Q 20/4012 : Verifying personal identification numbers [PIN]
• CPC Class Codes: G06Q 30/00 : Commerce
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0207 : Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 30/0241 : Advertisements
• CPC Class Codes: G06Q 30/0283 : Price estimation or determination
• CPC Class Codes: G06Q 30/04 : Billing or invoicing
• CPC Class Codes: G06Q 30/06 : Buying, selling or leasing transactions
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0603 : Catalogue ordering
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0641 : Shopping interfaces
• CPC Class Codes: G06Q 30/0643 : Graphical representation of items or shoppers
• CPC Class Codes: G06Q 40/04 : Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 30/142 : using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/1444 : Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields
• CPC Class Codes: G06V 30/1448 : based on markings or identifiers characterising the document or the area
• CPC Class Codes: G06V 30/248 : involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q" G06V30/242 takes precedence
• CPC Class Codes: G06V 30/262 : using context analysis, e.g. lexical, syntactic or semantic context
• CPC Class Codes: G06V 30/40 : Document-oriented image-based pattern recognition
• CPC Class Codes: G07F 17/40 : for devices for accepting orders, advertisements, or the like
• CPC Class Codes: H04N 1/00204 : with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 1/00244 : with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00326 : with a data reading, recognizing or recording apparatus, e.g. with a bar-code apparatus arrangements for the associated working of recording or reproducing apparatus with related apparatus G11B31/00
• CPC Class Codes: H04N 1/0036 : Alphanumeric symbols
• CPC Class Codes: H04N 1/00363 : Bar codes or the like
• CPC Class Codes: H04N 1/00374 : on the same page as at least a part of the image
• CPC Class Codes: H04N 1/00381 : Input by recognition or interpretation of visible user gestures manual input means, e.g. digitisers, writing tablets H04N1/00392; gesture input to computers G06F3/017; recognition algorithms G06V40/20
• CPC Class Codes: H04N 1/107 : with manual scanning
• CPC Class Codes: H04N 1/2166 : for mass storage, e.g. in document filing systems information retrieval G06F16/00
• CPC Class Codes: H04N 1/2187 : with image input from a plurality of different locations or from a non-central location, e.g. from one or more users
• CPC Class Codes: H04N 1/32128 : attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 1/34 : for coin-freed systems ; Pay systems
• CPC Class Codes: H04N 2201/0081 : Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/3215 : of a time or duration
• CPC Class Codes: H04N 2201/3249 : data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3253 : Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3263 : of a graphical motif or symbol, e.g. Christmas symbol, logo H04N2201/3211 takes precedence
• CPC Class Codes: H04N 2201/3266 : of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 2201/3277 : The additional information being stored in the same storage device as the image data
• CPC Class Codes: H04N 2201/3278 : Transmission
• CPC Class Codes: Y10S 707/99931 : Database or file accessing
• CPC Class Codes: Y10S 707/99934 : Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99936 : Pattern matching access
• CPC Class Codes: Y10S 707/99943 : Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 : Object-oriented database structure processing

Abstract

A system for processing a text capture operation is described. The system receives text captured from a rendered document in the text capture operation. The system also receives supplemental information distinct from the captured text. The system determines an action to perform in response to the text capture operation based upon both the captured text and the supplemental information.

Claims

1. A method comprising:

receiving, at a computing device, text captured from a rendered document during a text capture operation;

receiving, at the computing device, supplemental information relating to circumstances under which the text capture operation was performed, the supplemental information comprising information indicating a geographical location at which the text capture operation occurs, the information indicating a geographical location at which the text capture operation occurs comprising information indicating a location as being indoors or outdoors; and

determining, by the computing device and based on the captured text and the supplemental information, an action to be performed;

wherein a determination of the location as being indoors or outdoors is based on light entering a sensor of an optical capture device.

2. The method of claim 1, wherein determining the action to be performed comprises determining to discard at least one document from among a plurality of documents identified as having the captured text.

3. The method of claim 2, wherein determining the action to be performed further comprises determining to prioritize any documents of the plurality of documents remaining after discarding the at least one document.

4. The method of claim 1, wherein determining the action to be performed comprises determining to select a default action to perform from among a plurality of actions.

5. The method of claim 1, wherein the information indicating a geographical location at which the text capture operation occurs comprises information expressed in terms of latitude and longitude.

6. The method of claim 1, further comprising:

predicting, by the computing device, context with respect to a user of a capture device that performs the text capture operation,

wherein at least a portion of the supplemental information is based on the predicted context.

7. The method of claim 6,

wherein the predicted context is based on information indicating the user is on an airplane being flown to a particular city.

8. A method comprising:

receiving, at a computing device, text captured from a rendered document during a text capture operation;

receiving, at the computing device, supplemental information relating to circumstances under which the text capture operation was performed, the supplemental information comprising information indicating a geographical location at which the text capture operation occurs; and

determining, by the computing device and based on the captured text and the supplemental information, an action to be performed;

wherein the information indicating a geographical location at which the text capture operation occurs comprises information indicating a location deduced from an ambient noise of an audio capture device that performs the text capture operation.

9. A method comprising:

receiving, at a computing device, text captured from a rendered document during a text capture operation;

receiving, at the computing device, supplemental information relating to circumstances under which the text capture operation was performed;

predicting, by the computing device, context with respect to a user of a capture device that performs the text capture operation, wherein at least a portion of the supplemental information is based on the predicted context; and

determining, by the computing device and based on the captured text and the supplemental information, an action to be performed;

wherein the predicted context is based on information indicating the user is on an airplane being flown to a particular city; and

wherein the predicted context indicates the user is reading a document about the particular city.

10. A method comprising:

receiving, at a computing device, text captured from a rendered document during a text capture operation;

receiving, at the computing device, supplemental information relating to circumstances under which the text capture operation was performed;

predicting, by the computing device, context with respect to a user of a capture device that performs the text capture operation, wherein at least a portion of the supplemental information is based on the predicted context; and

determining, by the computing device and based on the captured text and the supplemental information, an action to be performed;

wherein the predicted context is based on schedule information indicating the user is meeting with a particular person.

11. The method of claim 10, wherein the predicted context indicates the user is reading a document by or about the particular person.

12. A system comprising:

an input receiver operable to receive text captured from a rendered document during a text capture operation, and supplemental information relating to circumstances of performing the text capture operation, the supplemental information comprising information indicating a geographical location at which the text capture operation occurs, the information indicating a geographical location at which the text capture operation occurs comprising information indicating a location as being indoors or outdoors;

a processing unit; and

a computer memory configured to store the captured text, the supplemental information, and computer-readable programs executable by the processing unit;

wherein the computer-readable program instructions are executable by the processing unit to determine, based on the captured text and the supplemental information, an action to be performed; and

wherein a determination of the location as being indoors or outdoors is based on light entering a sensor of an optical capture device.

13. The system of claim 12, wherein the input receiver comprises a network connection that connects the system to the Internet.

14. The system of claim 12, wherein the computer-readable program instructions are executable by the processing unit to perform the determined action.

15. The system of claim 12,

wherein the computer-readable program instructions are executable by the processing unit to predict context with respect to a user of a capture device that performs the text capture operation, and

wherein at least a portion of the supplemental information is based on the predicted context.

16. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

receiving, at a computing device, text captured from a rendered document during a text capture operation;

receiving, at the computing device, supplemental information relating to circumstances under which the text capture operation was performed, the supplemental information comprising information indicating a geographical location at which the text capture operation occurs, the information indicating a geographical location at which the text capture operation occurs comprising information indicating a location as being indoors or outdoors; and

determining, by the computing device and based on the captured text and the supplemental information, an action to be performed;

wherein a determination of the location as being indoors or outdoors is based on light entering a sensor of an optical capture device.

17. The non-transitory computer readable medium of claim 16, wherein the stored instructions, when executed by the computing device, further cause performance of the determined action.

18. The non-transitory computer readable medium of claim 16,

wherein the stored instructions, when executed by the computing device, further cause the computing device to predict context with respect to a user of a capture device that performs the text capture operation, and

wherein at least a portion of the supplemental information is based on the predicted context.