HANDHELD DEVICE FOR CAPTURING TEXT FROM BOTH A DOCUMENT PRINTED ON PAPER AND A DOCUMENT DISPLAYED ON A DYNAMIC DISPLAY DEVICE

• US 20110085211A1
• Filed: 10/06/2010
• Published: 04/14/2011
• Est. Priority Date: 02/15/2004
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kushler, Clifford A., Stafford-Fraser, James Q., King, Martin T., Grover, Dale L.
• Granted Patent: US 8,619,147 B2
• Time in Patent Office: Days
• US Class Current: 358/474
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06K 17/0016 Selecting or retrieving of images by means of their associated code-marks, e.g. coded microfilm or microfiche microfilm reading G03B21/11; computers therefor G06F16/00; teaching devices G09B5/02; accessing record carriers G11B27/002; static digital stores using optical elements G11C13/04; intermediate storage of facsimile picture signals H04N1/21
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 20/102 Bill distribution or payments
• CPC Class Codes: G06Q 20/12 specially adapted for electronic shopping systems
• CPC Class Codes: G06Q 20/4012 Verifying personal identification numbers [PIN]
• CPC Class Codes: G06Q 30/00 : Commerce
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0207 : Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 30/0241 : Advertisements
• CPC Class Codes: G06Q 30/0283 : Price estimation or determination
• CPC Class Codes: G06Q 30/04 : Billing or invoicing
• CPC Class Codes: G06Q 30/06 : Buying, selling or leasing transactions
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0603 : Catalogue ordering
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0641 : Shopping interfaces
• CPC Class Codes: G06Q 30/0643 : Graphical representation of items or shoppers
• CPC Class Codes: G06Q 40/04 : Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G06V 10/20 : Image preprocessing
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 30/142 : using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/1444 : Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields
• CPC Class Codes: G06V 30/1448 : based on markings or identifiers characterising the document or the area
• CPC Class Codes: G06V 30/248 : involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q" G06V30/242 takes precedence
• CPC Class Codes: G06V 30/262 : using context analysis, e.g. lexical, syntactic or semantic context
• CPC Class Codes: G06V 30/40 : Document-oriented image-based pattern recognition
• CPC Class Codes: G07F 17/40 : for devices for accepting orders, advertisements, or the like
• CPC Class Codes: H04N 1/00204 : with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 1/00244 : with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00326 : with a data reading, recognizing or recording apparatus, e.g. with a bar-code apparatus arrangements for the associated working of recording or reproducing apparatus with related apparatus G11B31/00
• CPC Class Codes: H04N 1/0036 : Alphanumeric symbols
• CPC Class Codes: H04N 1/00363 : Bar codes or the like
• CPC Class Codes: H04N 1/00374 : on the same page as at least a part of the image
• CPC Class Codes: H04N 1/00381 : Input by recognition or interpretation of visible user gestures manual input means, e.g. digitisers, writing tablets H04N1/00392; gesture input to computers G06F3/017; recognition algorithms G06V40/20
• CPC Class Codes: H04N 1/107 : with manual scanning
• CPC Class Codes: H04N 1/2166 : for mass storage, e.g. in document filing systems information retrieval G06F16/00
• CPC Class Codes: H04N 1/2187 : with image input from a plurality of different locations or from a non-central location, e.g. from one or more users
• CPC Class Codes: H04N 1/32128 : attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 1/34 : for coin-freed systems ; Pay systems
• CPC Class Codes: H04N 2201/0081 : Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/3215 : of a time or duration
• CPC Class Codes: H04N 2201/3249 : data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3253 : Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3263 : of a graphical motif or symbol, e.g. Christmas symbol, logo H04N2201/3211 takes precedence
• CPC Class Codes: H04N 2201/3266 : of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 2201/3277 : The additional information being stored in the same storage device as the image data
• CPC Class Codes: H04N 2201/3278 : Transmission
• CPC Class Codes: Y10S 707/99931 : Database or file accessing
• CPC Class Codes: Y10S 707/99934 : Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99936 : Pattern matching access
• CPC Class Codes: Y10S 707/99943 : Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 : Object-oriented database structure processing

Abstract

A device for capturing rendered text is described. The device incorporates one or more visual sensors that receive visual information as a part of capturing rendered text. The visual sensors are collectively capable of capturing both text that is permanently printed on a page, and text that is displayed transitorily on a dynamic device. The device further incorporates a visual information disposition subsystem for disposing of visual information received by the visual sensors. The device further incorporates a package that bears the visual sensors and the visual information disposition subsystem, and is suitable to be held in a human hand.

Claims

1-20. -20. (canceled)

21. A method in a computing system for capturing rendered text, comprising:

using one or more visual sensors to receive visual information as part of capturing rendered text, visual sensors collectively being capable of capturing both text properly printed on a page and text display transitorily on a dynamic display device, the visual sensors being borne by a package suitable to be held in a human hand; and

transmitting to a computer system data representing a position on a dynamic display device from which one of the visual sensors is receiving visual information.

22. The method of claim 21 wherein the data is transmitted at a time when the visual sensors receiving visual information from the position represented by the transmitted data,

the method further comprising determining the text currently being displayed at the position on the display device represented by the transmitted data without receiving from the device data directly representing an image of such text.

23. The method of claim 21 wherein the data is transmitted at a time after the visual sensor received visual information from the position represented by the transmitted data,

the method further comprising:

transmitting to the computer system a timestamp indicating a time at which the visual sensor receive visual information from the position represented by the transmitted data; and

determining the text that was displayed at the position on the display device represented by the transmitted data at the time indicated by the timestamp.

24. The method of claim 21 wherein the package further bears optics through which light passes on a path from the rendered text at least one of the visual sensors, the optics being capable of varying depth of field from the perspective of the visual sensors.

25. A computer-readable medium having contents are capable of causing a computing system to perform a method for capturing rendered text, the method comprising:

using one or more visual sensors to receive visual information as part of capturing rendered text, visual sensors collectively being capable of capturing both text properly printed on a page and text display transitorily on a dynamic display device, the visual sensors being borne by a package suitable to be held in a human hand;

determining the position on the dynamic display device from which one of the visual senses is receiving visual information based upon visual information received by at least one of the visual sensors; and

disposing of visual information received by the visual sensors.

26. A method in a computing system for capturing rendered text, comprising:

using one or more visual sensors to receive visual information as part of capturing rendered text, visual sensors collectively being capable of capturing both text properly printed on a page and text display transitorily on a dynamic display device, the visual sensors being borne by a package suitable to be held in a human hand;

automatically determining when the visual sensors are receiving visual information as part of capturing text displayed transitorily on a dynamic display device by one of optically sensing a backlighting generated by the dynamic display device, optically sensing a cathode ray tube raster scan generated by the dynamic display device, and sensing a texture of a surface of the dynamic display device; and

transmitting to a computer system data relating to the capture.

27. A computer-readable medium having contents capable of causing a computing system to capture rendered text, the method comprising:

using one or more visual sensors to receive visual information as part of capturing rendered text, visual sensors collectively being capable of capturing both text properly printed on a page and text display transitorily on a dynamic display device, the visual sensors being borne by a package suitable to be held in a human hand;

automatically determining when the visual sensors are receiving visual information as part of capturing text displayed transitorily on a dynamic display device by one of optically sensing a backlighting generated by the dynamic display device, optically sensing a cathode ray tube raster scan generated by the dynamic display device, and sensing a texture of a surface of the dynamic display device; and

transmitting to a computer system data relating to the capture.

28. A method in a computing system for capturing rendered text, comprising:

using one or more visual sensors to receive visual information as part of capturing rendered text, visual sensors collectively being capable of capturing both text properly printed on a page and text display transitorily on a dynamic display device, the visual sensors being borne by a package suitable to be held in a human hand;

automatically determining when the visual sensors are receiving visual information as part of capturing text displayed transitorily on a dynamic display device; and

determining a sensing frequency for the visual sensors based upon the determination of when the visual sensors are receiving visual information as part of capturing text displayed transitorily on a dynamic display device.

29. A computer-readable medium having contents are capable of causing a computing system to perform a method for capturing rendered text, the method comprising:

using one or more visual sensors to receive visual information as part of capturing rendered text, visual sensors collectively being capable of capturing both text properly printed on a page and text display transitorily on a dynamic display device, the visual sensors being borne by a package suitable to be held in a human hand;

automatically determining when the visual sensors are receiving visual information as part of capturing text displayed transitorily on a dynamic display device; and

activating an illumination device to illuminate captured text only at times when it is determined that the visual sensors are not receiving visual information as part of capturing text displayed transitorily on a dynamic display device.

30. A computer-readable medium having contents are capable of causing a computing system to perform a method for determining the location of a text scanning optical sensor on a dynamic display, the method comprising:

presenting at one or more selected locations on a display an optical signal not detectable by a human user;

detecting in the optical sensor a presented optical signal; and

determining the location on the display at which the detected optical signal was presented.

31. The computer-readable medium of claim 30 wherein the optical signals are presented for too brief a period to be detectable by a human user.

32. The computer-readable medium of claim 30 wherein the presented optical signals are not visually distinguishable enough to be detectable by a human user.

33. A method in a computing system for capturing rendered text, comprising:

using a visual sensor to capture text of a document displayed transitorily on a dynamic display;

using the textual content of the captured text to identify the document and the position in the document at which the capture is performed;

determining a set of actions appropriate in connection with a capture from the identified document at the identified position; and

presenting at least a portion of the determined set of actions to a user.

34. The method of claim 33, further comprising:

receiving user input selecting one of the presented actions; and

in response to the received user input, performing the selected action in connection with the capture.

35. A computer-readable medium having contents capable of causing a computing system to perform a method for capturing rendered text, the method comprising:

using a visual sensor to capture text of a document displayed transitorily on a dynamic display;

using the textual content of the captured text to identify the document and the position in the document at which the capture is performed;

determining a set of actions appropriate in connection with a capture from the identified document at the identified position; and

presenting at least a portion of the determined set of actions to a user.

36. The A computer-readable medium of claim 35, the method further comprising:

receiving user input selecting one of the presented actions; and

in response to the received user input, performing the selected action in connection with the capture.