Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display device

• US 9,773,167 B2
• Filed: 09/30/2016
• Issued: 09/26/2017
• Est. Priority Date: 04/19/2004
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: King, Martin T., Grover, Dale L., Kushler, Clifford A., Stafford-Fraser, James Q.
• Primary Examiner(s): HANNETT, JAMES M
• Application Number:
  US15/281,732
• Publication Number:
  US 20170017837A1
• Time in Patent Office: 361 Days
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 30/00 Commerce
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/36 Matching; Classification
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images
• CPC Class Codes: H04N 1/00241 using an image reading device as a local input to a computer input via network, e.g. network scanners H04N1/00225
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00488 providing an audible output to the user
• CPC Class Codes: H04N 1/0049 providing a visual indication to the user, e.g. using a lamp H04N1/00408 takes precedence
• CPC Class Codes: H04N 1/107 with manual scanning
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 2201/0041 Point to point H04N2201/0039 takes precedence
• CPC Class Codes: H04N 2201/0055 By radio
• CPC Class Codes: H04N 2201/0058 Docking-station, cradle or the like
• CPC Class Codes: H04N 2201/0081 Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0096 : Portable devices
• CPC Class Codes: H04N 2201/3249 : data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3264 : of sound signals
• CPC Class Codes: H04N 2201/3269 : of machine readable codes or marks, e.g. bar codes or glyphs

Abstract

A device for capturing rendered text is described. The device incorporates one or more visual sensors that receive visual information as a part of capturing rendered text. The visual sensors are collectively capable of capturing both text that is permanently printed on a page, and text that is displayed transitorily on a dynamic device. The device further incorporates a visual information disposition subsystem for disposing of visual information received by the visual sensors. The device further incorporates a package that bears the visual sensors and the visual information disposition subsystem, and is suitable to be held in a human hand.

Claims

1. A computer-implemented method comprising:

receiving an image of at least a portion of a rendered document;

determining word lengths of words from the image of the at least a portion of the rendered document;

generating a code based on the determined word lengths of the words from the image and a sequence of the determined word lengths that corresponds to a sequence of words from the image;

searching an index of electronic documents using the generated code, wherein the index of electronic documents includes data indexing the electronic documents based on word lengths of words included in the electronic documents; and

identifying, based on the generated code, an electronic document indexed in the index and that corresponds to the rendered document.

2. The method of claim 1, wherein the image of the at least a portion of the rendered document is captured at a handheld user device that includes an integrated visual sensor and an integrated visual display.

3. The method of claim 2, further comprising providing the handheld user device with access to the identified electronic document that corresponds to the rendered document.

4. The method of claim 1, wherein determining word lengths of words from the image is done without resolving individual characters of the words.

5. The method of claim 1, wherein determining word lengths of words from the image includes classifying words as having word lengths of short, long, and undetermined.

6. The method of claim 5, wherein a given word that has been classified as having an undetermined length is associated with a wildcard character in the generated code that is used when searching the index of electronic documents that have been indexed based on word lengths of words included in the electronic documents.

7. The method of claim 5, wherein a given word is classified as having a short length when the given word has less than a predetermined number of characters.

8. The method of claim 5, wherein a given word is classified as having a long length when the given word has more than a predetermined number of characters.

9. The method of claim 1, wherein the generated code describes the word lengths of the words from the image, further comprising determining individual characters of each word of the words from the image by locating a contiguous string of words in the identified electronic document having word lengths that matches the word lengths described by the generated code.

10. The method of claim 1, further comprising identifying font information associated with the words from the image, wherein the font information is used to disambiguate results of searching the index of electronic documents.

11. Non-transitory computer-readable media having contents capable of causing a computing system including one or more computers to perform operations comprising:

receiving an image of at least a portion of a rendered document;

determining word lengths of words from the image of the at least a portion of the rendered document;

generating a code based on the determined word lengths of the words from the image and a sequence of the determined word lengths that corresponds to a sequence of words from the image;

searching an index of electronic documents using the generated code, wherein the index of electronic documents includes data indexing the electronic documents based on word lengths of words included in the electronic documents; and

identifying, based on the generated code, an electronic document indexed in the index and that corresponds to the rendered document.

12. The non-transitory computer-readable media of claim 11, wherein the image of the at least a portion of the rendered document is captured at a handheld user device that includes an integrated visual sensor and an integrated visual display.

13. The non-transitory computer-readable media of claim 12, the operations further comprising providing the handheld user device with access to the identified electronic document that corresponds to the rendered document.

14. The non-transitory computer-readable media of claim 11, wherein determining word lengths of words from the image is done without resolving individual characters of the words.

15. The non-transitory computer-readable media of claim 11, wherein the generated code describes the word lengths of the words from the image, the operations further comprising determining individual characters of each word of the words from the image by locating a contiguous string of words in the identified electronic document having word lengths that matches the word lengths described by the generated code.

16. A system, comprising one or more computers having one or more memory devices on which are stored computer executable instructions operable to cause the one or more computers to perform operations comprising:

receiving an image of at least a portion of a rendered document;

determining word lengths of words from the image of the at least a portion of the rendered document;

generating a code based on the determined word lengths of the words from the image and a sequence of the determined word lengths that corresponds to a sequence of words from the image;

searching an index of electronic documents using the generated code, wherein the index of electronic documents includes data indexing the electronic documents based on word lengths of words included in the electronic documents; and

identifying, based on the generated code, an electronic document indexed in the index and that corresponds to the rendered document.

17. The system of claim 16, wherein the image of the at least a portion of the rendered document is captured at a handheld user device that includes an integrated visual sensor and an integrated visual display.

18. The system of claim 17, the operations further comprising providing the handheld user device with access to the identified electronic document that corresponds to the rendered document.

19. The system of claim 16, wherein determining word lengths of words from the image is done without resolving individual characters of the words.

20. The system of claim 16, wherein the generated code describes the word lengths of the words from the image, the operations further comprising determining individual characters of each word of the words from the image by locating a contiguous string of words in the identified electronic document having word lengths that matches the word lengths described by the generated code.