Gesture processing with low resolution images with high resolution processing for optical character recognition for a reading machine

• US 8,150,107 B2
• Filed: 11/16/2009
• Issued: 04/03/2012
• Est. Priority Date: 04/02/2004
• Status: Active Grant

• Current Assignee: National Federation of the Blind
• Original Assignee: K-NFB Reading Technology Inc.
• Inventors: Kurzweil, Raymond C., Albrecht, Paul, Gashel, James, Gibson, Lucy, Lvovsky, Lev
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US12/618,858
• Publication Number:
  US 20100074471A1
• Time in Patent Office: 869 Days
• Field of Search: 382/114, 348/62
• US Class Current: 382/114
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/1444 Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields

Abstract

A portable reading machine that operates in several modes and performs image preprocessing to prior to optical character recognition. The portable reading machine receives a low resolution image and a high resolution image of a scene and processing the low resolution image to recognize a user-initiated gesture using a gesturing item that indicates a command from the user to the reading machine and the high resolution image to recognize text in the image of the scene, according to the command from the user to the machine.

Claims

1. A method of operating a portable reading machine, the method comprising:

capturing by an image capture device, an image of a scene;

accessing a knowledge base that stores information for a gesturing mode the information to permit the portable reading machine to preferentially find specific types of visual elements of the scene;

processing the captured image by a computing device in the reading machine using information from the knowledge base to recognize a user-initiated gesture using a gesturing item, the gesture indicating a command from the user to the reading machine to describe to the user specific types of visual elements in the image; and

applying optical character recognition to recognize text in the image.

2. The method of claim 1 wherein the gesturing item is a portion of the user'"'"'s hand or an item held by the user.

3. The method of claim 2 wherein capturing comprises receiving a series of images;

and

processing the series of images to detect the presence of the gesturing item in the series of images.

4. The method of claim 2 wherein processing comprises:

applying pattern-recognition processing to the image to detect one of a set of pre-defined gestures that are interpreted by the reading machine as commands to the machine to describe specific visual elements found in the image.

5. The method of claim 1 wherein processing different sections comprises:

retrieving a template that a scene in the image is mapped to; and

processing the template with the gesturing commands to process portions of the scene corresponding to named parts of the template.

15. The method of claim 1 wherein the one or more target elements in the scene is a number on a door.

16. The method of claim 1 wherein the one or more target elements in the scene is an exit sign.

17. The method of claim 1 wherein the knowledge base is stored in computer storage on the reading machine.

18. The method of claim 1 wherein the knowledge base is downloaded to the reading machine.

19. The method of claim 1 wherein upon completion of processing of the image, the reading machine presents results to the user as speech to convey the identified item to the user.

20. The method of claim 1 further comprising:

processing the image using at least one, different mode of processing;

determining which of the gesture processing mode and the different mode of processing provides valid results; and

presenting the valid results.

27. The machine of claim 1 further configured to:

process the image using at least one, different mode of processing;

determine which of the gesture processing mode and the different mode of processing provides valid results; and

present the valid results.

6. A computer program product tangibly stored on a computer readable storage device- for operating a portable reading machine, the computer program product comprising instructions to:

receive an image of a scene from an capture device;

access a knowledge base that stores information for a gesturing mode, the information to permit the portable reading machine to preferentially find specific types of visual elements of the scene;

process the image to recognize a user-initiated gesture using a gesturing item the gesture indicating a command from the user to the reading machine to describe to the user specific types of visual elements found in the image based on information from the knowledge base in the image; and

apply optical character recognition to recognize text in the image.

7. The computer program product of claim 6 wherein the gesturing item is a portion of the user'"'"'s hand or an item held by the user and the computer program product further comprises instructions to:

receive the image to process the gesturing item and

to recognize text.

8. The computer program product of claim 7 wherein instructions to receive comprises instructions to:

receive a series of images; and

process the series of images to detect the presence of the gesturing item in the series of images.

9. The computer program product of claim 7 further comprises instructions to:

apply pattern-recognition processing to the image to detect one of a set of pre-defined gestures that are interpreted by the reading machine, as commands to the machine to describe specific visual elements found in the image.

10. The computer program product of claim 6 wherein instructions to process different sections, comprises instructions to:

retrieve a template that a scene in the image is mapped to; and

process the template with the gesturing commands to process portions of the scene corresponding to named parts of the template.

21. The computer program product of claim 6 wherein the one or more target elements in the scene recognized by the instructions is a number on a door.

22. The computer program product of claim 6 wherein the one or more target elements in the scene recognized by the instructions is an exit sign.

23. The computer program product of claim 6 wherein the knowledge base is stored in computer storage on the reading machine.

24. The computer program product of claim 6 wherein upon completion of processing of the image, the instructions present results to the user as speech to convey the identified item to the user.

11. A portable reading machine comprising:

a computing device;

an image input device coupled to the computing device for capturing images; and

a computer readable storage device storing a computer program product comprising instructions to cause the computing device to;

receive a image of a scene captured from the image input device;

access a knowledge base that stores information for a gesturing mode, the information to permit the portable reading machine to preferentially find specific types of visual elements of the scene;

process the image using information from the knowledge base to recognize a user-initiated gesture using a gesturing item that indicates a command from the user to the reading machine to describe to the user specific types of visual elements found in the image based on information from the knowledge base;

apply optical character recognition to process the image to recognize text in the image according to the command;

apply speech synthesis to the recognized text to render the recognized text as speech; and

render as speech the description of the specific visual elements.

12. The machine of claim 11 wherein the instructions of the computer program product to receive a image, comprises instructions to

receive a series of images;

and

process the series of images to detect the presence of the gesturing item in the series of images.

13. The machine of claim 11 wherein the instructions of the computer program product, further comprises instructions to:

apply pattern-recognition processing to the image to detect one of a set of pre-defined gestures that are interpreted by the reading machine, as commands to the machine to describe specific visual elements found in the image.

14. The machine of claim 11 wherein the instructions of the computer program product to process different sections, further comprises instructions to:

retrieve a template that a scene in the image is mapped to; and

process the template with the gesturing commands to process portions of the scene corresponding to named parts of the template.

25. The machine of claim 11 wherein the knowledge base is stored in computer storage on the reading machine.

26. The machine of claim 11 wherein the knowledge base is downloaded to the reading machine.