Voice-output reading system with gesture-based navigation

• US 6,115,482 A
• Filed: 10/22/1998
• Issued: 09/05/2000
• Est. Priority Date: 02/13/1996
• Status: Expired due to Fees

• Current Assignee: Ascent Technology, Inc.
• Original Assignee: Ascent Technology, Inc.
• Inventors: Sears, James T., Goldberg, David A.
• Primary Examiner(s): BELLA, MATTHEW C
• Application Number:
  US09/176,999
• Time in Patent Office: 684 Days
• Field of Search: 382/103, 382/173, 382/177, 382/114, 348/62, 235/472, 345/173, 345/157
• US Class Current: 382/114
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G06F 2203/0331 Finger worn pointing device
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09B 21/001 Teaching or communicating with blind persons G09B21/02 - G09B21/06 take precedence
• CPC Class Codes: G09B 21/006 using audible presentation of the information
• CPC Class Codes: G09B 21/007 using both tactile and audible presentation of the information
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems

Abstract

An optical-input print reading device with voice output for people with impaired or no vision in which the user provides input to the system from hand gestures. Images of the text to be read, on which the user performs finger- and hand-based gestural commands, are input to a computer, which decodes the text images into their symbolic meanings through optical character recognition, and further tracks the location and movement of the hand and fingers in order to interpret the gestural movements into their command meaning. In order to allow the user to select text and align printed material, feedback is provided to the user through audible and tactile means. Through a speech synthesizer, the text is spoken audibly. For users with residual vision, visual feedback of magnified and image enhanced text is provided. Multiple cameras of the same or different field of view can improve performance. In addition, alternative device configurations allow portable operation, including the use of cameras located on worn platforms, such as eyeglasses, or on a fingertip system. The use of gestural commands is natural, allowing for rapid training and ease of use. The device also has application as an aid in learning to read, and for data input and image capture for home and business uses.

Claims

1. A method for electronically reading text under interactive control by a user, the method comprising:

obtaining a first digital image of at least a portion of the text to be read;

performing symbology recognition on the first digital image;

capturing a temporal sequence of digital images of user-generated spatial configurations of at least one pointer;

determining a command signal from the temporal sequence of digital images;

choosing a subset of the recognized symbology to read at a spatial location on the text to be read on the basis of the determined command signal;

feeding back to the user the spatial location of the chosen subset to be read and the spatial location of at least one pointer relative to the recognized text, wherein the spatial location of the chosen subset and the pointer aids the user in selecting the text to be read; and

converting the chosen subset of recognized symbology into a humanly understandable version.

2. A method as defined in claim 1, wherein the pointer is comprised of a finger on the user'"'"'s hand.

3. A method as defined in claim 1, wherein a portion of the spatial configurations are comprised of the relative locations and orientations of two or more pointers.

4. A method as defined in claim 1, wherein a portion of the spatial configurations are comprised of the locations of the pointer relative to the text.

5. A method as defined in claim 1, wherein the pointer is attached to the user'"'"'s hand.

6. A method as defined in claim 1, wherein the spatial configurations of the pointer are determined from the digital image.

7. A method as defined in claim 1, wherein the digital image is obtained from a means attached to the user'"'"'s hand.

8. A method as defined in claim 7, wherein determining command signals comprises analyzing the relative location of text within the digital image from successive frames in order to determine the location and motion of the means of obtaining the digital image relative to the text.

9. A method as defined in claim 1, wherein a camera is used to obtain digital images and a second camera is used to capture temporal sequences of digital images.

10. A method as defined in claim 9, wherein the cameras have substantially different sized fields of view.

11. A method as defined in claim 1, wherein a single camera is used to both obtain the first digital image and to capture the temporal sequence of digital images.

12. A method as defined in claim 1, wherein the operation of feeding back includes providing feedback directly on the text to be read.

13. A method as defined in claim 1, wherein the operation of feeding back includes providing feedback through an electronic display of at least a portion of the digital image.

14. A method as defined in claim 1, wherein a flatbed scanner is used to obtain digital images.

15. A method as defined in claim 1, wherein digital images are obtained from an internal computer video buffer of a screen image.

16. A method as defined in claim 1, wherein the spatial configurations of the pointer are determined from the interaction of a proximity-sensitive display with the pointer.

17. A method as defined in claim 1, wherein the act of obtaining digital images can be accomplished via an imaging device worn by the user.

18. A method as defined in claim 17, wherein the imaging device is incorporated into glasses worn by the user.

19. A method as defined in claim 1, including additionally displaying enhanced images of the text to the user.

20. A method as defined in claim 19, wherein the degree of enhancement is controlled by the user.

21. A method as defined in claim 20, wherein the degree of image enhancement is controlled by the command signal.

22. A method as defined in claim 19, wherein the image enhancement is a magnification of the text.

23. A method as defined in claim 19, wherein the image enhancement comprises affecting a property of the image chosen from the group consisting of contrast, brightness, bit-depth, and color polarity.

24. A method as defined in claim 1, including additionally providing the user feedback information about the layout of text within the digital image.

25. A method as defined in claim 1, wherein the spatial information is fed back via tactile information.

26. A method as defined in claim 25, wherein the tactile information is transduced by vibrating pins.

27. A method as defined in claim 25, wherein the tactile information is transduced by a force-feedback mouse.

28. A method as defined in claim 1, wherein the spatial information is fed back via visual information.

29. A method as defined in claim 28, wherein the visual information is displayed on the text using a laser scanner.

30. A method as defined in claim 1, wherein the spatial information is fed back via audible information.

31. An electronic reading apparatus for converting text to spoken words for a user, comprising:

a pointer that is operated by the user to indicate commands, wherein commands are encoded in the location and movement of the pointer relative to the text;

a digital imaging device that converts text to a digital imaging signal and which additionally captures a temporal sequence of digital images of the pointer, wherein the temporal sequence of images contains information about the location and movement of the pointer relative to the text;

a character recognizer receptive of the digital imaging signal, the recognizer generating a recognized character signal comprising the symbolic identity of the recognized text and the location of the recognized text relative to the digital imaging signal;

a pointer tracker that determines the pointer location and movement, the tracker generating a pointer location and movement signal relative to the text;

a command interpreter receptive of the pointer location and movement signal and the recognized character signal, the interpreter generating a command signal indicating the text to be converted to spoken words;

a feedback device receptive of the command signal, and providing feedback to the user about the location of the text to be converted to spoken words and the location of the pointer relative to recognized text;

a controller receptive of the command signal and the recognized character signal, the controller generating an output signal representative of at least portions of the text recognized; and

a transducer receptive of the output signal for converting the output signal to speech.

32. A device as defined in claim 31, wherein the feedback device is additionally receptive of the recognized character signal and provides feedback to the user about the location of recognized text.

33. A method for electronically reading aloud text under interactive control by a user with a computer-based system, the method comprising:

obtaining a first image of at least a portion of the text and converting the image to a first signal representative thereof;

capturing a temporal sequence of images of at least one pointer under the control of the user and converting the temporal sequence of images to a second signal representative thereof;

supplying the first and second signals to the computer-based system;

performing symbology recognition on the first signal;

determining the position and movement of the pointer relative to the text from the first and second signals;

choosing a subset of the recognized symbology to read aloud on the basis of the determined position and movement of the pointer relative to the text and generating a command signal;

converting the chosen subset of recognized symbology into speech, additionally including providing feedback to the user of the particular subset of recognized symbology chosen and of the location of the pointer relative to the recognized symbology.