Communication and control system and method

• US 10,372,204 B2
• Filed: 10/30/2014
• Issued: 08/06/2019
• Est. Priority Date: 10/30/2013
• Status: Active Grant

• Current Assignee: Technology Against ALS
• Original Assignee: Technology Against ALS
• Inventors: Tungare, Sandeep, Gupte, Milind, Patil, Amitabh, Reddy, Naveen, Haberland, Bert
• Primary Examiner(s): Soto Lopez, Jose R
• Application Number:
  US15/027,731
• Publication Number:
  US 20160246367A1
• Time in Patent Office: 1,741 Days
• CPC Class Codes: A61F 4/00 Methods or devices enabling patients or disabled persons to operate an apparatus or a device not forming part of the body operating or control means for prostheses A61F2/48, A61F2/68
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B

Abstract

An input system includes at least one sensor and a controller communicatively coupled to the at least one sensor. The at least one sensor is configured to identify a position of an eye. The controller is configured to receive a first signal indicative of a first position of an eye from the at least one sensor and map the eye position to a user'"'"'s selection.

Claims

1. A system, comprising:

a support structure for supporting a plurality of visible indicia such that the plurality of visible indicia is located at a periphery of a user'"'"'s field of vision, the plurality of visible indicia collectively establishing a physical reference frame having a plurality of colors and configured to provide a first feedback mechanism;

a second feedback mechanism;

a sensor configured to identify a position of a pupil of an eye; and

a controller communicatively coupled to the sensor, the plurality of visible indicia, and the second feedback mechanism, the controller configured to;

receive a first signal from the sensor, the first signal indicative of a gaze direction of the pupil,

identify, based on the first signal, a gazed upon visible indicia, the gazed upon visible indicia being one of the plurality of visible indicia that most closely corresponds to the gaze direction,

map the gazed upon visible indicia to a virtual reference frame comprising a plurality of members of a set,

identify a provisional input selection from the plurality of members of the set,

cause at least one of the first feedback mechanism and the second feedback mechanism to signal that the provisional input selection has been made,

update the virtual reference frame if the provisional input selection is not one of a group of predetermined final input selections, and

transmit a second signal to the second feedback mechanism or to another device to initiate an action based on the provisional input selection if the provisional input selection is one of the group of predetermined final input selections.

2. The system of claim 1, wherein the signal that the provisional input selection has been made includes at least one of changing a brightness of at least one of the plurality of visible indicia and blinking at least one of the plurality of visible indicia.

3. The system of claim 2, wherein changing the brightness of at least one of the plurality of visible indicia includes gradually increasing the brightness of the at least one of the plurality of visible indicia.

4. The system of claim 1, wherein the support structure includes a frame which supports an infrared light source for illuminating the eye.

5. The system of claim 1, wherein the second feedback mechanism includes at least one of a speaker, an earpiece, a mounted heads-up display, and a visual display.

6. The system of claim 1, wherein the support structure includes glasses for being worn by the user.

7. The system of claim 6, further comprising a single camera located external to the support structure.

8. The system of claim 7, wherein the support structure includes an arm that extends in front of the glasses and supports the camera.

9. The system of claim 1, wherein the second feedback mechanism includes a visual display, and wherein the visual display is configured to display a representation of the virtual reference frame.

10. The system of claim 1, wherein the group of final input selections includes at least one of a letter, a number, a word, a phrase, and a sentence.

11. The system of claim 1, wherein the controller is configured to transmit a control signal to another device if the provisional input selection is one of the group of final input selections, and wherein the another device is selected from the group consisting of a television, a medical support device, and a light.

12. The system of claim 1, wherein the controller is configured to predict user selections using word and sentence completion techniques.

13. A method, comprising:

receiving, by a processor, a first signal from a sensor, the first signal indicative of a gaze direction of a pupil of an eye of a user gazing at at least one of a plurality of visible indicia located at a periphery of the user'"'"'s field of vision, the plurality of visible indicia establishing a physical reference frame having a plurality of colors and configured to provide a first feedback mechanism;

identify, based on the first signal, a gazed upon visible indicia, the gazed upon visible indicia being one of the plurality of visible indicia;

mapping, by the processor, the gazed upon visible indicia to a virtual reference frame comprising a plurality of members of a set;

identifying a provisional input selection from the plurality of members of the set;

causing at least one of the first feedback mechanism and a second feedback mechanism to signal that the provisional input selection has been made;

updating the virtual reference frame if the provisional input selection is not one of a group of predetermined final input selections; and

transmitting a second signal to the second feedback mechanism or to another device to initiate an action based on the provisional input selection if the provisional input selection is one of the group of predetermined final input selections.

14. The method of claim 13, wherein the first feedback mechanism signals that the provisional input selection has been identified.

15. The method of claim 14, further comprising adjusting a brightness of at least one of the plurality of visible indicia in response to identifying the provisional input selection.

16. The method of claim 15, wherein adjusting the brightness of at least one of the plurality of visible indicia includes gradually increasing the brightness.

17. The method of claim 14, further comprising causing at least one of the plurality of visible multi-colored indicia to blink in response to identifying the provisional input selection.

18. The method of claim 13, wherein the second signal is transmitted to the second feedback mechanism, the second feedback mechanism includes a controllable device, and the method further comprises causing the controllable device to perform a function in response to receiving the second signal.

19. A system, comprising:

a support structure for supporting a plurality of visible indicia such that the plurality of visible indicia is located at a periphery of a user'"'"'s field of vision, the plurality of visible indicia collectively establishing an external reference frame;

at least one sensor configured to identify a position of a pupil of an eye; and

a controller communicatively coupled to the at least one sensor and the plurality of visible indicia, the controller configured to;

display a virtual reference frame on a visual display, the virtual reference frame comprising a matrix comprising a plurality of cells of alphabetic characters,

receive a first signal from the at least one sensor, the first signal indicative of a first gaze direction of the pupil gazing at at least one of the plurality of visible indicia,

map the first position of the pupil relative to the external reference frame to a row or column of the virtual reference frame,

receive a second signal from the at least one sensor, the second signal indicative of a second position of the pupil gazing at at least one of the plurality of visible indicia;

map the second position of the pupil relative to the external reference frame to the other of a row or column of the virtual reference frame; and

identify a first input selection based on the mapping of the first position and the second position to the virtual reference frame, the first input selection being, of the plurality of cells, the cell corresponding to the row and column selected by the user.