Wireless Real-Time Tongue Tracking for Speech Impairment Diagnosis, Speech Therapy with Audiovisual Biofeedback, and Silent Speech Interfaces

• US 20140342324A1
• Filed: 05/19/2014
• Published: 11/20/2014
• Est. Priority Date: 05/20/2013
• Status: Active Grant

• Current Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Original Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Inventors: Ghovanloo, Maysam, Block, Jacob
• Granted Patent: US 9,911,358 B2
• Time in Patent Office: Days
• US Class Current: 434/185
• CPC Class Codes: G09B 19/04 Speaking with audible presentation of the material to be studied G09B5/04
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied

Abstract

A real-time wireless system for recording natural tongue movements in the 3D oral space. By attaching a small magnetic tracer to the tongue, either temporarily or semi-permanently, and placing an array of magnetic sensors around the mouth, the tracer can be localized with sub-millimeter precision. The system can also combine the tracer localization with acoustic, video, and flow data via additional sensors to form a comprehensive audiovisual biofeedback mechanism for diagnosing speech impairments and improving speech therapy. Additionally, the system can record tongue trajectories and create an indexed library of such traces. The indexed library can be used as a tongue tracking silent speech interface. The library can synthesize words, phrases, or execute commands tied to the individual patterns of magnetic field variations or tongue trajectories.

Claims

1. A system for wirelessly tracking at least one of a position, orientation, and movement of a tongue of a user during speech comprising:

a tracer unit adapted to be non-obstructively affixed to the tongue of the user;

a sensor system comprising a plurality of sensors, at least one sensor detecting a tongue characteristic, the sensor system in non-obstructive proximity to the tracer unit and mouth of the user;

a sensor control unit for transmitting a sensor signal related to at least one tongue characteristic to a receiver unit;

the receiver unit for processing at least one sensor signal and transmitting the processed signal to a computing platform;

the computing platform adapted to perform a function selected from the group consisting of;

additional processing of the processed signal;

tracking of the tracer unit;

fusing multiple processed signals;

generation of a graphical representation of the processed signal; and

displaying the graphical representation of the processed signal.

2. The system of claim 1 further comprising a plurality of sensor control units, a sensor control unit associated with a sensor, for transmitting a tongue characteristic from that sensor to the receiver unit;

wherein a change in at least one of the position and orientation of the tongue in proximity to the tracer provides an attendant change in at least one of the position and orientation of the tracer unit;

wherein a tongue characteristic is selected from the group consisting of a position of the tracer unit, an orientation of the tracer unit, a sound produced by the user, an air flow produced by the user, a lip movement produced by the user, a lip gesture produced by the user, a physical contact between the tongue and a palate of the user, a muscle movement produced by the user, and an electrical activity of a muscle or brain of the user;

wherein each sensor control unit transmits a sensor signal related to the tongue characteristic to the receiver unit;

wherein the receiver unit processes at least one sensor signal and transmits the processed signal and related data to the computing platform;

the computing platform adapted to perform a function selected from the further group consisting of;

additional processing of the processed signal and related data;

fusing multiple processed signals and related data;

generation of a graphical representation of the processed signal and related data; and

displaying the graphical representation of the processed signal and related data.

3. The system of claim 2, the tracer unit comprising a magnet.

4. The system of claim 2, the sensor system adapted to be placed within the mouth of the user.

5. The system of claim 2, the sensor system adapted for positioning outside the mouth of the user.

6. The system of claim 2, the sensor system comprising a plurality of magnetic field sensors and at least one of a microphone, a camera, an air pressure sensor, an air flow sensor, an electrode array adapted for electromyography of the user'"'"'s oral articulator muscles, an electrode array adapted for electropalatography, and an electrode array adapted for electroencephalography.

7. The system of claim 2 further comprising a remote display distal the user.

8. The system of claim 2, wherein the graphical representation of the processed signals and related data comprises at least one of a tracing, line drawing, 3D drawing, feedback, visual feedback, visual biofeedback, audiovisual biofeedback, vibrotactile biofeedback, and an interactive game.

9. The system of claim 7, wherein the remote display is capable of being used by a remote user selected from the group consisting of a speech therapist, a speech and language pathologist, a clinician, a tutor, a parent, a caretaker, a physician, and an insurance agent.

10. The system of claim 8, wherein the graphical representation is accompanied by an audio recording corresponding to the processed signals and related data.

11. The system of claim 9 further comprising a feedback system wherein a remote user of the remote display provides feedback to the user.

12. The system of claim 11, wherein the feedback is selected from the group consisting of therapeutic tasks for the user, a representation of correct tongue movement during speech of a target phoneme, an interactive game to quantify and monitor the user'"'"'s progress, audio feedback of correct speech of a target phoneme, visual feedback of correct speech of a target phoneme, audiovisual biofeedback of correct speech of a target phoneme, and vibrotactile biofeedback of correct speech of a target phoneme.

13. A system for wirelessly tracking at least one of a position, orientation, and movement of a tongue of a user during speech, the system comprising:

a tracer unit adapted to be non-obstructively affixed to the tongue of the user, wherein a change in position or orientation of the tongue of the user changes the position or orientation of the tracer unit;

a sensor system comprising a plurality of sensors, the sensor system detecting at least one signal indicative of at least one of;

a position of the tracer unit;

an orientation of the tracer unit;

an air flow produced by the user;

a lip movement produced by the user;

a lip gesture produced by the user;

a physical contact between the tongue and a palate of the user;

a muscle movement produced by the user;

an electrical activity of a muscle of the user; and

an electrical activity of a brain of the user;

wherein

a plurality of sensor units for transmitting at least one sensor signal to a controller unit based on the detected position, orientation, and movement of the tracer unit;

the controller unit for processing the at least one sensor signal and transmitting the processed signal and related data to a computing platform;

the computing platform adapted to perform at least one function comprising;

additional processing of the processed signal and related data;

tracking of the tracer unit;

fusing multiple processed signals and related data;

generation of a graphical representation of the processed signals and related data; and

displaying the graphical representation of the processed signals and related data.

14. The system of claim 13, the controller unit comprising a high-speed field programmable gate array.

15. A method of wirelessly tracking at least one of a position, orientation, and movement of a tongue of a user during speech comprising:

affixing a tracer unit in a non-obstructively manner to the tongue of the user;

positioning a sensor system in non-obstructive proximity to the tongue carrying the tracer unit;

detecting a tongue characteristic with the sensor system;

transmitting the tongue characteristic to a computing platform for processing; and

processing the tongue characteristic to obtain information comprising at least one of the position of the tracer unit and the orientation of the tracer unit during speech by the user.

16. The method according to claim 15, wherein a tongue characteristic is selected from the group consisting of a position of the tracer unit, an orientation of the tracer unit, a sound produced by the user, an air flow produced by the user, a lip movement produced by the user, a lip gesture produced by the user, a physical contact between the tongue and a palate of the user, a muscle movement produced by the user, and an electrical activity of a brain of the user.

17. The method according to claim 16 further comprising displaying a representation of the tongue characteristic, wherein the representation is selected from the group consisting of a tracing, line drawing, 3D drawing, visual feedback, audio feedback, visual biofeedback, audiovisual biofeedback, vibrotactile biofeedback, a trajectory of the user'"'"'s tongue during speech, and an interactive game.

18. The method according to claim 16 further comprising:

displaying a representation of the tongue characteristic, wherein the representation comprises a trajectory of the user'"'"'s tongue during speech and a magnetic and acoustic signatures of the magnetic tracer attached to the user'"'"'s tongue during speech;

wherein the magnetic and acoustic signature is comprised of a plurality of signals and waveforms generated by the sensor system during speech by the user; and

cataloging a plurality of trajectories and magnetic and acoustic signatures of the user'"'"'s tongue during speech into a library for use as a silent speech interface, wherein the plurality of trajectories and magnetic and acoustic signatures correspond to at least one of a word, a phrase, or a command.

19. The method according to claim 16, wherein the sensor system is incorporated into a headset comprising a head support, at least one controller, a receiver, and a wireless transmitter.

20. The method according to claim 16, wherein processing the tongue characteristic to obtain information comprises at least one of the position of the tracer unit and the orientation of the tracer unit during speech by the user comprises amplification, filtering, digitization, magnetic localization, smoothing, and pattern recognition of the magnetic sensors.

21. The method according to claim 17, wherein the representation of the tongue characteristic is generated according to the steps of showing the raw magnetic sensor outputs;

generating a fused combination of at least one sensor output comprising output from at least one magnetic and acoustic sensor;

calculating a vector sum representation of the magnetic sensor outputs;

producing a change of color based on the pitch or amplitude of the speech volume based on output from the acoustic sensors; and

showing the tongue trajectory in at least one of 1D, 2D, or 3D, an interactive game, and a goal-oriented game.