Wireless real-time tongue tracking for speech impairment diagnosis, speech therapy with audiovisual biofeedback, and silent speech interfaces

• US 9,911,358 B2
• Filed: 05/19/2014
• Issued: 03/06/2018
• Est. Priority Date: 05/20/2013
• Status: Active Grant

• Current Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Original Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Inventors: Ghovanloo, Maysam, Block, Jacob
• Primary Examiner(s): Yao, Sam
• Assistant Examiner(s): Page, Evan
• Application Number:
  US14/281,794
• Publication Number:
  US 20140342324A1
• Time in Patent Office: 1,387 Days
• CPC Class Codes: G09B 19/04 Speaking with audible presentation of the material to be studied G09B5/04
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied

Abstract

A real-time wireless system for recording natural tongue movements in the 3D oral space. By attaching a small magnetic tracer to the tongue, either temporarily or semi-permanently, and placing an array of magnetic sensors around the mouth, the tracer can be localized with sub-millimeter precision. The system can also combine the tracer localization with acoustic, video, and flow data via additional sensors to form a comprehensive audiovisual biofeedback mechanism for diagnosing speech impairments and improving speech therapy. Additionally, the system can record tongue trajectories and create an indexed library of such traces. The indexed library can be used as a tongue tracking silent speech interface. The library can synthesize words, phrases, or execute commands tied to the individual patterns of magnetic field variations or tongue trajectories.

Claims

1. A system for real-time tracking of at least one of a position, orientation, and movement of a tongue of a user during speech comprising:

a tracer unit;

magnetic sensors;

a speech characteristic sensor;

a sensor control unit;

a computing platform; and

a feedback system;

wherein the tracer unit is adapted to be non-obstructively affixed at a location on the tongue of the user;

wherein the magnetic sensors are configurable to measure a magnetic flux density at the location of the tracer unit, the magnetic sensors in non-obstructive proximity to the tracer unit and mouth of the user;

wherein the speech characteristic sensor configurable to detect a speech characteristic selected from the group consisting of sound produced by the user, an air flow produced by the user, a lip movement/gesture produced by the user, a physical contact between the tongue and a palate of the user, an articulator muscle movement produced by the user, and an electrical activity of a brain of the user;

wherein the sensor control unit is configurable to;

receive a magnetic sensor signal from a magnetic sensors;

receive a speech characteristic sensor signal from the speech characteristic sensor;

synchronize the magnetic sensor signal with the speech characteristic sensor signal; and

transmit the synchronized sensor signal to the computing platform;

wherein the computing platform is

configurable to;

perform a tongue tracking signal processing algorithm, the algorithm including minimizing a cost function that correlates to the measured magnetic flux density and an estimated magnetic flux density at the location of the tracer unit, the location of the tracer unit being defined within a three-dimensional, non-discrete space;

determine, based on the algorithm, one or more of a position, orientation, and movement of the tracer unit in real time within the three-dimensinal, non-discrete space;

generate a combination of the one or more determined position, determined orientation, and determined movement of the tracer unit with the speech characteristic sensor signal; and

transmit the combination to the feedback system; and

wherein the feedback system is configurable to provide assistive speech-related feedback to the user in real time, based on the combination, wherein the assistive speech-related feedback is selected from the group consisting of a representation of temporally and spatially continuous tracking of the user'"'"'s tongue movement during speech of a target phoneme within the three-dimensional non-discrete space, an interactive game to quantify and monitor the user'"'"'s progress, and vibrotactile biofeedback comparing speech of the user to correct speech of a target phoneme.

2. The system of claim 1, the tracer unit comprising a magnet.

3. The system of claim 1, the magnetic sensors adapted to be placed within the mouth of the user.

4. The system of claim 1, the magnetic sensors adapted for positioning outside the mouth of the user.

5. The system of claim 1, the speech characteristic sensor comprising at least one of a microphone, a camera, an air pressure sensor, an air flow sensor, an electrode array adapted for electromyography of the user'"'"'s oral articulator muscles, an electrode array adapted for electropalatography, and an electrode array adapted for electroencephalography.

6. The system of claim 1 further comprising a remote display distal the user.

7. The system of claim 6, wherein the remote display is capable of being used by a remote user selected from the group consisting of a speech therapist, a speech and language pathologist, a clinician, a tutor, a parent, a caretaker, a physician, and an insurance agent.

8. The system of claim 1, wherein the assistive speech-related feedback further includes an audio recording corresponding to the processed signals and related data.

9. The system of claim 1, the controller unit comprising a high-speed field programmable gate array.

14. The system of claim 1, wherein the sensor control unit is incorporated into a headset comprising a head support, a processor, a receiver configurable to receive sensor signals from the magnetic sensors and the speech characteristic sensor, and a wireless transmitter configurable to be in wireless communication with the control unit.

10. An assistive method of providing information useful in improving a user'"'"'s speech based on real-time tracking at least one of a position, orientation, and movement of a tongue of the user during speech comprising:

receiving magnetic sensor signals from magnetic sensors;

measuring, based on the magnetic sensor signals, a magnetic flux density at a location of a tracer unit, the tracer unit being located in a non-obstructive manner on the tongue of the user;

performing a tongue tracking signal processing algorithm, the algorithm including minimizing a cost function that correlates to the measured magnetic flux density and an estimated magnetic flux density at the location of the tracer unit, the location of the tracer unit being defined within a three-dimensinal, non-discrete space;

determining, based on the algorithm, one or more of a position, orientation, and movement of the tracer unit in real time within the three-dimensinal, non-discrete space;

receiving speech characteristic signals from speech characteristic sensors;

combining the one or more determined position, determined orientation, and determined movement of the tracer unit with the speech characteristic sensor signals; and

generating, based on the combining, assistive speech-related feedback data, the assistive speech-related feedback data representing temporally and spatially continuous tracking of the user'"'"'s tongue movement in real time.

11. The method according to claim 10, wherein the plurality of speech characteristic sensors are configured to sense two or more from the group consisting of a sound produced by the user, an air flow produced by the user, a lip movement produced by the user, a lip gesture produced by the user, a physical contact between the tongue and a palate of the user, a muscle movement produced by the user, and an electrical activity of a brain of the user.

12. The method according to claim 10 further comprising displaying a representation of the assistive speech-related feedback data, wherein the representation is one or both of vibrotactile biofeedback and an interactive game.

13. The method according to claim 10 further comprising:

cataloging assisitive speech-related feedback data into a library for use as a silent speech interface, wherein the assisitive speech-related feedback data correspond to at least one of a word and a phrase.

15. The method according to claim 10, wherein measuring the magnetic flux density comprises amplification, filtering, digitization, magnetic localization, smoothing, and pattern recognition of the magnetic sensor signals.

16. The method according to claim 12, wherein the assisitaive speech-related feedback data is generated according to the steps of showing the raw magnetic sensor outputs;

generating a fused combination of at least one sensor output comprising output from at least one magnetic and acoustic sensor;

calculating a vector sum representation of the magnetic sensor outputs;

producing a change of color based on the pitch or amplitude of the speech volume based on output from the acoustic sensors; and

showing the tongue trajectory in at least one of 1D, 2D, or 3D, an interactive game, and a goal-oriented game.