Wireless real-time tongue tracking for speech impairment diagnosis, speech therapy with audiovisual biofeedback, and silent speech interfaces
First Claim
1. A system for real-time tracking of at least one of a position, orientation, and movement of a tongue of a user during speech comprising:
- a tracer unit;
magnetic sensors;
a speech characteristic sensor;
a sensor control unit;
a computing platform; and
a feedback system;
wherein the tracer unit is adapted to be non-obstructively affixed at a location on the tongue of the user;
wherein the magnetic sensors are configurable to measure a magnetic flux density at the location of the tracer unit, the magnetic sensors in non-obstructive proximity to the tracer unit and mouth of the user;
wherein the speech characteristic sensor configurable to detect a speech characteristic selected from the group consisting of sound produced by the user, an air flow produced by the user, a lip movement/gesture produced by the user, a physical contact between the tongue and a palate of the user, an articulator muscle movement produced by the user, and an electrical activity of a brain of the user;
wherein the sensor control unit is configurable to;
receive a magnetic sensor signal from a magnetic sensors;
receive a speech characteristic sensor signal from the speech characteristic sensor;
synchronize the magnetic sensor signal with the speech characteristic sensor signal; and
transmit the synchronized sensor signal to the computing platform;
wherein the computing platform isconfigurable to;
perform a tongue tracking signal processing algorithm, the algorithm including minimizing a cost function that correlates to the measured magnetic flux density and an estimated magnetic flux density at the location of the tracer unit, the location of the tracer unit being defined within a three-dimensional, non-discrete space;
determine, based on the algorithm, one or more of a position, orientation, and movement of the tracer unit in real time within the three-dimensinal, non-discrete space;
generate a combination of the one or more determined position, determined orientation, and determined movement of the tracer unit with the speech characteristic sensor signal; and
transmit the combination to the feedback system; and
wherein the feedback system is configurable to provide assistive speech-related feedback to the user in real time, based on the combination, wherein the assistive speech-related feedback is selected from the group consisting of a representation of temporally and spatially continuous tracking of the user'"'"'s tongue movement during speech of a target phoneme within the three-dimensional non-discrete space, an interactive game to quantify and monitor the user'"'"'s progress, and vibrotactile biofeedback comparing speech of the user to correct speech of a target phoneme.
2 Assignments
0 Petitions
Accused Products
Abstract
A real-time wireless system for recording natural tongue movements in the 3D oral space. By attaching a small magnetic tracer to the tongue, either temporarily or semi-permanently, and placing an array of magnetic sensors around the mouth, the tracer can be localized with sub-millimeter precision. The system can also combine the tracer localization with acoustic, video, and flow data via additional sensors to form a comprehensive audiovisual biofeedback mechanism for diagnosing speech impairments and improving speech therapy. Additionally, the system can record tongue trajectories and create an indexed library of such traces. The indexed library can be used as a tongue tracking silent speech interface. The library can synthesize words, phrases, or execute commands tied to the individual patterns of magnetic field variations or tongue trajectories.
-
Citations
16 Claims
-
1. A system for real-time tracking of at least one of a position, orientation, and movement of a tongue of a user during speech comprising:
-
a tracer unit; magnetic sensors; a speech characteristic sensor; a sensor control unit; a computing platform; and a feedback system; wherein the tracer unit is adapted to be non-obstructively affixed at a location on the tongue of the user; wherein the magnetic sensors are configurable to measure a magnetic flux density at the location of the tracer unit, the magnetic sensors in non-obstructive proximity to the tracer unit and mouth of the user; wherein the speech characteristic sensor configurable to detect a speech characteristic selected from the group consisting of sound produced by the user, an air flow produced by the user, a lip movement/gesture produced by the user, a physical contact between the tongue and a palate of the user, an articulator muscle movement produced by the user, and an electrical activity of a brain of the user; wherein the sensor control unit is configurable to; receive a magnetic sensor signal from a magnetic sensors; receive a speech characteristic sensor signal from the speech characteristic sensor; synchronize the magnetic sensor signal with the speech characteristic sensor signal; and transmit the synchronized sensor signal to the computing platform; wherein the computing platform is configurable to; perform a tongue tracking signal processing algorithm, the algorithm including minimizing a cost function that correlates to the measured magnetic flux density and an estimated magnetic flux density at the location of the tracer unit, the location of the tracer unit being defined within a three-dimensional, non-discrete space; determine, based on the algorithm, one or more of a position, orientation, and movement of the tracer unit in real time within the three-dimensinal, non-discrete space; generate a combination of the one or more determined position, determined orientation, and determined movement of the tracer unit with the speech characteristic sensor signal; and transmit the combination to the feedback system; and wherein the feedback system is configurable to provide assistive speech-related feedback to the user in real time, based on the combination, wherein the assistive speech-related feedback is selected from the group consisting of a representation of temporally and spatially continuous tracking of the user'"'"'s tongue movement during speech of a target phoneme within the three-dimensional non-discrete space, an interactive game to quantify and monitor the user'"'"'s progress, and vibrotactile biofeedback comparing speech of the user to correct speech of a target phoneme. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 14)
-
-
10. An assistive method of providing information useful in improving a user'"'"'s speech based on real-time tracking at least one of a position, orientation, and movement of a tongue of the user during speech comprising:
-
receiving magnetic sensor signals from magnetic sensors; measuring, based on the magnetic sensor signals, a magnetic flux density at a location of a tracer unit, the tracer unit being located in a non-obstructive manner on the tongue of the user; performing a tongue tracking signal processing algorithm, the algorithm including minimizing a cost function that correlates to the measured magnetic flux density and an estimated magnetic flux density at the location of the tracer unit, the location of the tracer unit being defined within a three-dimensinal, non-discrete space; determining, based on the algorithm, one or more of a position, orientation, and movement of the tracer unit in real time within the three-dimensinal, non-discrete space; receiving speech characteristic signals from speech characteristic sensors; combining the one or more determined position, determined orientation, and determined movement of the tracer unit with the speech characteristic sensor signals; and generating, based on the combining, assistive speech-related feedback data, the assistive speech-related feedback data representing temporally and spatially continuous tracking of the user'"'"'s tongue movement in real time. - View Dependent Claims (11, 12, 13, 15, 16)
-
Specification