Speech recognition and control system, program product, and related methods

• US 20070288242A1
• Filed: 06/12/2006
• Published: 12/13/2007
• Est. Priority Date: 06/12/2006
• Status: Active Grant

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Spengler, Richard P., Russo, Jon C., Barnett, Gregory W., Armbruster, Kermit L.
• Granted Patent: US 7,774,202 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 21/0208 Noise filtering

Abstract

A speech activated control system for controlling aerial vehicle components, program product, and associated methods are provided. The system can include a host processor adapted to develop speech recognition models and to provide speech command recognition. The host processor can be positioned in communication with a database for storing and retrieving speech recognition models. The system can include an avionic computer in communication with the host processor and adapted to provide command function management, a display and control processor in communication with the avionic computer adapted to provide a user interface between a user and the avionic computer, and a data interface positioned in communication with the avionic computer and the host processor provided to divorce speech command recognition functionality from vehicle or aircraft-related speech-command functionality. The system can also include speech actuated command program product at least partially stored in the memory of the host processor and adapted to provide the speech recognition model training and speech recognition model recognition functionality.

Claims

1. A speech activated control system for controlling aerial vehicle components, comprising:

a host processor having memory and positioned in communication with a database for storing speech recognition models; and

speech actuated command program product at least partially stored in the memory of the host processor and including instructions that when executed by the host processor perform the operations of;

forming a digitized user-speech template,

thresholding bins of equalized portions of the digitized user-speech template representing a command annunciation to remove noise from within and around speech formants of the sampled speech data,

developing a set of feature vectors representing energy of a frequency content of the digitized user-speech template to thereby determine a unique pattern identifying the command annunciation,

applying a speech recognition engine to the set of feature vectors to form at least one speech recognition model associated with the command annunciation,

associating an index with the at least one speech recognition model associated with the command annunciation, and

storing the at least one speech recognition model and the associated index.

2. The system as defined in claim 1, wherein the speech actuated command program product further includes instructions to perform the operations of:

receiving the command associated annunciation real-time in-flight indicating a request for recognition;

returning the index associated with a stored model determined to match the command annunciation;

returning a confidence score indicating likelihood the match is correct; and

executing an assigned function or forming an assigned state when the confidence score is above a preselected or selected threshold value.

3. The system as defined in claim 1, further comprising:

an avionic computer in communication with the host processor and having memory and a portion of the speech actuated command program product defining a command function manager stored in the memory; and

a display and control processor in communication with the avionic computer and having memory and a portion of the speech actuated command program product defining a user interface stored in the memory.

4. The system as defined in claim 3, wherein the speech actuated command program product further includes instructions to perform the operations of:

receiving a command function or system state;

receiving the command associated annunciation real-time in-flight; and

associating the command function or system state with a representation of the command annunciation.

5. The system as defined in claim 1, wherein the speech actuated command program product further includes instructions to perform the operations of:

recording during aircraft operation a series of switch engagements to form a speech-activated switch macro describing a command function or system state; and

receiving the associated command annunciation real-time in-flight; and

associating the speech-activated switch macro with a representation of the command annunciation.

6. The system as defined in claim 1, wherein the speech actuated command program product further includes instructions to perform the operations of:

performing a dynamic range utilization analysis on the digitized user-speech template to determine if the command annunciation is below a preselected minimum threshold level;

performing a clipping analysis on the digitized user-speech template to determine if the command annunciation has exceeded a preselected threshold value;

performing a cropping analysis on the digitized user-speech template to determine if the command annunciation is missing portions of energy;

performing a misalignment analysis on the digitized user-speech template to determine if the command annunciation was successfully aligned during noise removal; and

returning an integrity score grading the quality of the command annunciation to determine if the command annunciation is acceptable for training or recognition.

7. The system as defined in claim 1, wherein the speech actuated command program product further includes instructions to perform the operation of aligning the command annunciation in the digitized user-speech template to enhance analysis of the command annunciation, the alignment including the operations of:

determining a geometric mean of each of a plurality of bins for each of a plurality of time slices to thereby form an alignment vector;

auto convolving the alignment vector to form a convolution of the alignment vector with itself;

determining a mean position of peaks of the convolution to identify the center of the speech; and

cyclically shifting the sampled data to center the speech in the observation window.

8. The system as defined in claim 1, wherein the operation of thresholding bins of equalized portions of the digitized user-speech template includes the operations of:

estimating noise power for each of a plurality of time slices on either side of the command annunciation near and outside boundaries of the command annunciation for each of a plurality of frequency ranges to thereby determine a background noise contour for noise within the template;

equalizing energy values of each bin across each of the plurality of frequency ranges responsive to the estimated noise power; and

thresholding each bin by a predetermined parameterized value to remove the noise from the speech.

10. The system as defined in claim 3, further comprising:

a data interface in communication with the avionic computer and the host processor positioned to divorce speech command recognition functionality from aircraft-related speech-command functionality; and

a mobile storage device interface to allow an operator to retrieve trained speech models, associated index numbers, and associated functions or vehicle system states from the database and to load previously trained speech models, associated index numbers, and associated functions or vehicle system states to the database.

9. The system as defined in claim 9, wherein the operation of the thresholding further includes performing the operations of:

comparing each of the plurality of bins for each of the plurality of time slices to the parameterized value; and

setting each bin having a value below the parameterized value to approximately zero to thereby remove the noise from the sampled data from within and around speech formants.

11. A method to provide speech-activated control of aerial vehicle components, the method comprising the steps of:

(a) sampling a speech signal representing speech to define sampled data;

(b) performing an integrity check on the sampled data to identify when the speech is below a preselected standard;

(c) aligning the sampled data in an observation window to enhance analysis of the speech;

(d) thresholding bins of equalized portions of the sampled data to remove noise from within and around speech formants;

(e) developing a set of feature vectors representing energy of a frequency content of the sampled data to thereby determine a unique pattern; and

(f) applying a speech recognition engine to the set of feature vectors to perform one of the following functions;

forming at least one speech recognition model associated with the speech signal, and matching the speech signal to a speech recognition model.

12. The method as defined in claim 11, wherein step (b) includes the steps of:

performing a dynamic range utilization analysis on the sampled data to determine if the speech is at or below a preselected minimum threshold level indicating the speech was too quiet;

performing a clipping analysis on the sampled data to determine if the speech met or has exceeded a preselected maximum value indicating that clipping has occurred;

providing an integrity score indicating quality of the speech; and

requesting repeat of the speech responsive to the integrity score when the integrity score is below a preselected or selected value to thereby enhance speech-model development when in a training mode or to thereby enhance recognition accuracy when in a recognition mode.

13. The method as defined in claim 11, wherein step (c) includes the steps of:

dividing the sampled data into a plurality of time slices;

performing a short time fourier transformation on each time slice to form fourier transformed data defining a spectrograph;

converting spectrograph amplitude values of each time slice to decibels;

thresholding the amplitude values by a centering threshold to normalize the energy values within each time slice;

determining a geometric mean of each of a plurality of bins for each time slice to thereby form an alignment vector;

auto convolving the alignment vector to form a convolution of the alignment vector with itself;

determining a mean position of peaks of the convolution to identify the center of the speech; and

cyclically shifting the sampled data to center the speech in the observation window.

14. The method as defined in claim 11, wherein step (d) includes the step of estimating noise power for each of a plurality of time slices on either side of the speech near and outside boundaries of the speech for each of a plurality of frequency ranges to thereby determine a background noise contour for noise within the observation window.

15. The method as defined in claim 14, wherein step (d) further includes the steps of:

equalizing the energy values of each bin across each of the plurality of frequency ranges responsive to the estimated noise power; and

thresholding each bin by a predetermined parameterized value to remove the noise from the speech.

16. The method as defined in claim 15, wherein the step of the thresholding includes the steps of:

comparing each of the plurality of bins for each of the plurality of time slices to the parameterized value; and

setting each bin having a value below the parameterized value to approximately zero to thereby remove from the sampled data the noise within and around speech formants.

17. The method as defined in claim 15, further comprising the step of:

performing a cropping analysis on the sampled data to determine if the speech is potentially missing portions of energy.

18. The method as defined in claim 15, further comprising the step of:

performing a misalignment analysis on the sampled data to determine if the speech was successfully aligned during noise removal.

19. The method as defined in claim 11,

wherein when performing speech training the method further includes the steps of assigning an index to the at least one speech recognition model associated with the speech and storing the at least one speech recognition model and the assigned index;

and

wherein when performing speech recognition the method further includes the steps of returning the index associated with a stored speech recognition model determined to match sampled data received during speech recognition, returning the confidence score indicating likelihood the match is correct, and executing an assigned function or forming an assigned state when the confidence score is above a preselected or selected threshold value.

20. The method as defined in claim 11, wherein step (f) includes relaxing constraints within a Hidden Markov model when performing speech training.

21. The method as defined in claim 11,

wherein step (e) includes developing a first set of feature vectors for Hidden Markov Model (HMM) modeling and a second set of feature vectors for Neural Network (NN) modeling;

and

wherein step (f) includes applying a HMM speech recognition engine to the first set of feature vectors and a NN speech recognition engine to the second set of feature vectors; and

wherein the method further comprises executing a voting scheme between a plurality of speech recognition engines including at least one of the following;

the HMM speech recognition engine and the NN speech recognition engine.

22. The method as defined in claim 19,

wherein the speech recognition models are stored in one of a plurality of vocabulary templates;

wherein the vocabulary template is associated with one of a plurality of predetermined operational profiles; and

wherein the parameterized value is preselected responsive to analysis of noise conditions of a selected one of the plurality of predetermined operational profiles.

23. The method as defined in claim 11, further comprising the step of iteratively forming a plurality of speech recognition models for each of a separate plurality of operational profiles having differing environmental characteristics including a substantially different background noise contour

24. The method as defined in claim 11, wherein the step of developing a set of feature vectors includes developing one or more sets of feature vectors, and wherein the method further comprises:

applying each of a plurality of speech recognition engines separately to at least one of the one or more sets of feature vectors; and

executing a voting scheme between the plurality of speech recognition engines.

25. A method to provide speech-activated control of aerial vehicle components, the method comprising the steps of:

(a) thresholding bins of equalized portions of sampled speech data representing a command annunciation to remove noise from within and around speech formants of the sampled speech data;

(b) developing a set of feature vectors representing energy of a frequency content of the sampled data to thereby determine a unique pattern identifying the command annunciation;

(c) applying a speech recognition engine to the set of feature vectors to thereby form at least one speech recognition model; and

(d) associating an index with the at least one speech recognition model associated with the command annunciation; and

(e) storing the at least one speech recognition model and the assigned index.

26. The method as defined in claim 25, further comprising the steps of:

receiving a command function or system state;

receiving the command associated annunciation real-time in-flight; and

associating the command function or system state with a representation of the command annunciation.

27. The method as defined in claim 25, further comprising the steps of:

recording during aircraft operation a series of switch engagements to form a speech-activated switch macro describing a command function or system state;

receiving the associated command annunciation real-time in-flight; and

associating the speech-activated switch macro with a representation of the command annunciation.

28. The method as defined in claim 25, wherein the sampled data is a first sampled data processed to form a stored speech model during a speech recognition model training event, and wherein the method further comprises the steps of:

returning the index associated with the stored speech model determined to match a second sampled data;

returning a confidence score indicating likelihood the match is correct; and

executing an assigned function or forming an assigned state when the confidence score is above a preselected or selected threshold value.

29. The method as defined in claim 25, wherein the sampled data is a first sampled data processed to form a stored speech model during a speech recognition model training event, and wherein the method further comprises the steps of:

performing a dynamic range utilization analysis on the second sampled data to determine if the speech is below a preselected minimum threshold level indicating the speech was too quiet;

performing a clipping analysis on the second sampled data to determine if the speech has exceeded a preselected maximum value indicating that clipping has occurred; and

performing a cropping analysis on the second sampled data to determine if the speech is potentially missing portions of energy.