Signal injection coupling into the human vocal tract for robust audible and inaudible voice recognition

• US 7,082,395 B2
• Filed: 10/03/2002
• Issued: 07/25/2006
• Est. Priority Date: 07/06/1999
• Status: Expired due to Term

• Current Assignee: Carol A. Tosaya, John W. Sliwa Jr
• Original Assignee: Carol A. Tosaya, John W. Sliwa Jr
• Inventors: Tosaya, Carol A., Sliwa, Jr., John W.
• Primary Examiner(s): ABEBE, DANIEL DEMELASH
• Application Number:
  US10/265,468
• Publication Number:
  US 20030061050A1
• Time in Patent Office: 1,391 Days
• Field of Search: 704/200, 704/231, 704/246, 704/261
• US Class Current: 704/246
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 19/08 Determination or coding of the excitation function; Determination or coding of the long-term prediction parameters

Abstract

A means and method are provided for enhancing or replacing the natural excitation of the human vocal tract by artificial excitation means, wherein the artificially created acoustics present additional spectral, temporal, or phase data useful for (1) enhancing the machine recognition robustness of audible speech or (2) enabling more robust machine-recognition of relatively inaudible mouthed or whispered speech. The artificial excitation (a) may be arranged to be audible or inaudible, (b) may be designed to be non-interfering with another user'"'"'s similar means, (c) may be used in one or both of a vocal content-enhancement mode or a complimentary vocal tract-probing mode, and/or (d) may be used for the recognition of audible or inaudible continuous speech or isolated spoken commands.

Claims

1. A speech recognition system for processing sounds emanating from a living body'"'"'s vocal tract, said sounds including sounds excited by at least one artificial exciter coupled, either directly or indirectly, into said vocal tract to introduce artificial excitations, said at least one artificial excitation modified or modulated by said vocal tract and emanating therefrom, said speech recognition system including:

means for representation, modeling or classification or both, and searching of artificially excited speech signals or signal components;

means for representation, modeling or classification or both, and searching of naturally excited speech signals or signal components;

at least one of said searching means having access to at least one of an acoustic model, lexical model or language model; and

at least one training means.

2. The speech recognition system of claim 1 wherein said sounds are one of continuous speech, command-style speech, or an utterance.

3. The speech recognition system of claim 1 further including means for directing at least a naturally excited second modified or modulated signal to a speech representation means which samples said naturally excited signal to produce a second sequence of speech representation vectors, representative at least in part of said natural speech signal.

4. The speech recognition system of claim 3 further including second means for modeling or classifying said second sequence of vectors representative, at least in part, of said naturally excited speech signal.

5. The speech recognition system of claim 4 further including second means for subjecting said modeled or classified natural speech vectors to a search in a second search module said search module having access to at least one of an acoustic model, a lexical model or a language model.

6. The speech recognition system of claim 1 wherein training means are provided for both naturally excited signals and artificially excited signals, said means being one of independent or the same means, said signals being one of separate or combined.

7. The speech recognition system of claim 1 wherein artificial excitations are adapted to an individual user.

8. The speech recognition system of claim 7 wherein said adapted excitations are portable across at least one of multiple recognition systems, computers, networks, and speech-conversant devices.

9. The speech recognition system of claim 1 further including a separator, deconvolution, or subtraction means to discern naturally excited sounds or sound components from artificially excited sounds or sound components.

10. The speech recognition system of claim 1 wherein said artificially excited sounds permit inaudible speaking or command-giving to a computer, computer-coupled device, or computer-containing device.

11. The speech recognition system of claim 1 adapted for processing sounds that are both naturally excited and artificially excited, said sounds, or signal representations thereof, being substantially processed as one of separate or separated signals or signal-components or as a combined signal.

12. The speech recognition system of claim 11 wherein said artificially excited sounds permit improved recognition-accuracy or improved recognition-speed of natural speech, sounds or utterances.

13. The speech recognition system of claim 11 wherein said artificially excited and naturally excited speech sounds emanating from said tract temporally overlap at least part of the time.

14. The speech recognition system of claim 11 wherein said artificially excited and naturally excited speech sounds emanating from said tract are not identical in spectral content at least part of the time.

15. The speech recognition system of claim 11 wherein said artificially excited signal, before or after tract modification or modulation, includes at least one of the following aspects:

(a) said artificially excited signal contains a harmonic or subharmonic of a natural formant, (b) said artificially excited signal contains phase information which is utilized in the recognizer, (c) said artificially excited signal is broadband in nature, (d) said artificially excited signal is selected or set as a function of any natural signal parameter, (e) said artificially excited signal contains tones or frequency components which interact with each other as a function of a vocal tract parameter, (f) said artificially excited signal contains at least one tone or frequency component which is modulated or modified by any portion of the vocal tract anatomy, (g) said artificially excited signal is generally inaudible to the unaided ear of a separate listener, and (h) said artificially excited signal is swept in frequency.

16. The speech recognition system of claim 1 wherein said vocal tract includes at least one element selected from the group consisting of vocal chords, larynx, laryngeal valve, the glottal opening, the glottis, the arytenoids, the pharynx, the esophagus, the tongue, the pharyngeal walls, the velum, the hard palate, the alveolar ridge, the lips, teeth, gums, cheeks or any nasal cavity, at least said one element modifying or modulating said artificial excitation as the speaker articulates speech either audibly or inaudibly.

17. The speech recognition system of claim 1 further including means for directing at least a first modified or modulated artificially excited speech signal to a first speech representation means which samples at least said first signal to produce a first sequence of speech representation vectors, representative at least in part, of said artificially excited signal.

18. The speech recognition system of claim 17 wherein both the artificially excited signal and the naturally excited signal are represented by a single set of representation vectors.

19. The speech recognition system of claim 17 further including means for modeling or classifying said first sequence of vectors.

20. The speech recognition system of claim 19 further including means for subjecting said modeled or classified vectors to a search in a search module, said search module having access to at least one of an acoustic model, a lexical model, or a language model.

21. The speech recognition system of claim 20 wherein two search modules operate, one arranged to process naturally excited signals and the other to process artificially excited signals, said system utilizing the results of both modules to decide what speech took place or what words were articulated.

22. The speech recognition system of claim 1 further including a training data means capable of supporting training using at least the artificially excited speech signals.

23. The speech recognition system of claim 1 wherein said at least one artificial excitation is chosen based on an optimized correlation between it and known words or utterances made available during training.

24. A method of performing speech recognition on silently-mouthed, silently-articulated or whispered speech or unterances from a living body'"'"'s vocal tract, comprising:

providing a source of artificial acoustic excitation;

coupling said artificial acoustic excitation, directly or indirectly, into said vocal tract of a speaker;

allowing said artificial acoustic excitation to be modified or modulated by said speaker'"'"'s mouthing, articulation or whispering action by a state of at least a portion of said speaker'"'"'s vocal tract;

performing speech-recognition processing on at least a portion of or component of said modified acoustic excitation to contribute to the identification of said speech or utterance; and

choosing said at least one artificial excitation based on an optimized correlation between it and known words or utterances made available during training,

wherein said speech-recognition processing includes processing said modified acoustic excitation through representation, modeling or classification or both, and searching to produce identified words or utterances.

25. The method of claim 24 wherein said speech or utterance is silently mouthed and any modified acoustic excitation is primarily sourced from said artificial excitation.

26. The method of claim 24 wherein said speech or utterance is whispered and the modified acoustic excitation is sourced both by said artificial excitation as well as by, at least in part, natural aspiration excitation.

27. The method of claim 24 wherein said speech or utterance is one of continuous speech, discrete speech, or command-style speech.

28. The method of claim 24 wherein said coupling is acoustic coupling of an infrasonic, sonic, or ultrasonic transducing device, directly or indirectly, to at least one portion of said vocal tract.

29. The method of claim 24 wherein said coupling is one of tissue coupling or air-coupling.

30. The method of claim 24 wherein said recognized speech or utterance is at least one of recorded, converted to text, transmitted using a telephony link, transmitted to a remote recipient or computer, or used in carrying out a further task.

31. The method of claim 24 wherein said artificial excitation is itself inaudible at least to an external observer.

32. A method of enhancing the accuracy or speed of speech recognition of the speech or utterances emanating from a living body'"'"'s vocal tract, comprising:

coupling artificial acoustic excitation, directly or indirectly, into said vocal tract of a speaker;

allowing said speaker to audibly speak;

at least during portions of said audible speech, allowing said artificial acoustic excitation to be modified or modulated by said speaker'"'"'s mouthing, articulation or whispering action by a state of at least a portion of said speaker'"'"'s vocal tract to provide an artificially excited output of said speaker; and

performing speech-recognition processing using at least portions of both naturally excited and artificially excited outputs of said speaker, to thereby provide enhanced accuracy or speed of said speech or utterance recognition;

wherein said speech-recognition processing includes processing said modified acoustic excitation through representation, modeling or classification or both, and searching to produce identified words or utterances and wherein the acoustic output of said vocal tract containing both types of acoustic outputs is speech-recoanition processed, at least in part, as separate natural and artificial signals.

33. The method of claim 32 wherein the acoustic output of said vocal tract containing both types of acoustic outputs is speech-recognition processed, at least in part, as a combined signal.

34. The method of claim 32 wherein said speech is one of continuous speech, discrete speech, or command-style speech or utterance.

35. The method of claim 32 wherein said artificial acoustic excitation is temporally overlaid or interleaved, at least in part, with a natural tract excitation.

36. The method of claim 32 wherein said artificial acoustic excitation is applied using feedback information relating to a state of a natural excitation or of an articulatory position or state.

37. The method of claim 32 wherein said artificially excited acoustic output is recognition-processed when naturally-produced acoustic output is determined insufficient to alone identify said speech or utterance with a desired accuracy or speed.

38. The method of claim 32 wherein an artificial acoustic excitation is triggered by a state of natural excitation or a state or an articulator or vocal tract element.

39. The method of claim 32 further comprising:

choosing said at least one artificial excitation based on an optimized correlation between it and known words or utterances made available during training.

40. A system for improving the accuracy, robustness, or speed of the speech-recognition of utterances made by a speaker comprising:

a subject speaker emanating at least one naturally audible utterance from his/her vocal tract;

an artificial acoustic excitation means for speech-recognition enhancement of natural and audible speech and utterances coupled, either directly or indirectly, into said speaker'"'"'s vocal tract, without interfering with or modifying breathing or aspiration, for use at least in healthy vocal tracts;

the excitation means operated by said system to excite at least one additional of (a) an artificial audible emanation from the speakers modulating tract and (b) an artificial inaudible emanation from the speakers modulating tract;

the system utilizing both the natural audible signal and at least one of the additional artificial audible or artificial inaudible signals to perform speech recognition;

the system having representation means and at least one of modeling means and classification means, a searching means, a training means and a directly or indirectly tract-coupled exciter; and

the system being teachable by the speaker.

41. The system of claim 40 wherein the utterance is one of (a) words, (b) commands, (c) whispered words or commands, (d) mouthed words or commands, (e) a name or password, or (f) a response to a prompt given by a security device.

42. The system of claim 40 wherein the artificial excitation means includes at least one acoustic transducer either directly or indirectly coupled into said speaker'"'"'s vocal tract.

43. The system of claim 40 wherein the teaching step includes the system listening to both audible natural signals and at least one of artificial audible or artificial inaudible signals, either simultaneously or in sequence as both are modulated by the speaker.

44. The system of claim 40 wherein an artificial excitation is encoded or chosen such that it is identifiable as belonging to an assigned speaker.

45. The system of claim 44 wherein such encoding or choosing allows a speech recognition system to help differentiate between two or more speakers who both speak simultaneously or in sequence.

46. The system of claim 40 as used to improve speech recognition accuracy, robustness, or speed in a noisy environment.

47. The system of claim 40 wherein at least one acoustic excitation is optimized for a particular speaker or for a particular environment.

48. The system of claim 40 wherein one or more audible and/or inaudible microphones or pickups is utilized to acoustically receive said emanating signals.

49. The system of claim 40 wherein said speech recognition is performed locally or remotely and the resulting recognized speech is at least one of:

(a) transmitted to a remote location, (b) recorded, (c) used to instruct a machine or computer or other electronic device, (d) used as part of a security system, (e) converted to text or to a synthetic voice, (f) translated into another language, (g) recognized using stored teaching data or personal files, and (h) representing spoken words, whispered words, commands, or mouthed utterances.

50. The system of claim 40 wherein after said teaching utilizing natural audible and at least one inaudible signal is completed, the trained system is then used to recognize inaudible speech or whispering.

51. The system of claim 40 as used to identify the voice of an unknown speaker for security reasons.

52. The system of claim 40 wherein the exciter is excited by at least one of:

(a) broadband excitation, (b) narrow band excitation(s) or a harmonic frequency of a natural formant, (c) at least one tone, (d) excitations which are delivered as a function of the success of ongoing natural-speech recognition, and (e) excitations which are feedback-optimized for each speaker.

53. The system of claim 40 wherein during recognition processing, both natural speech and artificial speech components are processed at at least some point.

54. A system for providing speech or utterance-recognition processing of improved accuracy or robustness, said speech or utterances comprising one or more of audible or inaudible (a) speech, (b) commands, (c) whispers, (d) mouthings or sounds, said system comprising:

a speech recognition system capable of recognizing at least one natural or artificially induced audible or inaudible acoustic emanation emanating from a speaker'"'"'s vocal tract, said speech recognition system including an artificial exciter for speech-recognition enhancement of natural audible and utterances that does not interfere with or modify breathing or aspiration, for use at least in healthy vocal tracts, said speech recognition system further including representation means and at least one of modeling means and classification means, a searching means, a training means and a directly or indirectly tract-coupled exciter, and

a vocal tract acoustic-probing means which measures at least one of (a) an acoustic impedance or admittance, (b) an acoustic resonance or harmonic, (c) a vocal tract spectral fingerprint,

wherein a combination of the first speech recognition process and at least one second acoustic measurement process provide complimentary information allowing for a more accurate or robust overall speech recognition process.

55. The system of claim 54 wherein the acoustic probing means is also capable of artificially exciting audible or inaudible vocal tract emanations at least some of which are modulated by said tract.

56. The arrangement of claim 54 wherein during recognition processing, both natural speech and artificial speech components are processed at at least some point.

57. An arrangement for improving the accuracy, robustness, or speed of the speech-recognition of natural audible speech and utterances made by a speaker, said arrangement comprising:

(a) a teachable speech recognition system comprising at least a speech sampling means and a search means, said system capable of detecting modulations or modulation activity of said speaker'"'"'s vocal tract and relating said modulations to learned utterances; and

(b) an artificial acoustic excitation means and an associated artificial acoustic reception means, represented collectively by one or more body-contacting, skin contacting or bone contacting transducers for speech-recognition enhancement of natural audible speech and utterances that does not interfere with or modify breathing or aspiration, for use at least in healthy vocal tracts;

wherein;

(1) the artificial acoustic excitation is transmitted directly or indirectly into the speakers vocal tract and is modulated by said utterances;

(2) said modulated artificial excitations being detected by at least one said contacting transducer reception means; and

(3) said speech recognition system utilizing at least said artificial modulated excitations and representation means, at least one of modeling means and classification means, a training means and a directly or indirectly tract-coupled exciter for recognition processing.

58. The arrangement of claim 57 wherein during recognition processing, both natural speech and artificial speech components are processed at at least some point.

59. An arrangement for providing speech or utterance-recognition processing of improved accuracy or robustness, said speech or utterances comprising one or more of audible or inaudible (a) speech, (b) commands, (c) whispers, (d) mouthings or sounds, said arrangement comprising:

(a) a speech recognition system capable of recognizing at least one of natural or artificially induced audible or inaudible acoustic modulations of a speaker'"'"'s vocal tract, and

(b) at least one artificial acoustic emitter, exciter or transducer and at least one acoustic receiver or pickup, or a single acoustic device serving both emitter/exciter and receiver functions, the artificial emitter, exciter or transducer capable of speech-recognition enhancement of natural audible and utterances that do not interfere with or modify breathing or aspiration, for use at least in healthy vocal tracts,

wherein;

(1) at least one of said acoustic devices is skin or bone coupled,

(2) the emitter/exciter exciting artificial audible or inaudible acoustics which are modulated, at least in part, by the speaker'"'"'s vocal tract,

(3) the receiver detecting said modulated artificial acoustics and making them available in support of recognition-processing by the speech recognition system, and

(4) the system being taught to recognize using utterances which comprise, at least in part, said modulated artificial acoustics, the system further including representation means and at least one of modeling means and classification means, a searching means, a training means and a directly or indirectly tract-coupled exciter.

60. The arrangement of claim 59 wherein at least one acoustic emitter and at least one acoustic receiver are each at least one of skin or bone coupled.

61. The arrangement of claim 60 wherein at least one receiver receives acoustic emissions originating from at least one emitter after said emitted artificial acoustics have been modulated by the speaker'"'"'s modulating vocal tract.

62. The arrangement of claim 61 wherein at least one emitter and at least one receiver are at different skin or bone-contacting locations on the body.

63. The arrangement of claim 59 wherein said recognition portion of the system is, at least in part, remote from the speaker.

64. The arrangement of claim 59 wherein during recognition processing, both natural speech and artificial speech components are processed at at least some point.