Applications of sub-audible speech recognition based upon electromyographic signals

• US 7,574,357 B1
• Filed: 06/24/2005
• Issued: 08/11/2009
• Est. Priority Date: 06/24/2005
• Status: Active Grant

• Current Assignee: Charles C. Jorgensen
• Original Assignee: The United States of America As Represented By The Secretary of Agriculture
• Inventors: Jorgensen, C. Charles, Betts, Bradley J.
• Primary Examiner(s): Hudspeth; David R
• Assistant Examiner(s): Rider; Justin W
• Application Number:
  US11/169,265
• Time in Patent Office: 1,509 Days
• Field of Search: 704/247, 704/201, 704/208, 704/231, 704/236
• US Class Current: 704/236
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

Method and system for generating electromyographic or sub-audible signals (“SAWPs”) and for transmitting and recognizing the SAWPs that represent the original words and/or phrases. The SAWPs may be generated in an environment that interferes excessively with normal speech or that requires stealth communications, and may be transmitted using encoded, enciphered or otherwise transformed signals that are less subject to signal distortion or degradation in the ambient environment.

Claims

1. A method for communicating information in an environment that interferes with audible speech, the method comprising:

choosing at least a first sub-audible word or phrase (SAWP) and a second SAWP to be first and second representational SAWPs (Rep-SAWPs), respectively, where each Rep-SAWP is chosen to provide optimal likelihood of recognition of the transmitted SAWP signal in an ambient environment,

receiving at least one SAWP, from among the first SAWP and the second SAWP, at a wireless signaling module that processes and transmits sub-audible speech signals over a signaling channel to at least one recipient module;

determining if the at least one SAWP is a word or phrase in a SAWP lexicon associated with the module; and

when the at least one SAWP is recognized as belonging to the SAWP lexicon;

(i) transmitting the least one SAWP representing the received SAWP over the signaling channel; and

(ii) receiving the at least one SAWP from the signaling channel at the recipient module and converting the received at least one SAWP to at least one of visually perceptible alphanumeric text, audibly perceptible speech, and a coded signal representing the SAWP.

2. The method of claim 1, further comprising including in said SAWP lexicon at least one of the following words and phrases:

“

evacuate,”

“

Mayday,”

“

mantrap,”

“

fire safe,”

“

stop,”

“

go,”

“

left,”

“

right,”

“

alpha,”

“

omega,”

“

zero,”

“

one,”

“

two,”

“

three,”

“

four,”

“

five,”

“

six,”

“

seven,”

“

eight,” and

“

nine.”

3. The method of claim 1, further comprising choosing at least one of said first and second Rep-SAWPs to be at least one of an encoded SAWP, an encrypted SAWP and a SAWP that is transformed to a format that resists signal degradation when transmitted in said ambient environment.

4. The method of claim 1, wherein said process of determining if said at least one SAWP is a word or phrase in a SAWP lexicon associated with said module comprises:

receiving R signal sequences associated with said at least one SAWP, with each sequence including an instance of a sub-audible speech pattern (SASP), numbered r=1, . . . , R with R≧

2, and each SASP including at least one word drawn from a selected database of Q words, numbered q=1, . . . , Q with Q≧

2;

estimating where each of the R SASPs begins in the sequences;

for each of the signal sequences, numbered r=1, . . . , R;

providing signal values of the received signal, number r, within a temporal window having a selected window width Δ

t(win);

transforming each of the R SASPs, using an Signal Processing Transform (SPT) operation that is expressed in terms of at least one transform parameter having a selected sequence of parameter values;

providing a matrix M with entries equal to the SPTs for the R SASPs, ordered according to the at least one transform parameter along a first matrix axis;

tessellating the matrix M into a sequence of exhaustive and mutually exclusive cells, with each cell containing a collection of contiguous matrix entries, where each cell is characterized according to at least one selected cell criterion;

providing, for each cell, a cell representative value (“

feature”

), depending upon at least one of the matrix entries within the cell;

organizing the features into a vector V with entry values v_k (k=1, . . . , K), where K is the number of features for the tessellated matrix M; and

analyzing the entry values v_k, using a neural net classifier, a selected neural net architecture, and a sequence of estimated weight coefficient values associated with at least one of the neural net classifier layers, where the neural net classifier provides a sequence of output values dependent upon the weight coefficient values and upon at least one of the entry values v_k;

providing a sequence of target output values A(q;

ref)_g, one for each word number q, for the neural net classifier output values; and

adjusting the weight coefficient values to provide a reference set of weight coefficient values for which the neural net classifier output values for the R SASPs and for each of the Q words agree with the target output values within a selected threshold error number ε

(thr;

1).

5. The method of claim 4, further comprising replacing at least one of said features by a normalized feature for at least one of said cells corresponding to said matrix M.

6. The method of claim 4, wherein said process of adjusting said weight coefficient values comprises:

providing a first sequence of estimated weight coefficients {w_1,k,h} and a second sequence of estimated weight coefficients {w_1,h,g} (k=1, . . . , K;

h=1, . . . , H;

g=1, . . . , G), for first and second layers, respectively, of said neural net classifier;

forming a first sum S1(q;

r)_h of products of said entry values v_k and corresponding weight coefficients w_1,k,h;

computing a first activation function A1{S1(q;

r)_h} applied to the sum S1(q;

r)_h as argument;

forming a second sum S2(q;

r)_g of products of the first activation function values A1{S1(q;

r)_h} and corresponding weight coefficients w_2,h,g;

computing a second activation function A2{S2(q;

r)_g} applied to the sum S2(q;

r)_g as argument;

adjusting the first and second weight coefficient sequences to obtain a first reference set {w_1,k,h(q;

ref)}_k and a second reference set {w_2,h,g(q;

ref)}_h of the weight coefficients, associated with said at least one word number q, for which, for each of said instances R, a difference between the activation function value A2{S2(q;

r)_g} for said instance r and said target output values A(q;

ref)_g, computed using the reference set of weight coefficients, has a magnitude that is no greater than said selected positive error threshold number ε

(thr;

     1).

7. The method of claim 6, further comprising:

receiving a new SASP containing an unknown word, referred to as a “

new”

word, number q′

;

estimating where the new SASP begins;

providing signal values of the received signal for the new SASP within each of said J temporal windows, numbered j=1, . . . , J with J>

2, that are shifted in time relative to each other by multiples of a selected displacement time Δ

t(displ);

for the signal values within each of the time-shifted windows, numbered j=1, . . . , J;

transforming each of the signal values of the new SASP, using said Signal Processing Transform operation;

providing a matrix M′

with entries equal to said SPTs, ordered according to said at least one transform parameter along a first matrix axis;

tessellating the matrix M′

into said sequence of exhaustive and mutually exclusive cells that correspond to said cells for said tessellated matrix M;

providing, for each cell in the matrix M′

, a cell representative value or “

feature”

, depending upon at least one of the matrix entries within the cell; and

organizing the features into a vector V′

with entry values v′

_k (k=1, . . . , K);

applying said neural net classifier and said reference set of said weight coefficients to compute said neural net classifier output values for each of the time-shifted sequences of the new SASP;

forming comparison differences between said neural net classifier target output values and said neural net classifier output values for each of the time-shifted sequences of the new SASP;

for each of said time shifted sequences of the new SASP, choosing at least one word, number q″

, in said database for which the magnitude of the comparison difference is no greater than a magnitude of the comparison difference for any other word in said database, and accumulating a weighted point each time the at least one word number q″

is chosen; and

when the at least one word number q″

has a higher number of weighted points than the number of weighted points for any other word in said database, interpreting this condition as indicating that the new SASP includes the word number q″

.

8. The method of claim 7, further comprising replacing at least one of said features for said matrix M′

by a normalized feature for at least one of said cells for said matrix M′

.

9. The method of claim 7, further comprising:

when at least two distinct words, number q1 and q2, in said database have substantially equal numbers of said weighted points and the substantially equal numbers of said weighted points of the at least two words, number q1 and q2, are substantially greater than said number of said weighted points of any other word in said database, interpreting this condition as indicating that said new word included in said new SASP cannot be unambiguously identified.

10. The method of claim 7, further comprising choosing said weighting for said weighted points from the group of weighting formats consisting of (1) substantially uniform weighting and (ii) a weighting that decreases monotonically as said magnitude of said comparison difference increases.

11. The method of claim 7, wherein said steps of applying said neural net classifier and forming said comparison differences comprise:

forming a sum S1′

(q′

;

j;

ref)_h over said index k of a product of said first reference set of weight coefficient w_1,k,h(q′

;

j;

ref) and said vector entries v′

_k, and computing an activation function value A1{S1′

(q′

;

j;

ref)_h} depending upon the sum S1′

(q′

;

j;

ref)_h};

forming a sum S2′

(q′

;

j;

ref)_g over said index h of a product of said second reference set of weight coefficient w_2,h,g(q′

;

j;

ref) and the activation function values A1{S1′

(q′

;

j;

ref)_h} and computing an activation function value A2{S2′

(q′

;

j;

ref)_g} depending upon the sum S2′

(q′

;

j;

ref)_g}; and

forming a comparison difference between the activation function value A2{S2′

(q′

;

j;

ref)_g} for the new SASP and said target output value A(q;

ref)_g for at least one of said words, number q, in said database.

12. The method of claim 4, further comprising determining said reference set of said weight coefficients to be independent of said word number q in said database.

13. The method of claim 4, further comprising determining said reference set of said weight coefficients so that at least one reference set weight coefficient for a first selected word number q1 in said database differs from a corresponding reference set weight coefficient for a second selected word number q2 in said database.

14. The method of claim 4, further comprising selecting said window width Δ

t(win) in a range 1-4 sec.

15. The method of claim 4, further comprising selecting each of said matrix cells to be rectangularly shaped.

16. The method of claim 15, further comprising selecting at least two of said matrix cells to have different sizes.

17. The method of claim 4, further comprising determining said matrix cells by at least one of a first order statistical coefficient and a second order statistical coefficient for said entries in that cell.

18. The method of claim 17, further comprising:

selecting said first order and said second order statistical coefficients to be a statistical mean μ and

a standard deviation σ

, respectively for a cell; and

including in at least one of said cells a collection of contiguous entry values u of said matrix for which a normalized difference |u−

μ

|/σ

is no greater than a selected positive number.

19. The method of claim 4, further comprising choosing said SPT operations from the group of SPT operations consisting of (i) a windowed short time interval Fourier Transform (STFT);

(ii) discrete wavelets (DWTs) and continuous wavelets (CWTs) using Daubechies 5 and 7 bases;

(iii) dual tree wavelets (DTWTs) with a near_sym_a 5,7 tap filter and a Q-shift 14,14 tap filter;

(iv) Hartley Transform;

(v) Linear Predictive Coding (LPC) coefficients;

(vi) a moving average of a selected number of said sample values with uniform weighting; and

(vii) a moving average of a selected number of said sample values with non-uniform weighting.

20. The method of claim 4, further comprising selecting said database to include at least one of the words “

stop”

, “

go”

, “

left”

, “

right”

, “

alpha”

, “

omega”

, “

one”

, “

two”

, “

three”

, “

four”

, “

five”

, “

six”

, “

seven”

, “

eight”

, “

nine” and

“

ten.”

21. The method of claim 4, further comprising selecting said error threshold number to lie in a range ε

(thr;

1)≦

0.01.

22. The method of claim 4, further comprising applying a back propagation of error method in said neural net classifier analysis of said features of said cells of said matrix M.

23. The method of claim 4, further comprising:

receiving a new SASP signal containing an unknown word, referred to as a “

new”

word, number q′

;

estimating where a new SASP begins in the new SASP signal;

providing signal values of the received new signal for the new SASP within each of said J temporal windows, numbered j=1, . . . , J with J≧

2, that are shifted in time relative to each other by selected multiples of a selected displacement time Δ

t(displ);

for the signal values within each of the time-shifted windows, numbered j=1, . . . , J;

transforming each of the signal values of the new SASP, using said Signal Processing Transform operation;

providing a matrix M′

with entries equal to said SPTs, ordered according to said at least one transform parameter along a first matrix axis;

tessellating the matrix M′

into said sequence of exhaustive and mutually exclusive cells that correspond to said cells for said tessellated matrix M;

providing, for each cell in the matrix M′

, a cell representative value or “

feature”

, depending upon at least one of the matrix entries within the cell; and

organizing the features into a vector V′

with entry values v′

_k (k=1, . . . , K);

applying said neural net classifier and said reference set of said weight coefficients to compute said neural net classifier output values for each of the time-shifted sequences of the new SASP;

forming said comparison differences between said neural net classifier target output values and said neural net classifier output values for each of the time-shifted sequences of the new SASP;

for each of said time shifted sequences of the new SASP, choosing at least one word, number q″

, in said database for which (i) the magnitude of the comparison difference is no greater than the comparison difference for any other word in said database and (ii) the magnitude of the comparison difference is no greater than a second selected positive error threshold ε

(thr;

2), and accumulating a weighted point each time the at least one word number q″

is chosen; and

when the at least one word number q″

has a higher number of weighted points than the number of weighted points for any other word in said database, interpreting this condition as indicating that the new SASP includes the word number q″

.

24. A system for communicating information in an environment that interferes with audible speech, the system comprising:

a receiver-evaluator for receiving at least one of a first sub-audible word or phrase (SAWP) and a second SAWP and for determining if the SAWP is a word or phrase in a SAWP lexicon associated with the evaluator; and

a transmitter for transmitting a SAWP representing at least one of the first and second received SAWPs over a signaling channel to at least one recipient, when the received SAWP is determined to be contained in the SAWP lexicon,

wherein the SAWPs associated with first and second received SAWPs are respective first and second representational SAWPs (Rep-SAWPs) that are chosen to provide optimal likelihood of recognition of the transmitted SAWPs in an ambient environment.

25. The system of claim 24, further comprising a SAWP receiver, associated with said at least one recipient, for receiving and converting said SAWP to at least one of visually perceptible alphanumeric text, audibly perceptible speech, and a coded signal representing said received SAWP.

26. The system of claim 24, wherein said SAWP lexicon includes at least one of the following words and phrases:

“

evacuate,”

“

Mayday,”

“

mantrap,”

“

fire safe,”

“

stop,”

“

go,”

“

left,”

“

right,”

“

alpha,”

“

omega,”

“

zero,”

“

one,”

“

two,”

“

three,”

“

four,”

“

five,”

“

six,”

“

seven,”

“

eight,” and

“

nine.”

27. The system of claim 24, wherein at least one of said first and second Rep-SAWPs is chosen to be at least one of an encoded SAWP, an encrypted SAWP and a SAWP that is transformed to a format that resists signal degradation when transmitted in said ambient environment.