Speech recognition system, speech recognition method, speech synthesis system, speech synthesis method, and program product

• US 20030171921A1
• Filed: 03/04/2003
• Published: 09/11/2003
• Est. Priority Date: 03/04/2002
• Status: Active Grant

• Current Assignee: NTT Docomo Incorporated (Nippon Telegraph and Telephone Corporation)
• Original Assignee: NTT Docomo Incorporated (Nippon Telegraph and Telephone Corporation)
• Inventors: Sugimura, Toshiaki, Hiraiwa, Akira, Manabe, Hiroyuki
• Granted Patent: US 7,369,991 B2
• Time in Patent Office: Days
• US Class Current: 704/232
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 2021/0135 Voice conversion or morphing

Abstract

The object of the present invention is to keep a high success rate in recognition with a low-volume of sound signal, without being affected by noise.

The speech recognition system comprises a sound signal processor 10 configured to acquire a sound signal, and to calculate a sound signal parameter based on the acquired sound signal; an electromyographic signal processor 13 configured to acquire potential changes on a surface of the object as an electromyographic signal, and to calculate an electromyographic signal parameter based on the acquired electromyographic signal; an image information processor 16 configured to acquire image information by taking an image of the object, and to calculate an image information parameter based on the acquired image information; a speech recognizer 20 configured to recognize a speech signal vocalized by the object, based on the sound signal parameter, the electromyographic signal parameter and the image information parameter; and a recognition result provider 21 configured to provide a result recognized by the speech recognizer 20.

Claims

1. A speech recognition system comprises;

a sound signal processor configured to acquire a sound signal from an object, and to calculate a sound signal parameter based on the acquired sound signal;

an electromyographic signal processor configured to acquire potential changes on a surface of the object as an electromyographic signal, and to calculate an electromyographic signal parameter based on the acquired electromyographic signal;

an image information processor configured to acquire image information by taking an image of the object, and to calculate an image information parameter based on the acquired image information;

a speech recognizer configured to recognize a speech signal vocalized by the object, based on the sound signal parameter, the electromyographic signal parameter and the image information parameter; and

a recognition result provider configured to provide a result recognized by the speech recognizer.

2. The speech recognition system according to claim 1, wherein the speech recognizer recognizes a speech signal based on each of the sound signal parameter, the electromyographic signal parameter and the image information parameter, compares each of the recognized speech signals, and recognizes the speech signal based on the compared result.

3. The speech recognition system according to claim 1, wherein the speech recognizer recognizes the speech signal using the sound signal parameter, the electromyographic signal parameter and the image information parameter simultaneously.

4. The speech recognition system according to claim 1, wherein

the speech recognizer comprises a hierarchical network in which a plurality of non-linear components comprising an input unit and an output unit are located from upstream to downstream hierarchically;

the output unit of the upstream non-linear component is connected to the input unit of the downstream non-linear component within adjacent non-linear components;

a weight value is assigned to the connection or a combination of the connections;

each of the non-linear components calculates data which is outputted from the output unit and determines the connection to which the calculated data is outputted, in accordance with data inputted to the input unit and the weight value assigned to the connection or the combinations,

the sound signal parameter, the electromyographic signal parameter and the image information parameter are inputted to the most upstream non-linear components in the hierarchical network as the inputted data,

the recognized speech signals are outputted from the output unit of the most downstream non-linear components in the hierarchical network as the outputted data, and

the speech recognizer recognizes the speech signal based on the outputted data.

5. The speech recognition system according to claim 4, wherein the speech recognizer comprises a learning function configured to change the weight assigned to the non-linear components by inputting sampling data which is transferred from downstream to upstream.

6. The speech recognition system according to claim 1, wherein

the sound signal processor comprises a microphone configured to acquire the sound signal from a sound source, the microphone configured to communicate with a communications device;

the electromyographic signal processor comprises electrodes configured to acquire the potential changes on a surface around the sound source as the electromyographic signal, the electrodes being installed on a surface of a communications device;

the image information processor comprises a camera configured to acquire the image information by taking an image of the motion of the sound source, the camera being installed on a terminal separated from the communications device; and

the communications device transmits and receives data from the terminal.

7. The speech recognition system according to claim 6, wherein

the terminal comprises a body on which the camera is installed, and a belt for fixing the body;

and

the recognition result provider is a display for displaying the result, the display being installed on the surface of the body.

8. The speech recognition system according to claim 1, the system further comprising a positioning device and a holding device;

wherein

the sound signal processor comprises a microphone configured to acquire the sound signal from a sound source;

the electromyographic signal processor comprises electrodes configured to acquire the potential changes on a surface around the sound source as the electromyographic signal;

the image information processor comprises a camera configured to acquire the image information by taking an image of the motion of the sound source;

the positioning device fixes the microphone and the electrodes adjacent to the sound source; and

the holding device holds the camera and the positioning device.

9. The speech recognition system according to claim 6, wherein the recognition result provider displays the result in a translucent display, the recognition result provider being installed in the holding device.

10. A speech synthesis system comprising

a speech recognizer configured to recognize a speech signal;

a sound signal acquirer configured to acquire a sound signal;

a first spectrum acquirer configured to acquire a spectrum of the acquired sound signal as a first spectrum;

a second spectrum generator configured to generate a reconfigured spectrum of the sound signal, based on the speech signal recognized by the speech recognizer, as a second spectrum;

a modified spectrum generator configured to generate a modified spectrum in accordance with the first spectrum and the second spectrum; and

an outputter configured to output a synthesized speech signal based-on the modified spectrum.

11. The speech synthesis system according to claim 10, wherein the outputter comprises a communicator configured to transmit the synthesized speech signal as data.

12. A speech recognition method comprises the steps of:

(A) acquiring a sound signal from an object, and calculating a sound signal parameter based on the acquired sound signal;

(B) acquiring potential changes on a surface of the object as an electromyographic signal, and calculating an electromyographic signal parameter based on the acquired electromyographic signal;

(C) acquiring image information by taking an image of the object, and calculating an image information parameter based on the acquired image information;

(D) recognizing a speech signal vocalized by the object, based on the sound signal parameter, the electromyographic signal parameter and the image information parameter; and

(E) providing a result recognized by the speech recognizer.

13. The speech synthesis method comprising the steps of:

(A) recognizing a speech signal;

(B) acquiring a sound signal;

(C) acquiring a spectrum of the acquired sound signal as a first spectrum;

(D) generating a reconfigured spectrum of the sound signal, based on the speech signal recognized by the speech recognizer, as a second spectrum;

(E) generating a modified spectrum in accordance with the first spectrum and the second spectrum; and

(F) outputting a synthesized speech signal based on the modified spectrum.

14. A program product for recognizing a speech signal in a computer, the computer executing the steps of:

(A) acquiring a sound signal from an object, and calculating a sound signal parameter based on the acquired sound signal;

(B) acquiring potential changes on a surface of the object as an electromyographic signal, and calculating an electromyographic signal parameter based on the acquired electromyographic signal;

(C) acquiring image information by taking an image of the object, and calculating an image information parameter based on the acquired image information;

(D) recognizing a speech signal vocalized by the object, based on the sound signal parameter, the electromyographic signal parameter and the image information parameter; and

(E) providing a result recognized by the speech recognizer.

15. A program product for synthesizing a speech signal in a computer, the computer executing the steps of:

(A) recognizing a speech signal;

(B) acquiring a sound signal;

(C) acquiring a spectrum of the acquired sound signal as a first spectrum;

(D) generating a reconfigured spectrum of the sound signal, based on the speech signal recognized by the speech recognizer, as a second spectrum;

(E) generating a modified spectrum in accordance with the first spectrum with the second spectrum; and

(F) outputting a synthesized speech signal based on the modified spectrum.