Process for automatic control of one or more devices by voice commands or by real-time voice dialog and apparatus for carrying out this process

• US 6,839,670 B1
• Filed: 09/09/1996
• Issued: 01/04/2005
• Est. Priority Date: 09/11/1995
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Inventors: Heinrich, Christian, Reh, Frank, Class, Fritz, Nussle, Gerhard, Stammler, Walter, Buschkuhl, Burkard, M{hacek over (o)}ller, Carsten-Uwe
• Primary Examiner(s): Knepper, David D.
• Application Number:
  US09/043,134
• Time in Patent Office: 3,039 Days
• Field of Search: 704246-257, 704270-275
• US Class Current: 704/251
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A speech dialog system wherein a process for automatic control of devices by speech dialog is used applying methods of speech input, speech signal processing and speech recognition, syntatical-grammatical postediting as well as dialog, executive sequencing and interface control, and which is characterized in that syntax and command structures are set during real-time dialog operation; preprocessing, recognition and dialog control are designed for operation in a noise-encumbered environment; no user training is required for recognition of general commands; training of individual users is necessary for recognition of special commands; the input of commands is done in linked form, the number of words used to form a command for speech input being variable; a real-time processing and execution of the speech dialog is established; and the speech input and output is done in the hands-free mode.

Claims

1. A process for the automatic control of one or several devices by speech command or by speech dialog in real-time operation, wherein:

entered speech commands are recognized by a speaker-independent compound-word speech recognizer and a speaker-dependent speech recognizer and are classified according to their recognition probability;

recognized, admissible speech commands are checked for their plausibility, and the admissible and plausible speech command with the highest recognition probability is identified as the entered speech command, and the functions assigned to this speech command of the device or devices or responses of the speech dialog system are initiated or generated;

a process wherein;

one of the speech commands and the speech dialogs is formed or controlled, respectively on the basis of at least one syntax structure, at least one base command vocabulary and, if necessary, at least one speaker-specific additional command vocabulary;

the syntax structure and the base command vocabulary are provided in speaker-independent form and are fixed during real-time operation;

the speaker-specific additional command vocabulary or vocabularies are entered or changed by the respective speaker in that during training phases during or outside of the real-time operation, the speech recognizer that operates on the basis of a speaker-dependent recognition method is trained by the respective speaker through single or multiple input of the additional commands for the speech-specific features of the respective speaker;

in the real-time operation, the speech dialog or the control of the device or devices takes place as follows;

speech commands spoken in by the respective speaker are transmitted to a speaker-independent compound-word recognizer operating on the basis of phonemes or whole-word models and to the speaker-dependent speech recognizer, where they are respectively subjected to a feature extraction and

are examined and classified in the compound-word speech recognizer with the aid of the features extracted there to determine the existence of base commands from the respective base command vocabulary according to the respectively specified syntax structure, and

are examined and classified in the speaker-dependent speech recognizer with the aid of the features extracted there to determine the existence of additional commands from the respective additional command vocabulary;

the commands that have been classified as recognized with a certain probability and the syntax structures of the two speech recognizers are then joined to form hypothetical speech commands, and that these are examined and classified according to the specified syntax structure as to their reliability and recognition probability;

the admissible hypothetical speech commands are subsequently examined as to their plausibility on the basis of predetermined criteria, and that among the hypothetical speech commands recognized as plausible, the one with the highest recognition probability is selected and is identified as the speech command entered by the respective speaker;

that subsequently

a function or functions assigned to the identified speech command of the respective device to be controlled are initiated or

a response or responses are generated in accordance with a specified speech dialog structure for continuing the speech dialog.

2. A process according to claim 1, characterized in that the input of speech commands occurs acoustically and preferably in the hands-off operation.

3. A process according to claim 2, characterized in that the acoustically input speech commands are transmitted noise-reduced to the two speech recognizers, in that noise signals, caused by stationary or quasi-stationary environmental noises, are compensated in the speech signal receiving channel in front of the two speech recognizers and preferably by means of adaptive digital filtering methods.

4. A process according to claim 2, characterized in that the acoustically input speech commands are transmitted echo-compensated to the two speech recognizers, in that signals of a speech or music output unit that are fed back into the speech signal receiving channel are compensated in the speech signal receiving channel in front of the two speech recognizers, in particular in front of the noise reduction unit, and preferably by means of adaptive digital filtering methods.

5. A process according to claim 1, characterized in that the entered speech commands initially are combined in blocks after digitizing, are converted to the frequency range following a weighting of a spectral transformation, preferably a Fast Fourier Transformation (FFT), and are subsequently combined to form channel vectors through sum formation and subsequent audio-related MEL filtering, and that this is followed by a segmentation.

6. A process according to claim 5, characterized in that the segmentation is divided into a rough and a precise segmentation.

7. A process according to claim 5, characterized in that feature extraction is carried out in the speaker-independent compound-word recognizer such that

the channel vectors are transformed with a discrete cosine transformation (DCT) into cepstral vectors;

additionally the energy of the associated signal is calculated and standardized;

in order to adapt the recognizer to the respective speaker or the respective transmission characteristics of the speech signal receiving channel, the cepstral vector mean value is constantly computed and is subtracted from the cepstral vectors;

the cepstral vectors freed of the cepstral vector mean value and the computed, standardized signal energy are combined to form mean-value free cepstral coefficients (CMF vectors).

8. A process according to claim 5, characterized in that the speech dialog system in vehicles is used for individual or a plurality of the functions named in the following:

the operation of individual or multiple devices, e.g. car telephone, car radio (if necessary with tape deck, CD changer, sound system), navigation system, emergency call, onboard monitor, air-conditioning system, heater, travel computer, lighting, sun roof, window opener, seat adjuster;

information polling of parameters, e.g. oil pressure, oil temperature, cooling-water temperature, consumption, tire pressure;

information concerning necessary measures such as a cooling-water temperature that is too high, a tire pressure that is too low;

warning of the driver in case of vehicle malfunctions.

9. A process according to claim 8, characterized in that the speech controlled selection of a new station on the car radio occurs based on the following processes:

issuing a command for the search operation up or down;

speech input of the station frequency, preferably in the colloquial form, and preferably also including the frequency information;

speech input of the commonly used station name.

10. A process according to claim 8, characterized in that the desired temperature for the air-conditioning system can be set relatively and/or preferably absolutely through speech input, and that in addition, a command for a minimum, maximum, or average temperature or a normal temperature can preferably be issued.

11. A process according to claim 8, characterized in that a navigation system is informed of a target location (location name, street name) through the input of the letter columns in the “

spelling mode,”

wherein the beginning of a name is preferably sufficient for the input and wherein the navigation system, if necessary, offers several candidates for selection.

12. A process according to claim 8, characterized in that one or several of the following, user-specific name lists are set up:

a list for storing telephone numbers under names/abbreviations that can be specified;

a list for storing targets for the navigation system under names/abbreviations that can be specified;

a list for storing function names for commands or command sequences;

a list for storing station frequencies for car radios under specifiable station names or abbreviations.

13. A process according to claim 8, characterized in that the volume for the speech output and the control sound or the control sounds, if necessary also the radio volume, are adaptively adjusted or set by taking into account one or several of the following parameters:

vehicle speed

fan setting

rotational number

opening width for the window and sun roof

vehicle type

importance of the speech output in the respective dialog situation.

14. A process according to claim 1, characterized in that the speaker-independent compound-word recognizer uses Hidden Markov Models (HMM) based on phonemes or whole words for classification.

15. A process according to claim 14, characterized in that the classification is carried out with the aid of a Viterbi algorithm and that the Viterbi algorithm preferably is complemented by a specified word-sequence statistic.

16. A process according to claim 1, characterized in that for classification, filler words or filler phonemes or other faulty commands not included in a specified basic vocabulary are recognized as such, are correspondingly classified, and are separated out.

17. A process according to claim 1, characterized in that the speaker-independent compound-word speech recognizer and the speaker-dependent additional speech recognizer build onto the same signal preprocessing for the input speech commands, preferably including methods for noise reduction, echo compensation, and segmentation.

18. A process according to claim 1, characterized in that the additional speech recognizer operates as a single-word speech recognizer, preferably based on the dynamic time warping process.

19. A process according to claim 1, characterized in that the speaker-independent compound-word speech recognizer and the speaker-dependent speech recognizer operate jointly in the compound-word mode.

20. A process according to claim 1, characterized in that during real-time operation, there is a continuous adaptation of the speech signal receiving channel to the analog transmission characteristic, in particular to the characteristic for acoustic or microphone or amplifier and/or speaker.

21. A process according to claim 1, characterized in that the predetermined basic commands are specified and stored in speech-encoded form or the additional commands input by the respective speaker during the training phases or the speech commands input during the real-time operation are further processed in the speech-encoded form following their input or are stored in a non-volatile memory, and that encoded speech commands that must be output acoustically are speech-decoded prior to their output.

22. A process according claim 1, characterized in that the specified basic commands or the additional commands or the speech commands input during the real-time operation are stored in text form, and that speech commands that must be output acoustically are subjected to a text-to-language synthesis prior to their output.

23. A process according to claim 1, characterized in that the syntax structure and the speaker-independent commands are created and fixed ahead of time in the “

off-line dialog editor mode”

in the laboratory and are transmitted to the compound-word speech recognizer in the form of data files.

24. A process according to claim 1, characterized in that

the word order in the speech commands can be changed by exchanging the individual words in a command or

specified synonymous words can be used for generating the speech command or

the same function can be realized through speech commands with a varying number of words.

25. A process according to claim 1, characterized in that for the recognition and subsequent separating out of insertions or other commands not belonging to the vocabulary, additional words or phonemes are added to the admissible vocabulary or that word spotting approaches are used.

26. A process according to claim 1, characterized in that the dialog structure has the following features:

a flat hierarchy with only a few hierarchy levels, preferably one or two hierarchy levels,

integration of ellipses for the processing of the speech dialog;

including of auxiliary and information menus;

including of repetition requests from the speech dialog system in case of unsure decisions by the recognizer

including of speech outputs, in order to increase the recognition certainty by stimulating certain manners of speech.

27. A process according to claim 1, characterized in that the speech recognition or the speech dialog for the control of one or several device functions is preferably activated by a one-time actuation of a push-to-talk key (PTT) and that this activation is preferably acknowledged acoustically or optically.

28. A process according to claim 27, characterized in that the activation is terminated automatically if no speech input has occurred, following a time interval that can be specified or adaptively adjusted to the respective user, or following a repetition request by the speech dialog system, or if the dialog selected by the user has been completed according to plan.

29. A process according to the claim 27, characterized in that the push-to-talk key either makes use of or contains multifunctions, e.g. when using the telephone (“

replace receiver,”

“

lift off receiver”

) or for the restart of the speech dialog system or when terminating the telephone dialing operation;

or is complemented by an additional switch or an additional switching position that permits, for example a restart of the speech dialog system or when terminating the telephone dialing operation.

30. A process according to claim 1, characterized in that the speech dialog or the input of speech commands can be terminated through the input of a specified, special termination speech command at defined locations in the speech dialog or at any time by actuating a key, preferably a push-to-talk key or an escape key.

31. A process according to claim 1, characterized in that the speech dialog system has at least one of the following performance characteristics:

the specific (e.g. trained) speech commands from various speakers are stored, if necessary, for a later reuse;

speech commands or names trained by the speaker are not only transmitted to the recognition system during a training phase, but are also recorded as to their time history, are transmitted to a data compression (“

speech encoding”

), and are stored in a non-volatile memory;

speech commands trained by the speaker are processed during the training phase such that environmental noises are for the most part compensated during the recording.

32. A process according to claim 1, characterized in that the completion of a recognition operation is acknowledged acoustically with a control sound.

33. A process according to claim 1, characterized in that a recognition result is acoustically repeated (speech output), especially for decisions involving safety, item, or cost and that the speaker is given the option of preventing or reversing the carrying out of the function assigned to the speech command with the aid of a speech command or by actuating a switch, preferably a push-to-talk key or an escape key.

34. A process according to claim 1, characterized in that the speech dialog system is connected to an optical display medium, preferably a LCD display or a monitor or a display for a selected device.

35. A process according to claim 34, characterized in that the optical display medium takes over individual or a plurality of the following functions:

output of the recognized speech command for control purposes;

illustration of the functions adjusted by the target device in reaction to the speech command;

illustration of various functions/alternatives, which are subsequently adjusted or selected or modified by speech command.

36. A process according to claim 1, characterized in that each speaker can set up his/her own name or abbreviation lists, comprising at least one of the following features:

the name trained by the speaker on the speaker-dependent recognizer represents a number sequence, a letter sequence and/or a command or a command sequence, input in the speaker-independent operating mode;

the user can input the list designation and the name selected by the user in place of the renewed input of the number sequence, letter sequence or command sequence, or the user can input a suitable command in addition to the name, which suggests the correct list;

the list can be expanded at any time by speech control to comprise further entries;

the list can be deleted completely or selectively with speech control;

the list can be listened to for a speech command, wherein the names input by the user and, if necessary, the associated number sequence, letter sequence or commands can be output acoustically;

the acoustic output of the list be terminated at any point in time.

37. A process according to claim 1, characterized in that a sequence of numbers (number column) can be spoken in either in a linked form or in blocks, wherein the speech input or the speech dialog exhibits at least one of the following features:

each input pause is followed by an acknowledgment in which the last input block if repeated by the speech output;

following the acknowledgment through a speech command “

error”

or the like, the last input block is deleted and the remaining, stored blocks are acoustically output;

following the acknowledgment through a speech command “

delete”

or the like, all entered number blocks are deleted;

following the acknowledgment through a speech command “

repeat”

or the like, the blocks stored until then are output acoustically;

following the acknowledgment through a speech command “

termination”

or the like, the input of the number column is terminated completely;

additional numbers or number blocks can be input following the acknowledgment;

following the acknowledgement, the number input is ended with a suitable speech command “

stop,”

“

store,”

or the like;

the input is completed by entering a speech command starting an action/function, e.g. “

select”

or the like, which initiates the action/function associated with the speech command.

38. A process according to claim 37, characterized in that the same blocking as for the input is used for the output of the numbers entered until then, which output follows the speech command “

error”

or the like or the speech command “

repeat”

or the like.

39. A process according to claim 1, characterized in that a sequence of letters (letter column) is spoken in, which is provided for the selection of complex functions or the input of a plurality of information bits, wherein the letter column is entered either linked together or in blocks and the speech input or the speech dialog exhibits at lease one of the following features:

each input pause is followed by an acknowledgement, in which the last input block is repeated by the speech output;

following the acknowledgment through a speech command “

error”

or the like, the last input block is deleted and the remaining, stored blocks are output acoustically;

following the acknowledgment through a speech command “

delete” and

the like, all previously entered letters are deleted and a new input can subsequently take place;

following the acknowledgement through a speech command “

repeat”

or the like, the blocks stored until then are output acoustically;

following the acknowledgment, additional letters or letter blocks can be input;

if necessary, the letter column or the individual letter blocks are matched with a stored word list, and the most suitable word(s) is (are) extracted from this;

following the acknowledgment through a speech command “

termination”

or the like, the input of the letter column is terminated completely;

following the acknowledgment, the letter input is completed with a speech command “

stop,”

“

store,”

or the like;

the input is completed by entering a speech command starting an action/function, such as “

select”

or the like, and the action/function associated with the speech command is initiated.

40. A process according to claim 1, characterized in that the speech output volume and the control sound volume are adapted to the environmental noises, wherein the environmental noises are detected during the speaking pauses with respect to the strength and characteristic.

41. A process according to claim 1, characterized in that access to the speech dialog system or access to user-specific data/commands can be gained only through the input of special command words or the input of special command words from an authorized speaker, whose speech characteristics are known to the speech dialog system and rare analyzed by this system.

42. A process according to claim 1, characterized in that speech output operations of a longer duration (e.g. information menus) can be terminated prematurely through spoken or manual termination commands.

43. A process according to claim 1, characterized in that the speech dialog system in one of the following forms complements or replaces the manual operation of the above functions (e.g. by switch, key, rotary button):

the speech command control exists in addition to the manual operation, so that it is possible at any time to have a manual operation or to continue the operation manually;

some special performance characteristics can be activated only by speech input, while other device functions and operating functions continue to be controlled manually as well as by speech;

the number of manual operating elements is clearly reduced, and individual keys or rotary knobs take over multiple functions;

manual operating element are assigned a special function by each language;

only the essential operating functions can still be actuated manually;

the speech command control forms the basis for the operating functions.

44. A process according to claim 1, characterized in that a plurality of different devices as well as device functions can be addressed and modified with a single one-word or multiword command, and a multistage action is therefore not required at all or only to some degree.

45. An apparatus for the automatic control of one or several devices by speech commands or by speech dialog in real-time operation, wherein the entered speech commands are recognized by a speaker-independent compound-word speech recognizer and a speaker-dependent speech recognizer and are classified according to their recognition probability, recognized, admissible speech commands are checked for their plausibility, the admissible and plausible speech command with the highest recognition probability is identified as the entered speech command, and the functions associated with the identified speech command for the device or devices, or the responses of the speech dialog system are initiated or generated;

with the apparatus including a voice input/output unit that is connected via a speech signal preprocessing unit with a speech recognition unit, which in turn is connected to a sequencing control, a dialog control, and an interface control, and wherein the speech recognition unit consists of a speaker independent compound-word recognizer and a speaker-dependent additional speech recognizer, which are both connected on the output side with a unit for combined syntactical-grammatical or semantical post processing, with said unit being linked to the sequencing control, the dialog control, and the interface control.

46. An apparatus according to claim 45, characterized in that the speech signal preprocessing unit includes a noise reduction device and/or an echo compensation device and/or a segmenting device.

47. An apparatus according to claim 45, characterized in that the speech input/output unit includes a speech encoder, a speech decoder, as well as a speech memory.

48. An apparatus according to claim 45, characterized in that the sequencing control, the dialog control, and the interface control, the speech input/output, as well as the speech signal preprocessing, the speech recognition, the syntactical-grammatical and semantical postprocessing are carried out with microprocessors and signal processors, memories, and interface modules, but preferably with a single digital signal processor or microprocessor as well as the required external memories for data and programs, the interfaces, as well as the associated driver modules, a clock generator, a control logic, and the microphones and speakers necessary for the speech input/output, including the associated converters and amplifiers, as well as a push-to-talk (PTT) key and an escape key if necessary.

49. An apparatus according to claim 48, characterized in that with the aid of one or several interfaces,

data and/or parameters can be loaded or reloaded, e.g. to realize processing changes or a speech dialog system for another language;

the syntax structure, dialog structure, sequencing control, speech output, etc., which are fixed or modified on a separate computer, are transmitted to the speech dialog system (“

off-line dialog editor”

);

diagnostic and status information can be requested and collected by the speech dialog system.

50. An apparatus according to claim 48, characterized in that this apparatus is linked via a bus system or a ring-shaped net with several devices to be controlled, and that control data and/or audio signals and/or status reports of the speech dialog system and/or the devices to be operated can be transmitted via this bus or the net.

51. An apparatus according to claim 50, characterized in that this bus or this net is an optical data bus, and that control signals as well as audio signals or status reports from the speech dialog system and the devices to be operated are transmitted via this data bus or net.

52. An apparatus according to claim 45 for use in vehicles, characterized in that the individual devices to be selected do not contain a separate speech dialog system each, but are operated with the aid of a single, joint speech dialog system.

53. An apparatus according to claim 45, characterized by the existence of one or several interfaces to vehicle components or vehicle computers, which are used to provide the speech dialog system with permanent or up-to-date vehicle data, e.g., the speed.

54. An apparatus according to claim 45 characterized in that this apparatus takes on other functions, e.g. for the radio, telephone, etc., during the waiting periods in which no speech input or speech output occurs.

55. An apparatus according claim 45, characterized in that a multilingual, speaker-independent dialog system is realized by means of an expanded memory, which permits the switching between the dialog systems of various languages.

56. An apparatus according to claim 45, characterized in that an optical display is coupled to the speech dialog system via a special interface or via the bus connection.

57. An apparatus according to claim 45, characterized in that the complete speech dialog system is coupled via a PCMCIA interface with the speech-controlled or speech-operated device or with a host computer or an application computer.