DIGITAL ASSISTANT PROVIDING WHISPERED SPEECH

• US 20170358301A1
• Filed: 09/15/2016
• Published: 12/14/2017
• Est. Priority Date: 06/10/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: RAITIO, Tuomo J., HUNT, Melvyn J., RICHARDS, Hywel B., CHINTHAKUNTA, Madhusudan
• Granted Patent: US 10,192,552 B2
• Time in Patent Office: Days
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum

Abstract

Systems and processes for detecting and/or providing a whispered speech response are provided. In one example process, speech is received from a user, and based on the speech input, determined that a whispered speech response is to be provided. Upon determining that a whispered speech response is to be provided, the whispered speech response is generated and provided to the user.

Claims

1. An electronic device, comprising:

one or more processors;

memory; and

one or more programs stored in memory, the one or more programs including instructions for;

receiving a speech input from a user;

determining whether providing a whispered speech response is disabled;

in accordance with a determination that providing the whispered speech response is not disabled,

determining, based on the speech input, that a whispered speech response is to be provided;

upon determining that a whispered speech response is to be provided, generating the whispered speech response;

providing the whispered speech response to the user;

in accordance with the determination that providing the whispered speech response is disabled,

generating a non-whispered speech response, and

providing the non-whispered speech response to the user in lieu of the whispered speech response.

2. The electronic device of claim 1, wherein the speech input comprises at least one of an informational request or a request to perform a task.

3. The electronic device of claim 2, wherein the whispered speech response comprises at least one of a response to the informational request or a response associated with performing the task.

4. The electronic device of claim 1, wherein determining that the whispered speech response is to be provided comprises at least one of:

determining whether the speech input includes a whispered speech input; and

determining whether context data indicates that the whispered speech response is expected.

5. The electronic device of claim 4, wherein the whispered speech input is associated with a first spectrum having one or more first spectrum characteristics associated with a whispered speech.

6. The electronic device of claim 5, wherein the one or more first spectrum characteristics comprise at least one of:

a first amplitude, wherein the first amplitude is less than a second amplitude below a threshold frequency, the second amplitude being associated with the non-whispered speech;

a first energy, wherein the first energy is less than a second energy below the threshold frequency, the second energy being associated with the non-whispered speech;

a first volume, wherein the first volume is less than a second volume by a threshold volume percentage, the second volume being associated with the non-whispered speech; and

a first slope of the first spectrum, wherein the first slope of the first spectrum is shifted by a threshold slope percentage with respect to a second slope of the second spectrum, the second slope of the second spectrum being associated with the non-whispered speech.

7. The electronic device of claim 5, wherein determining whether the speech input includes a whispered speech input comprises:

determining whether the speech input includes a whispered speech input using one or more features of the speech input, wherein the one or more features represent one or more spectrum characteristics associated with a spectrum of the speech input.

8. The electronic device of claim 7, wherein determining whether the speech input includes a whispered speech input using the one or more features comprises:

obtaining the spectrum of the speech input;

determining the one or more spectrum characteristics associated with the spectrum of the speech input; and

determining a first feature and a second feature based on the one or more spectrum characteristics associated with the spectrum of the speech input.

9. The electronic device of claim 8,

wherein the first feature is a first mel-frequency cepstrum coefficient (MFCC0) representing an energy or an amplitude associated with the spectrum of the speech input;

and

wherein the second feature is a second mel-frequency cepstrum coefficient (MFCC1) representing the slope associated with the spectrum of the speech input.

10. The electronic device of claim 8, further comprising:

obtaining a whisper score based on the first feature to the second feature; and

determining whether the whisper score satisfies a score threshold.

11. The electronic device of claim 4, wherein determining whether the context data indicates that the whispered speech response is expected comprises:

obtaining the context data provided by the at least one of the user device or one or more additional devices communicatively connected to the user device; and

determining whether the context data satisfy one or more conditions for providing the whispered speech response.

13. The electronic device of claim 1, wherein generating the whispered speech response comprises:

generating an intermediate speech based on the speech input; and

generating the whispered speech response using the intermediate speech.

14. The electronic device of claim 13, wherein the intermediate speech has substantially the same content as the whispered speech response.

15. The electronic device of claim 13, wherein generating the intermediate speech based on the speech input comprises:

generating text based on the speech input;

performing natural language processing of the text;

identifying a user intent based on a result of the natural language processing; and

generating the intermediate speech according to the user intent.

16. The electronic device of claim 13, wherein generating the whispered speech response using the intermediate speech comprises:

obtaining a residual signal based on a linear prediction analysis of the intermediate speech;

modifying the residual signal; and

obtaining the whispered speech response based on a linear prediction synthesis of the modified residual signal.

17. The electronic device of claim 16, wherein obtaining the residual signal based on a linear prediction analysis of the intermediate speech comprises:

obtaining a plurality of speech frames using the intermediate speech; and

performing the linear prediction analysis of the plurality of speech frames.

18. The electronic device of claim 17, wherein performing the linear prediction analysis of the plurality of speech frames comprises:

pre-emphasizing the plurality of speech frames;

estimating a plurality of linear prediction coefficients; and

inverse filtering the pre-emphasized speech frames to obtain the residual signal.

19. The electronic device of claim 18, wherein estimating the plurality of linear prediction coefficients comprises:

performing a windowing on the pre-emphasized plurality of speech frames.

20. The electronic device of claim 16, wherein modifying the residual signal comprises:

receiving a white noise sequence;

estimating energy of the white noise sequence and the residual signal;

correlating the energy of the white noise sequence and the energy of the residual signal; and

compensating the correlated white noise sequence.

21. The electronic device of claim 20, wherein compensating the correlated white noise sequence comprises performing at least one of differentiating, high-pass filtering, or band-pass filtering with respect to the correlated white noise sequence.

22. The electronic device of claim 16, wherein obtaining the whispered speech response based on the linear prediction synthesis of the modified residual signal comprises:

obtaining a plurality of linear prediction coefficients;

modifying the linear prediction coefficients; and

performing a linear prediction synthesis of the modified residual signal using the modified linear prediction coefficients.

23. The electronic device of claim 22, wherein modifying the linear prediction coefficients comprises:

converting the plurality of linear prediction coefficients to line spectral frequencies;

modifying the line spectral frequencies; and

generating modified linear prediction coefficients based on the modified line spectral frequencies.

24. The electronic device of claim 22, wherein performing a linear prediction synthesis of the modified residual signal using the modified linear prediction coefficients comprises:

generating a plurality of whispered frames using a synthesis filter and the modified linear prediction coefficients; and

generating the whispered speech response using the plurality of whispered frames.

12. (canceled)

25. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receive a speech input from a user;

determining whether providing a whispered speech response is disabled;

in accordance with a determination that providing the whispered speech response is not disabled,

determine, based on the speech input, that a whispered speech response is to be provided;

upon determining that a whispered speech response is to be provided, generate the whispered speech response;

provide the whispered speech response to the user;

in accordance with the determination that providing the whispered speech response is disabled,

generating a non-whispered speech response, and

providing the non-whispered speech response to the user in lieu of the whispered speech response.

26. A method for operating a digital assistant, comprising:

at a user device with one or more processors and memory;

receiving a speech input from a user;

in accordance with a determination that providing the whispered speech response is not disabled,

determining, based on the speech input, that a whispered speech response is to be provided;

upon determining that a whispered speech response is to be provided, generating the whispered speech response; and

providing the whispered speech response to the user;

in accordance with the determination that providing the whispered speech response is disabled,

generating a non-whispered speech response, and

providing the non-whispered speech response to the user in lieu of the whispered speech response.