Securing audio data

• US 9,940,930 B1
• Filed: 12/07/2016
• Issued: 04/10/2018
• Est. Priority Date: 12/07/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Campbell, Lee, Beder, Samuel Kramer
• Primary Examiner(s): ROBERTS, SHAUN A
• Application Number:
  US15/372,182
• Time in Patent Office: 489 Days
• Field of Search: 704273
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04W 12/02 Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for securing audio data. In one aspect, a method includes restricting access by the device to audio information detected by a microphone, receiving data indicating that the device is authorized to access audio information detected by the microphone during a limited period of time, and in response to receiving data indicating that the device is authorized to access audio information detected by the microphone during the limited period of time, providing audio information to the device. The method also includes monitoring audio information detected by the microphone during the limited period of time for the presence of a hotword and after the end of the limited period of time, restricting access by the device to audio information detected by the microphone.

Claims

1. A method comprising:

restricting, by a processing module in a device, access by a processor in the device to audio information detected by a microphone in the device;

receiving, by the processing module in the device, data from the processor in the device indicating that the processor in the device is authorized during a limited period of time to access audio information detected by the microphone in the device;

in response to receiving data indicating that the processor in the device is authorized during the limited period of time to access audio information detected by the microphone in the device, providing the processor in the device access to audio information detected by the microphone in the device;

monitoring, by the processing module in the device, audio information detected by the microphone in the device during the limited period of time for the presence of a hotword; and

after the end of the limited period of time, restricting, by the processing module in the device, access by the processor in the device to audio information detected by the microphone in the device.

2. The method of claim 1, further comprising restricting access to audio information detected by the microphone in response to determining that the limited period of time has ended.

3. The method of claim 1, wherein receiving the data from the processor in the device indicating that the processor in the device is authorized during a limited period of time to access audio information detected by the microphone in the device comprises:

receiving a token from the processor indicating (i) that the processor in the device is authorized to access the audio information detected by the microphone during the limited period of time and (ii) a duration of the limited period of time;

wherein the method further comprises;

evaluating the token to verify validity of the token; and

in response to verifying the validity of the token, extracting data of the token indicating (i) that the processor in the device is authorized to access the audio information detected by the microphone during the limited period of time and (ii) a duration of the limited period of time.

4. The method of claim 1, wherein restricting access by the processor in the device to audio information detected by the microphone comprises at least one of:

encrypting the audio information detected by the microphone;

or

blocking access of the audio information from the processing module to the processor in the device.

5. The method of claim 1, further comprising:

detecting the presence of the hotword during the limited period of time; and

in response to detecting the presence of the hotword during the limited period of time, restricting, during the limited period of time, access by the processor in the device to audio information that is detected by the microphone and that follows the hotword.

6. The method of claim 5, further comprising:

detecting, during the limited period of time, a voice request that follows the hotword, wherein the voice request is a spoken command or request;

determining that the voice request has ended before the end of the limited period of time; and

in response to determining that the voice request has ended before the end of the limited period of time, providing, to the processor in the device, unencrypted audio information detected after the end of the voice request.

7. The method of claim 6, further comprising:

after determining that the voice request has ended before the end of the limited period of time, and after providing unencrypted audio information detected after the end of the voice request, detecting a second voice request that occurs during the limited period of time and that is not preceded by an additional hotword; and

in response to detecting the second voice request that occurs during the limited period of time and is not preceded by an additional hotword, providing encrypted audio corresponding to the second voice request to the processor in the device.

8. The method of claim 1, further comprising receiving a request to access audio information detected by the microphone, wherein the request is provided by the processor in the device in response to detecting user interaction with the processor in the device.

9. The method of claim 8, wherein receiving the request to access audio information detected by the microphone comprises associating the request with data derived from (i) an identifier for the processing module and (ii) data indicating a time.

20. The method of claim 1, wherein restricting, by a processing module in a device, access by a processor in the device to audio information detected by a microphone in the device comprises providing the processor in the device audio information detected by the microphone in the device in an encrypted form,

wherein in response to receiving data indicating that the processor in the device is authorized during the limited period of time to access audio information detected by the microphone in the device, providing the processor in the device access to audio information detected by the microphone in the device comprises in response to receiving data indicating that the processor in the device is authorized during the limited period of time to access audio information detected by the microphone in the device, providing the processor in the device access to audio information detected by the microphone in the device in an unencrypted form.

10. A device, comprising:

a microphone;

a processor that handles primary functions of the device; and

a processing module configured to perform operations comprising;

restricting, by a processing module in a device, access by a processor in the device to audio information detected by a microphone in the device;

receiving, by the processing module in the device, data from the processor in the device indicating that the processor in the device is authorized during a limited period of time to access audio information detected by the microphone in the device;

in response to receiving data indicating that the processor in the device is authorized during the limited period of time to access audio information detected by the microphone in the device, providing the processor in the device access to audio information detected by the microphone in the device;

monitoring, by the processing module in the device, audio information detected by the microphone in the device during the limited period of time for the presence of a hotword; and

after the end of the limited period of time, restricting, by the processing module in the device, access by the processor in the device to audio information detected by the microphone in the device.

11. The device of claim 10, wherein the operations further comprise restricting access to audio information detected by the microphone in response to determining that the limited period of time has ended.

12. The device of claim 10, wherein receiving the data from the processor in the device indicating that the processor in the device is authorized during a limited period of time to access audio information detected by the microphone in the device comprises:

receiving a token from the processor indicating (i) that the processor in the device is authorized to access the audio information detected by the microphone during the limited period of time and (ii) a duration of the limited period of time;

wherein the method further comprises;

evaluating the token to verify validity of the token; and

in response to verifying the validity of the token, extracting data of the token indicating (i) that the processor in the device is authorized to access the audio information detected by the microphone during the limited period of time and (ii) a duration of the limited period of time.

13. The device of claim 10, wherein restricting access by the processor in the device to audio information detected by the microphone comprises at least one of:

encrypting the audio information detected by the microphone;

or

blocking access of the audio information from the processing module to the processor in the device.

14. The device of claim 10, wherein the operations further comprise:

detecting the presence of the hotword during the limited period of time; and

in response to detecting the presence of the hotword during the limited period of time, restricting, during the limited period of time, access by the processor in the device to audio information that is detected by the microphone and that follows the hotword.

15. The device of claim 14, wherein the operations further comprise:

detecting, during the limited period of time, a voice request that follows the hotword, wherein the voice request is a spoken command or request;

determining that the voice request has ended before the end of the limited period of time; and

in response to determining that the voice request has ended before the end of the limited period of time, providing, to the processor in the device, unencrypted audio information detected after the end of the voice request.

16. The device of claim 15, wherein the operations further comprise:

after determining that the voice request has ended before the end of the limited period of time, and after providing unencrypted audio information detected after the end of the voice request, detecting a second voice request that occurs during the limited period of time and that is not preceded by an additional hotword; and

in response to detecting the second voice request that occurs during the limited period of time and is not preceded by an additional hotword, providing encrypted audio corresponding to the second voice request to the processor in the device.

17. The device of claim 10, wherein the operations further comprise receiving a request to access audio information detected by the microphone, wherein the request is provided by the processor in the device in response to detecting user interaction with the processor in the device.

18. The device of claim 17, wherein receiving the request to access audio information detected by the microphone comprises associating the request with data derived from (i) an identifier for the processing module and (ii) data indicating a time.

19. One or more non-transitory computer-readable media storing software comprising instructions executable by one or more processors that, upon such execution, cause the one or more processors to perform operations comprising:

restricting, by a processing module in a device, access by a processor in the device to audio information detected by a microphone in the device;

receiving, by the processing module in the device, data from the processor in the device indicating that the processor in the device is authorized during a limited period of time to access audio information detected by the microphone in the device;

in response to receiving data indicating that the processor in the device is authorized during the limited period of time to access audio information detected by the microphone in the device, providing the processor in the device access to audio information detected by the microphone in the device;

monitoring, by the processing module in the device, audio information detected by the microphone in the device during the limited period of time for the presence of a hotword; and

after the end of the limited period of time, restricting, by the processing module in the device, access by the processor in the device to audio information detected by the microphone in the device.