Protection of private information in a client/server automatic speech recognition system

• US 9,131,369 B2
• Filed: 01/24/2013
• Issued: 09/08/2015
• Est. Priority Date: 01/24/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Ganong, William F. III, Vozila, Paul J.
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US13/748,878
• Publication Number:
  US 20140207442A1
• Time in Patent Office: 957 Days
• Field of Search: 704/201
• US Class Current: 1/1
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 21/00 Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility G10L19/00 takes precedence
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04W 12/02 Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]

Abstract

A mobile device is adapted for protecting private information on the mobile device in a hybrid automatic speech recognition arrangement. The mobile device includes a speech input component for receiving a speech input signal from a user. Additionally, the mobile device includes a local ASR arrangement for performing local ASR processing of the speech input signal and determining if private information is included within the speech input signal. A control unit on the mobile device obscures private information in the speech input signal if the local ASR arrangement identifies information within a speech recognition result as private information. The control unit releases the speech input signal with the obscured private information for transmission to a remote server for further ASR processing.

Results from the remote server'"'"'s ASR processing are integrated and combined with results from local ASR processing to display information on the mobile device.

Claims

1. A mobile device adapted for automatic speech recognition (ASR) for protecting private information on the mobile device comprising:

a speech input component for receiving a speech input signal from a user;

a local data store;

a local ASR arrangement for performing local ASR processing of the speech input signal to produce a local speech recognition result and for determining if private information is included within the speech input signal by comparison of the local speech recognition result with data within the local data store; and

a control unit for obscuring private information in the speech input signal if the local ASR arrangement identifies private information and for sending the speech input signal with the obscured private information to a remote server for ASR processing.

2. A mobile device according to claim 1, wherein the control unit creates a hash code of identified private information and inserts the hash code into the speech input signal prior to sending the speech input signal to the remote server.

3. A mobile device according to claim 1, further including:

a user interface allowing a user to identify private information not to be transmitted to a remote server.

4. A mobile device according to claim 1, wherein the control unit obscures the private information by removing a portion of the speech input signal associated with the private information prior to transmitting the speech input signal to the remote server.

5. A mobile device according to claim 2, wherein the control unit receives a recognized version of the speech input including an indicia of the hash code and the control unit inserts the private information into the recognized version of the speech input at the indicia.

6. A mobile device according to claim 1, wherein the control unit obscures the private information by identifying the private information within the speech input signal and replacing the private information with a different signal.

7. A mobile device according to claim 1, wherein a log including speech processing events is sent to the remote server by the control unit.

8. A mobile device according to claim 7, wherein the log includes a text presentation of the speech input signal as recognized by the local ASR arrangement and any private information within the log is obscured with a hash code.

9. A mobile device according to claim 1, wherein, if the private information identified by the local ASR arrangement is contact information, the control unit accesses a contact list on the mobile device and transmits only a portion of the contact list to the server.

10. A mobile device according to claim 9, wherein the portion of the contact list includes last names without first names.

11. A mobile device adapted for automatic speech recognition (ASR) for protecting private information on the mobile device comprising:

a speech input for receiving a speech input signal from a user;

a local data store;

a local ASR arrangement for performing local ASR processing of the speech input signal using identified private information sources within the local data store on the mobile device;

wherein, if private information is identified in the result of the local ASR processing of the speech input signal, the control unit declassifies the private information; and

a control unit for sending the speech input signal and declassified private information from the identified private information sources to a remote server for ASR processing.

12. A mobile device according to claim 11, wherein the identified private information sources include an address book.

13. A mobile device according to claim 11, wherein the result of the local ASR processing may include an N-best list and the declassified private information may include words or names in the N-best list and associated data.

14. A mobile device according to claim 11, wherein a subset of data from the private information sources is sent to the remote server for ASR processing.

15. A mobile device according to claim 14, wherein the subset of data may include a limited number of fields from a private information source database.

16. A method for protecting private information on a mobile device during automatic speech recognition, the method comprising:

performing automatic speech recognition on an unknown speech signal on the mobile device producing a local speech recognition result;

identifying private information within the local speech recognition result based on a comparison with data within a local data store on the mobile device;

obscuring private information within the speech signal;

transmitting the speech signal with the obscured private information to a server;

receiving a remote speech recognition result from the server; and

outputting a combined speech recognition result based upon the local speech recognition result and the remote speech recognition result, wherein the private information is provided in the combined speech recognition result.

17. A method according to claim 16, further comprising:

creating a speech log including a speech recognition result.

18. A method according to claim 17, further comprising:

obscuring private information within the speech log prior to transmitting the speech log to the server.

19. A method according to claim 18, wherein the combined speech recognition result is output to a display device providing one or more possible speech recognition results based upon probability data from both the local speech recognition result and the remote speech recognition result.

20. A method for protecting private information on a mobile device during automatic speech recognition, the method comprising:

performing automatic speech recognition on an unknown speech signal on the mobile device producing a local speech recognition result;

identifying private contact information within the local speech recognition result based on a comparison with a contact list database within a local data store on the mobile device;

obscuring a first portion of the private contact information within the speech signal;

transmitting the speech signal with a second portion of the private contact information to a server, wherein the first portion of the private contact information is obscured;

receiving a remote speech recognition result from the server; and

outputting a combined speech recognition result based upon the local speech recognition result and the remote speech recognition result, wherein the private contact information is provided in the combined speech recognition result.

21. A method according to claim 20, further comprising:

providing a user interface for selection of the second portion of the private contact information to transmit.