Device-described natural language control

• US 10,229,678 B2
• Filed: 10/14/2016
• Issued: 03/12/2019
• Est. Priority Date: 10/14/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hutchings, Justin A.
• Primary Examiner(s): Yen, Eric
• Application Number:
  US15/294,322
• Publication Number:
  US 20180108349A1
• Time in Patent Office: 879 Days
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/28 Constructional details of speech recognition systems
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

A remote device has an associated natural language description that includes a record of commands supported by the remote device. This record of commands includes command names, the command functions to which those names correspond, and natural language strings that are the natural language words or phrases that correspond to the command. A computing device includes a device control module that obtains the natural language description for the remote device and provides the natural language strings to a natural language assistant on the computing device. The natural language assistant monitors the natural language inputs to the computing device, and notifies the device control module when a natural language input matches one of the natural language strings. The device control module uses the natural language description to determine the command name that corresponds to the matching natural language string, and communicates the command name to the remote device.

Claims

1. A method implemented in a computing device, the method comprising:

establishing a communication link with a remote device having a plurality of natural language descriptions that map one or more natural language strings to individual functionalities of the remote device in a plurality of different natural languages that are supported by the remote device;

determining that the plurality of different natural languages that are supported by the remote device includes a particular natural language that is used by a natural language assistant that operates on the computing device;

sending, to the remote device having the plurality of natural language descriptions, a request for at least one natural language description for the remote device in the particular natural language that is used by the natural language assistant to receive natural language inputs at the computing device;

receiving, from the remote device, the at least one natural language description in the particular natural language;

updating, at the computing device, the natural language assistant based on the at least one natural language description to enable the natural language assistant to monitor, at the computing device, the natural language inputs for the remote device in the particular natural language;

receiving, from the natural language assistant, an indication that a particular natural language input to the computing device matches a particular natural language string in the at least one natural language description;

identifying a particular functionality of the remote device that maps to the particular natural language string; and

communicating a command to the remote device to perform the particular functionality.

2. The method as recited in claim 1, the one or more natural language strings comprising one or more words.

3. The method as recited in claim 1, the one or more natural language strings comprising one or more phrases.

4. The method as recited in claim 1, the one or more natural language strings comprising one or more optional words or characters.

5. The method as recited in claim 1, the particular natural language input comprising voice input.

6. The method as recited in claim 1, the particular natural language input comprising text or chat input.

7. The method as recited in claim 1, the at least one natural language description being a schema for a class of devices that includes the remote device.

8. The method as recited in claim 1, the remote device having a different natural language description for each of the plurality of different languages that are supported by the remote device.

9. The method as recited in claim 1, the remote device comprising a smart device.

10. A computing device comprising:

one or more processing units; and

a computer-readable medium having encoded thereon computer-executable instructions to cause the one or more processing units to;

determine that a plurality of languages that are supported by a remote device includes a particular language that is used by a natural language assistant that operates on the computing device;

communicate to the remote device, that maintains natural language descriptions corresponding to the plurality of languages, a request for a natural language description for the remote device in the particular language that is used by the natural language assistant that operates on the computing device, the natural language description for the remote device mapping one or more natural language strings, in the particular language, to one or more commands for the remote device;

obtain, from the remote device, the natural language description in the particular language;

update the natural language assistant based on the one or more natural language strings to enable the natural language assistant to monitor, at the computing device, natural language inputs for the remote device in the particular language that is used by the natural language assistant that operates on the computing device;

receive, from the natural language assistant, an indication that a particular natural language input to the computing device in the particular language matches a particular natural language string of the natural language description;

identify a particular command of the remote device that maps to the particular natural language string;

communicate, to the remote device, the particular command to cause the remote device to perform a functionality corresponding to the particular command.

11. The computing device as recited in claim 10, the one or more natural language strings comprising at least one word and at least one phrase.

12. The computing device as recited in claim 10, the particular natural language input comprising voice input.

13. The computing device as recited in claim 10, the remote device having a different natural language description for each of multiple different grammars, the computer-executable instructions to cause the one or more processing units to request a natural language description for a particular one of the multiple different grammars.

14. The computing device as recited in claim 10, the computer-executable instructions to cause the one or more processing units to receive the natural language description from the remote device in response to the request for the natural language description.

15. A computing device comprising:

a processor; and

a computer-readable storage medium having stored thereon multiple instructions that, responsive to execution by the processor, cause the one or more processors to perform acts comprising;

determining that a plurality of languages that are supported by a remote device includes a particular language that is used by a natural language assistant that operates on the computing device;

sending to the remote device, that maintains natural language descriptions corresponding to the plurality of languages, a request for a natural language description for the remote device in the particular language that is used by the natural language assistant to receive natural language inputs for the remote device at the computing device, the natural language description for the remote device mapping one or more natural language strings, in the particular language, to one or more commands for the remote device;

receiving, from the remote device, the natural language description in the particular language;

updating the natural language assistant based on the natural language description to enable the natural language assistant to monitor the natural language inputs for the remote device in the particular language;

receiving, from the natural language assistant, an indication that a particular natural language input to the computing device in the particular language matches a particular natural language string in the natural language description;

identifying a particular command of the remote device that maps to the particular natural language string; and

communicating the particular command to the remote device.

16. The computing device as recited in claim 15, the one or more natural language strings comprising at least one word and at least one phrase.

17. The computing device as recited in claim 15, the remote device having a different natural language description for each of multiple different grammars, and wherein to send the request is to send a request for a natural language description for a particular one of the multiple different grammars.

18. The computing device as recited in claim 15, wherein the plurality of languages include at least two of an English language, a Chinese language, a Spanish language, and a German language.

19. The computing device as recited in claim 15, wherein the determining that the plurality of languages that are supported by the remote device includes the particular language includes receiving an indication of one or more spoken languages that are supported by the remote device for initiating one or more functionalities at the remote device.

20. The computing device as recited in claim 15, wherein the determining that the plurality of languages that are supported by the remote device includes the particular language includes querying one or more settings of the computing device.