Computer network, computer-implemented method, computer program product, client, and server for natural language-based control of a digital network

• US 9,590,818 B2
• Filed: 03/06/2012
• Issued: 03/07/2017
• Est. Priority Date: 03/07/2011
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: Mammoliti, Fabio, d'Errico, Daniele, Capuozzo, Giuseppe, Borghini, Marco
• Primary Examiner(s): Armstrong, Angela A
• Application Number:
  US13/413,373
• Publication Number:
  US 20120232886A1
• Time in Patent Office: 1,827 Days
• Field of Search: 704/9, 704/270.1
• US Class Current: 1/1
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04L 12/2814 Exchanging control software or macros for controlling appliance services in a home automation network arrangements for maintenance or administration involving configuration of the network and network elements H04L41/08
• CPC Class Codes: H04L 12/2818 from a device located outside both the home and the home network access arrangements H04L12/2856; for remote control or remote monitoring of applications H04L67/025; arrangements for transmitting signals characterised by the use of a wireless eletrical link G08C17/00; telephonic communication systems adapted for combination with remote control systems H04M11/007
• CPC Class Codes: H04L 2012/285 Generic home appliances, e.g. refrigerators

Abstract

The present application relates to a computer network, a computer-implemented method, a computer program product, a client, and a server for natural language-based control of a digital network. In one aspect, the computer network for natural language-based control of a digital network may comprise: a digital network operable to provide sharing of access to a network between a plurality of devices connected in the digital network; a client installed in the digital network and operable to provide a unified natural language interface to a user to control the digital network using natural language; a server connected to the client over the network and operable to process a user request of the user performed through the unified natural language interface; and one or more software agents operable to execute at least one action on at least one of the plurality of devices based on the processed user request.

Claims

1. A method comprising:

providing, by one or more processors of a client device and for display, a unified natural language interface;

receiving, by the one or more processors and using the unified natural language interface, a mode selection comprising one of an active mode or a passive mode;

selectively operating, by the one or more processors, in the one of the active mode or the passive mode based on the mode selection,

the one of the active mode or the passive mode being used to receive an instruction for performing a device action,

the active mode occurring prior to receiving the instruction for performing the device action, and when the mode selection identifies the active mode,

the passive mode occurring prior to receiving the instruction for performing the device action, and when the mode selection identifies the passive mode,

when operated in the active mode, the unified natural language interface being operable to interpret verbal information from a user input, and to control at least one device based on the verbal information interpreted from the user input, and

when operated in the passive mode, the unified natural language interface being operable to continuously listen for a user request, to extract non-verbal information and verbal information from the user request, and to control the at least one device based on the non-verbal information and the verbal information extracted from the user request;

receiving, by the one or more processors, the instruction for performing the device action;

transmitting, by the one or more processors, based on selectively operating in the one of the active mode or the passive mode, and using a network, information based on the mode selection and the instruction for performing the device action to a server device,

the network being operable to provide communication between the client device, the server device, and a plurality of devices,

each of the plurality of devices being capable of performing a particular device action based on receiving a particular instruction from the client device,

the client device processing the mode selection of the active mode by performing a first set of communications,

the server device processing the mode selection of the active mode by performing a second set of communications,

the client device processing the mode selection of the passive mode by performing a third set of communications,

the third set of communications being different than the first set of communications, and

the server device processing the mode selection of the passive mode by performing a fourth set of communications,

the fourth set of communications being different than the second set of communications;

receiving, by the one or more processors and based on transmitting the information regarding the mode selection, response information from the server device; and

instructing, by the one or more processors and based on receiving the response information from the server device, a device, of the plurality of devices, to perform the device action.

2. The method of claim 1, further comprising:

receiving a verbal request from a user;

formatting the verbal request into speech signal information; and

transmitting the speech signal information.

3. The method of claim 1, further comprising:

selecting the device, of the plurality of devices, based on analyzing a list of tags included in the response information.

4. The method of claim 1, further comprising:

providing, based on instructing the device to perform the device action, an audio message indicating a result of the device action.

5. A non-transitory computer-readable medium storing instructions, the instructions comprising:

one or more instructions which, when executed by one or more processors of a client device, cause the one or more processors to;

provide, for display, a unified natural language interface;

receive, using the unified natural language interface, a mode selection comprising of one of an active mode or a passive mode;

selectively operate in the one of the active mode or the passive mode based on the mode selection,

the one of the active mode or the passive mode being used to receive an instruction for performing a device action,

the active mode occurring prior to receipt of the instruction for performing the device action, and when the mode selection identifies the active mode,

the passive mode occurring prior to receipt of the instruction for performing the device action, and when the mode selection identifies the passive mode,

when operated in the active mode, the unified natural language interface being operable to interpret verbal information, from a user input, and to control at least one device based on the verbal information interpreted from the user input, and

when operated in the passive mode, the unified natural language interface being operable to continuously listen for a user request, to extract non-verbal information and verbal information from the user request, and to control the at least one device based on the non-verbal information and the verbal information extracted from the user request;

receive the instruction for performing the device action;

transmit, based on selectively operating in the one of the active mode or the passive mode, and using a network, information based on the mode selection and the instruction for performing the device action to a server device,

the network being operable to provide communication between the client device, the server device, and a plurality of devices,

each of the plurality of devices being capable of performing a particular device action based on receiving a particular instruction from the client device,

the client device processing the mode selection of the active mode by performing a first set of communications,

the server device processing the mode selection of the active mode by performing a second set of communications,

the client device processing the mode selection of the passive mode by performing a third set of communications,

the third set of communications being different than the first set of communications, and

the server device processing the mode selection of the passive mode by performing a fourth set of communications,

the fourth set of communications being different than the second set of communications;

receive, based on transmitting the information regarding the mode selection, response information from the server device; and

instruct, based on receiving the response information from the server device, a device, of the plurality of devices, to perform the device action.

6. The non-transitory computer-readable medium of claim 5, where the instructions further include:

one or more instructions which, when executed by the one or more processors, cause the one or more processors to;

receive a verbal request from a user; and

format the verbal request into speech signal information; and

transmit the speech signal information.

7. The non-transitory computer-readable medium of claim 5, where the instructions further include:

one or more instructions which, when executed by the one or more processors, cause the one or more processors to;

select the device, of the plurality of devices, based on analyzing a list of tags included in the response information.

8. The non-transitory computer-readable medium of claim 5, where the instructions further include:

one or more instructions which, when executed by the one or more processors, cause the one or more processors to;

provide, based on instructing the device to perform the device action, an audio message indicating a result of the device action.

9. A client device comprising:

a memory to store instructions; and

a processor to execute the instructions to;

provide, for display, a unified natural language interface;

receive, using the unified natural language interface, a mode selection comprising one of an active mode or a passive mode;

selectively operate in the one of the active mode or the passive mode based on the mode selection,

the one of the active mode or the passive mode being used to receive an instruction for performing a device action,

the active mode occurring prior to receipt of the instruction for performing the device action, and when the mode selection identifies the active mode,

the passive mode occurring prior to receipt of the instruction for performing the device action, and when the mode selection identifies the passive mode,

when operated in the active mode, the unified natural language interface being operable to interpret verbal information, from a user input, and to control at least one device based on the verbal information interpreted from the user input, and

when operated in the passive mode, the unified natural language interface being operable to continuously listen for a user request, extract non-verbal information and verbal information from the user request, and to control the at least one device based on the non-verbal information and the verbal information extracted from the user request;

receive the instruction for performing the device action;

transmit, based on selectively operating in the one of the active mode or the passive mode, and using a network, information based on the mode selection and the instruction for performing the device action to a server device,

the network being operable to provide communication between the client device, the server device, and a plurality of devices,

each of the plurality of devices being capable of performing a particular device action based on receiving a particular instruction from the client device,

the client device processing the mode selection of the active mode by performing a first set of communications,

the server device processing the mode selection of the active mode by performing a second set of communications,

the client device processing the mode selection of the passive mode by performing a third set of communications,

the third set of communications being different than the first set of communications, and

the server device processing the mode selection of the passive mode by performing a fourth set of communications,

the fourth set of communications being different than the second set of communications;

receive, based on transmitting the information regarding the mode selection, response information from the server device; and

instruct, based on receiving the response information from the server device, a device, of the plurality of devices, to perform the device action.

10. The client device of claim 9, where the processor is further to:

receive a verbal request from a user;

format the verbal request into speech signal information; and

transmit the speech signal information.

11. The client device of claim 9, where the processor is further to:

select the device, of the plurality of devices, based on analyzing a list of tags included in the response information.

12. The client device of claim 9, where the processor is further to:

provide, based on instructing the device to perform the device action, an audio message indicating a result of the device action.

13. A method comprising:

receiving, by a server device, from a client device, and using a network, information based on a mode selection comprising one of an active mode or a passive mode;

the client device selectively operating in the one of the active mode or the passive mode based on the mode selection,

the one of the active mode or the passive mode being used to receive an instruction for performing a device action,

the client device operating in the active mode prior to receiving the instruction for performing the device action, and when the mode selection identifies the active mode,

the client device operating in the passive mode prior to receiving the instruction for performing the device action, and when the mode selection identifies the passive mode,

when operated in the active mode, the client device being operable to interpret verbal information from a user input, and to control at least one device based on the verbal information interpreted from the user input, and

when operated in the passive mode, the client device being operable to continuously listen for a user request, extract non-verbal information and verbal information from the user request, and to control the at least one device based on the non-verbal information and the verbal information extracted from the user request;

receiving, by the server device and based on the client device selectively operating in the one of the active mode or the passive mode, information based on the mode selection and the instruction for performing the device action,

the network being operable to provide communication between the client device, the server device, and a plurality of devices,

each of the plurality of devices being capable of performing a device action based on receiving an instruction from the client device, the client device processing the mode selection of the active mode by performing

a first set of communications,

processing, by the server device, the mode selection of the active mode by performing a second set of communications,

the client device processing the mode selection of the passive mode by performing a third set of communications,

the third set of communications being different than the first set of communications,

processing, by the server device, the mode selection of the passive mode by performing a fourth set of communications,

the fourth set of communications being different than the second set of communications; and

transmitting, by the server device and based on receiving the information regarding the mode selection and the instruction for performing the device action, response information to the client device,

the response information being used by the client device to instruct a device, of the plurality of devices, to perform the device action.

14. The method of claim 13, further comprising:

identifying, based on receiving the information based on the mode selection and the instruction for performing the device action, user information,

the user information being identified based on speech information associated with the instruction for performing the device action.

15. The method of claim 14, further comprising:

generating, based on the speech information, text information;

generating, based on the text information, one or more tags; and

transmitting the one or more tags to the client device.

16. The method of claim 15, further comprising:

using sematic information to parse the instruction for performing the device action into one or more text strings; and

generating the one or more tags based on the one or more text strings.

17. A server device comprising:

a memory to store instructions; and

a processor to execute the instructions to;

receive, from a client device, and using a network, information based on a mode selection comprising one of an active mode or a passive mode;

the client device selectively operating in the one of the active mode or the passive mode based on the mode selection,

the one of the active mode or the passive mode being used to receive an instruction for performing a device action,

the client device operating in the active mode prior to receipt of the instruction for performing the device action, and when the mode selection identifies the active mode,

the client device operating in the passive mode prior to receipt of the instruction for performing the device action, and when the mode selection identifies the passive mode,

when operated in the active mode, the client device being operable to interpret verbal information from a user input, and to control at least one device based on the verbal information interpreted from the user input, and

when operated in the passive mode, the client device being operable to continuously listen for a user request, to extract non-verbal information and verbal information from the user request, and to control the at least one device based on the non-verbal information and the verbal information extracted from the user request;

receive, based on the client device selectively operating in the one of the active mode or the passive mode, and using a network, information based on the mode selection and the instruction for performing the device action,

the network being operable to provide communication between the client device, the server device, and a plurality of devices,

each of the plurality of devices being capable of performing a device action based on receiving an instruction from the client device,

the client device processing the mode selection of the active mode by performing a first set of communications,

process the mode selection of the active mode by performing a second set of communications,

the client device processing the mode selection of the passive mode by performing a third set of communications,

the third set of communications being different than the first set of communications, and

process the mode selection of the passive mode by performing a fourth set of communications,

the fourth set of communications being different than the second set of communications; and

transmit, based on receiving the information regarding the mode selection and the instruction for performing the device action, response information to the client device,

the response information being used by the client device to instruct a device, of the plurality of devices, to perform the device action.

18. The device of claim 17, where the processor is further to:

identify, based on receiving the information based on the mode selection and the instruction for performing the device action, user information,

the user information being identified based on speech information associated with the instruction for performing the device action.

19. The device of claim 18, where the processor is further to:

generate, based on the speech information, text information;

generate, based on the text information, one or more tags; and

transmit the one or more tags to the client device.

20. The device of claim 19, where the processor is further to:

use sematic information to parse the instruction for performing the device action into one or more text strings; and

generate the one or more tags based on the one or more text strings.