VIRTUAL ASSISTANT SYSTEM TO REMOTELY CONTROL EXTERNAL SERVICES AND SELECTIVELY SHARE CONTROL

• US 20150213355A1
• Filed: 01/29/2015
• Published: 07/30/2015
• Est. Priority Date: 01/30/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Elhum Amjadi, Vishal Sharma
• Inventors: Sharma, Vishal, Amjadi, Elhum
• Granted Patent: US 9,111,214 B1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06Q 10/02 Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06Q 50/12 Hotels or restaurants
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 63/06 for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/101 Access control lists [ACL]
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/63 Routing a service request depending on the request content or context
• CPC Class Codes: H04W 12/082 using revocation of authorisation
• CPC Class Codes: H04W 12/088 using filters or firewalls
• CPC Class Codes: H04W 4/20 : Services signaling; Auxiliary data signalling, i.e. transmitting data via a non-traffic channel
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

A virtual assistant system includes a mobile device, an assistant server and devices and services included within the virtual assistant system desired to be controlled. The virtual assistant system can control any such devices and services by receiving an input to perform a function at the devices and services, translate the input into a language sharable by all of the external devices and services through a respective plugin so that such inputs can be understood between the plurality of devices and services, and functions can be performed at the desired device(s) or service(s) either remotely or locally. This control is sharable with other users as instructed.

Claims

1. A virtual assistant system to interactively communicate information between devices and services, the virtual assistant system comprising:

a mobile device to receive an input command corresponding to a function to be performed at an external device or service, to translate the input command into a semantic information interchange format language in which different devices and services using respective proprietary languages can communicate therebetween to share information, and to transmit the translated input command;

at least one external device or service to perform the corresponding function; and

an assistant server to receive the translated input command from the mobile device, the assistant server including a plugin corresponding to each of the at least one external device or service, the plugins being configured to convert the received input command into a proprietary language command recognizable by the corresponding at least one device or service to perform the associated function.

2. The virtual assistant system of claim 1, wherein the assistant server analyzes context information of the input command before converting the input command to accurately determine the intended function to be performed by the input command.

3. The virtual assistant system of claim 2, wherein the context information analyzed by the assistant server includes at least one of a location of the mobile device when the command was input, a time of day when the command was input, co-presence of specific individuals, involuntary actions by a user, and action sequences taken by a user of the mobile device.

4. The virtual assistant system of claim 3, wherein the action sequences taken by the user include a plurality of input commands made simultaneously or sequentially.

5. The virtual assistant system of claim 3, wherein the action sequences taken by the user include an input command that is invalidated and followed by another input command.

6. The virtual assistant system of claim 2, wherein when the assistant server determines that there exists more than one possible intended function to be performed at a device or service by the input command, the assistant server:

clarifies which function is intended to be performed by forwarding the determined possible intended options back to the mobile device to select the intended command; and

gathers additional context information associated with the selection of the intended command input at the mobile device to update a stored library of recognizable input commands.

7. The virtual assistant system of claim 2, wherein when a first form of an input command is not recognized at the assistant server and then a follow-up second form of the input command is input, the assistant server creates and stores a new command in the first form of the input which is recognizable to perform the intended function at the external device or service.

8. The virtual assistant system of claim 1, wherein when the assistant server recognizes a repeated cluster of input commands, the assistant server creates a new consolidated input command to represent performing the cluster of input commands simultaneously and presents the new consolidated input command to the mobile device for future use by a user.

9. The virtual assistant system of claim 8, wherein the repeated cluster of input commands and the new consolidated input command are gesture-type input commands.

10. The virtual assistant system of claim 2, wherein the context information analyzed by the assistant server includes data to identify at least one specific user of the mobile device and the assistant server converts the received input command according to the at least one specific identified user.

11. The virtual assistant system of claim 1, wherein:

the assistant server stores credentials corresponding to a user of the mobile device, the credentials including data to authorize the user to control the at least one external device or service through the input commands.

12. The virtual assistant system of claim 11, wherein an input command at the mobile device authorizes at least one other user of another respective mobile device to control the at least one external device or service through input commands.

13. The virtual assistant system of claim 12, wherein another input command at the mobile device revokes the authorization of the at least one other user of the another respective mobile device to control the at least one external device or service.

14. The virtual assistant system of claim 13, wherein revocation of authorization of the at least one other user includes:

displaying a history of users provided with the authorization at the mobile device; and

selecting among the history of users displayed in which access is to be revoked.

15. The virtual assistant system of claim 12, wherein the input command to authorize the at least one other user to control the at least one external device or service includes information to limit the authorization to at least one of a level of control, a location in which control is permitted, a time frame in which the control is permitted and an access path in which the other user approaches the external device or service.

16. The virtual assistant system of claim 15, wherein the location in which control is permitted to at least one other user is a security protected building or room.

17. The virtual assistant system of claim 15, wherein the access path in which control is permitted to at least one other user is a predetermined secured entryway among plural secured entryways into a security protected building or room.

18. The virtual assistant system of claim 12, wherein the authorization of the at least one other user is provided through the stored credentials of the authorizing user so that the authorization and revocation of authorization remains under the control of the user with the stored credentials.

19. The virtual assistant system of claim 15, wherein the authorized other user is prevented from entering other credentials and from obtaining access to the authorizing user'"'"'s credentials.

20. The virtual assistant system of claim 12, wherein the assistant server:

determines context information regarding providing authorization by the mobile device; and

automatically authorizes the at least one other user of the another mobile device to control the at least one external device or service through input commands according to the stored credentials, the authorization being based on the determined context information.

21. The virtual assistant system of claim 20, wherein the determined context information identifies at least one specific user of the other mobile device.

22. The virtual assistant system of claim 20, wherein the determined context information includes data regarding a historic time range in which the user of the mobile device has authorized other users to control the at least one external device or service.

23. The virtual assistant system of claim 11, wherein the assistant server connects to a plurality of external devices or services corresponding to the stored credentials of the user.

24. The virtual assistant system of claim 23, wherein the assistant server connects to the plurality of external devices or services by presenting the stored credentials to the respective external devices or services and determining whether further authentication is necessary to input commands to the plurality of external devices or services.

25. An assistant server, comprising:

a storage system to store a library of commands to perform functions at a plurality of external devices and services, the commands being provided in a semantic information interchange language format that can be shared between the plurality of external devices and services through the assistant server;

a plugin corresponding to each of the plurality of external devices and services, each plugin being configured to translate between the semantic information interchange language used in the assistant server and a respective proprietary language of the corresponding device or service; and

a platform configured to wirelessly share the library of semantic information interchange language format commands between the plurality of external devices and services by using the proprietary language translated at the respective plugins and to wirelessly share the library of semantic information interchange language format commands with a mobile device in which commands to perform a function are initially input and transmitted to the platform in the semantic information interchange language format.

26. The assistant server of claim 25, wherein the storage system stores at least one command set corresponding to a respective one of the plurality of external devices and services, each command set including semantic information interchange language format commands to perform corresponding functions at the respective external device or service.

27. The assistant server of claim 25, wherein each plugin is created at the assistant server to include a set of commands selected from the stored library of commands, the set of commands to correspond with associated functions to be performed at the corresponding device or service.

28. The assistant server of claim 25, wherein the platform is configured to wirelessly connect to a plurality of mobile devices to share information therebetween and to receive input commands therefrom, the information being shared and input commands being received are provided in the semantic information interchange language format translated at the mobile device.

29. A virtual assistant system, comprising:

a mobile device to transmit an input command corresponding to a function to be performed at a predetermined external device or service, the input command to first be translated into a semantic information interchange format language that can be communicated with and between a plurality of external devices and services wirelessly connected within the virtual assistant system before being transmitted; and

an assistant server to receive the transmitted input command from the mobile device and to determine whether the function to be performed at the predetermined external device or service will be controlled by the mobile device or the assistant server, the determination being based on at least one of a location of the mobile device with respect to the predetermined external device or service and a network communication established between the mobile device and the external device or service, the assistant server including a plugin for each external device or service wirelessly connected to the virtual assistant system, each plugin being configured to attach proprietary language of the corresponding external device or service to the received input command required to perform the associated function at either the mobile device or the assistant server according to the control determination.

30. The virtual assistant system of claim 29, wherein each plugin attaches the proprietary language of the corresponding external device or service to the received input command by embedding the proprietary language therein to enable communication with the external device or service to be controlled by the assistant server or to be transmitted back to the mobile device to enable the mobile device to directly control the external device or service.