Virtual assistant system

• US 10,354,014 B2
• Filed: 06/25/2018
• Issued: 07/16/2019
• Est. Priority Date: 01/30/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sharma, Vishal, Amjadi, Elhum
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US16/016,654
• Publication Number:
  US 20180307682A1
• Time in Patent Office: 386 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06Q 10/02 Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06Q 50/12 Hotels or restaurants
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 63/06 for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/101 Access control lists [ACL]
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/63 Routing a service request depending on the request content or context
• CPC Class Codes: H04W 12/082 using revocation of authorisation
• CPC Class Codes: H04W 12/088 using filters or firewalls
• CPC Class Codes: H04W 4/20 : Services signaling; Auxiliary data signalling, i.e. transmitting data via a non-traffic channel
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

A mobile device to operate a virtual assistant system to control any wirelessly connected devices and services by receiving an input command to perform a function at the devices and services, translate the input command into a common language sharable between the external devices and services so that such input commands can be understood between the plurality of devices and services, and functions can be performed at the devices and or service in response to the input command. This control can be shared with other mobile devices as instructed by an input command.

Claims

1. An apparatus comprising:

a communicator configured to receive a user input command formatted according to a semantic information interchange language;

a storage system mapping the received user input command to at least one predefined semantic command, the at least one predefined semantic command comprising a manual gesture;

a first plugin configured to issue a first external command to a first external device or service based on the at least one predefined semantic command; and

a second plugin configured to issue a second external command to a second external device or service based on the at least one predefined semantic command, the second external command being distinct from the first external command.

2. The apparatus of claim 1, the at least one predefined semantic command further comprising an icon touch or voice data.

3. The apparatus of claim 1, at least one of the first and second external commands comprising a control command for a security system, lighting, a thermostat, an automotive device, a wearable device, or an online service.

4. The apparatus of claim 1, the manual gesture being captured by a camera, motion detector, proximity detector, infrared sensor, temperature sensor, global positioning device, or wearable device.

5. The apparatus of claim 1, the at least one predefined semantic command comprising first and second semantic commands, the first plugin configured to issue primary and secondary first external commands to the first external device or service responsive to the first and second semantic commands, respectively.

6. The apparatus of claim 5, the first external device or service comprising an electronic lock system, the first semantic command comprising closing a fist, the second semantic command comprising closing two fists, the primary first external command comprising locking a door, the secondary first external command comprising engaging a deadbolt.

7. The apparatus of claim 1, the first external device or service comprising a face recognition program, the at least one predefined semantic command comprising a waving gesture, the first external command comprising identifying a person in a picture.

8. The apparatus of claim 1, further comprising an assistant interface configured to translate the received user input command into the at least one predefined semantic command, and a sensor comprising at least one of a camera, motion detector, proximity detector, infrared sensor, temperature sensor, global positioning device, or wearable device for receiving the manual gesture as the user input command.

9. The apparatus of claim 1, the communicator comprising a web communications system for transmitting and receiving information wirelessly over the World Wide Web.

10. The apparatus of claim 1, the first plugin further configured to issue distinct first external commands based on context information analyzed by the assistant server.

11. The apparatus of claim 10, said context information comprising at least one of a location of a mobile device at which the user input command was received, a time of day when the user input command was received, co-presence of specific individuals, involuntary actions by a user, and action sequences taken by a user of the mobile device.

12. A method comprising:

receiving a user input command formatted according to a semantic information interchange language;

mapping the received user input command to at least one predefined semantic command, the at least one predefined semantic command comprising a manual gesture;

issuing a first external command to a first external device or service based on the at least one predefined semantic command; and

issuing a second external command to a second external device or service based on the at least one predefined semantic command, the second external command being distinct from the first external command.

13. The method of claim 12, the at least one predefined semantic command further comprising an icon touch or voice data.

14. The method of claim 12, at least one of the first and second external commands comprising a control command for a security system, lighting, a thermostat, an automotive device, a wearable device, or an online service.

15. The method of claim 12, the manual gesture being captured by a camera, motion detector, proximity detector, infrared sensor, temperature sensor, global positioning device, or wearable device.

16. The method of claim 12, the at least one predefined semantic command comprising first and second semantic commands, the issuing the first external command comprising issuing primary and secondary first external commands to the first external device or service responsive to the first and second semantic commands, respectively.

17. The method of claim 16, the first external device or service comprising an electronic lock system, the first semantic command comprising closing a fist, the second semantic command comprising closing two fists, the primary first external command comprising locking a door, the secondary first external command comprising engaging a deadbolt.

18. The method of claim 12, the first external device or service comprising a face recognition program, the at least one predefined semantic command comprising a waving gesture, the first external command comprising identifying a person in a picture.

19. The method of claim 12, the receiving comprising receiving the user input command wirelessly over the World Wide Web.

20. An apparatus comprising:

a processor;

a memory storing instructions operable to cause the processor to;

receive a user input command formatted according to a semantic information interchange language;

map the received user input command to at least one predefined semantic command, the at least one predefined semantic command comprising a manual gesture;

issue a first external command to a first external device or service based on the at least one predefined semantic command; and

issue a second external command to a second external device or service based on the at least one predefined semantic command, the second external command being distinct from the first external command.