Internet of things (IoT) human interface apparatus, system, and method

• US 10,832,665 B2
• Filed: 12/06/2016
• Issued: 11/10/2020
• Est. Priority Date: 05/27/2016
• Status: Active Grant

• Current Assignee: CenturyLink Intellectual Property LLC (Lumen Technologies, Inc.)
• Original Assignee: CenturyLink Intellectual Property LLC (Lumen Technologies, Inc.)
• Inventors: Barnett, Jr., Thomas C.
• Primary Examiner(s): Washburn, Daniel C
• Assistant Examiner(s): Ogunbiyi, Oluwadamilola M
• Application Number:
  US15/370,764
• Publication Number:
  US 20170345420A1
• Time in Patent Office: 1,435 Days
• Field of Search: 704257, 704E1504
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

Novel tools and techniques are provided for implementing Internet of Things (“IoT”) functionality. In some embodiments, microphones of an IoT human interface device might receive user voice input. The IoT human interface device and/or a computing system might identify explicit commands in the voice input, identify first IoT-capable devices to which the explicit commands are applicable, receive sensor data from IoT sensors, and analyze the voice input in view of previous user voice inputs and in view of the sensor data to determine whether the voice input contains any implicit commands. If so, second IoT-capable devices to which an implicit command is additionally applicable might be identified, instructions based on a combination of the explicit and implicit commands may be generated and sent to the second IoT-capable devices. Instructions based only on the explicit commands are generated and sent to first IoT-capable devices to which implicit commands are not applicable.

Claims

1. A method, comprising:

receiving, with each of a plurality of microphones of an Internet of Things (“

IoT”

) human interface device, a first voice input from a user, the plurality of microphones being disposed within a plane that is parallel with a cylindrical base of the IoT human interface device;

identifying, with a computing system, one or more explicit commands in the first voice input;

identifying, with the computing system, one or more first IoT-capable devices of a plurality of IoT-capable devices to which the one or more explicit commands are applicable;

receiving, with the computing system, one or more first sensor data from each of at least one first sensor;

analyzing, with the computing system, the first voice input in view of previous voice inputs from the user and in view of the one or more first sensor data, to determine whether the first voice input additionally contains any implicit commands;

based on a determination that the first voice input contains at least one implicit command, identifying, with the computing system, one or more second IoT-capable devices of the one or more first IoT-capable devices to which the at least one implicit command is additionally applicable, generating, with the computing system, second instructions for each of the one or more second IoT-capable devices, using a combination of the one or more explicit commands and the at least one implicit command, and sending, with the computing system, the generated second instructions to the one or more second IoT-capable devices; and

for each of the one or more first IoT-capable devices to which the at least one implicit command is not applicable, generating, with the computing system, first instructions, using the one or more explicit commands, and sending, with the computing system, the generated first instructions to the one or more first IoT-capable devices to which the at least one implicit command is not applicable;

wherein analyzing the first voice input to determine whether the first voice input additionally contains any implicit commands comprises;

determining, with the computing system, an intent of the user based at least in part on an identity of the user inputting the first voice input, based at least in part on the previous voice inputs received from the user identified by the computing system and the one or more first sensor data, and based at least in part on a detected mood of the user;

based on the detected mood of the user, determining, with the computing system, whether the at least one implicit command comprises a warning command to warn one or more persons different from the user of the detected mood of the user; and

determining, with the computing system, a match probability that indicates a probability that the determined intent of the user matches actual intent of the user;

wherein identifying, with the computing system, the one or more second IoT-capable devices of the one or more first IoT-capable devices to which the at least one implicit command is additionally applicable comprises;

based on a determination that the at least one implicit command comprises the warning command to warn the one or more persons of the detected mood of the user, identifying, with the computing system, the one or more second IoT-capable devices of the one or more first IoT-capable devices to use to warn the one or more persons of the detected mood of the user;

wherein the second instructions comprise a data structure that comprises the determined intent of the user, the determined match probability, and a tagged list listing at least the one or more second IoT-capable devices.

2. The method of claim 1, further comprising:

identifying, with the computing system, one or more third IoT-capable devices of the plurality of IoT-capable devices to which the at least one implicit command is applicable, the one or more third IoT-capable devices being separate from each of the one or more first IoT-capable devices and the one or more second IoT-capable devices;

generating, with the computing system, third instructions for each of the one or more third IoT-capable devices, using the at least one implicit command; and

sending, with the computing system, the generated third instructions to the one or more third IoT-capable devices.

3. The method of claim 1, wherein analyzing, with the computing system, the first voice input comprises analyzing, with the computing system, the first voice input utilizing artificial intelligence (“

AI”

) to improve interactions with the user.

4. The method of claim 3, wherein the AI is further utilized to improve machine-to-machine interactions and to improve utilization of the at least one first sensor and the plurality of IoT-capable devices.

5. The method of claim 1, wherein the plurality of IoT-capable devices are separate from the IoT human interface device, wherein the plurality of IoT-capable devices comprises at least one of one or more sensor devices, one or more household appliances, one or more kitchen appliances, one or more lighting systems, one or more automated door locking systems, one or more automated door opening or closing systems, one or more automated window locking systems, one or more automated window opening or closing systems, one or more smart windows, one or more window covering control systems, one or more solar cells, one or more customer premises security systems, one or more customer premises environmental control systems, one or more electrical outlets, one or more power strips, one or more dimmer switches, one or more data ports, one or more display devices, one or more clocks, one or more sprinkler systems, one or more vehicles, one or more mobile user devices, one or more media recording or playback devices, one or more medical devices, one or more fitness trackers, or one or more exercise equipment.

6. The method of claim 5, wherein the one or more sensor devices comprises the at least one first sensor, and wherein the one or more sensor devices comprises at least one of one or more temperature sensors, one or more light sensors, one or more humidity sensors, one or more motion sensors, one or more air quality sensors, one or more carbon monoxide sensors, one or more smoke detectors, one or more water leak detectors, one or more contact sensors, one or more audio sensors, one or more accelerometers, one or more proximity sensors, one or more biometrics sensors, one or more location sensors, one or more radiation sensors, one or more telecommunications signal sensors, or one or more cameras.

7. The method of claim 1, wherein the IoT human interface device comprises the at least one first sensor, and wherein the at least one first sensor comprises at least one of one or more temperature sensors, one or more light sensors, one or more humidity sensors, or one or more motion sensors.

8. The method of claim 1, wherein the computing system comprises one of a single processor disposed within the IoT human interface device, a plurality of processors disposed within the IoT human interface device, a server computer remote from the IoT human interface device, a cloud computing system, a distributed computing system that integrates computing resources from two or more IoT-capable devices of the plurality of IoT-capable devices, or a combination of two or more of these computing systems.

9. The method of claim 1, wherein the plurality of IoT-capable devices are separate from the IoT human interface device, wherein receiving, with the computing system, one or more first sensor data from each of at least one first sensor comprises receiving, with the computing system, one or more first sensor data from each of at least one first sensor, via one or more application programming interfaces (“

APIs”

) established between the IoT human interface device and each of the at least one first sensor.

10. The method of claim 1, wherein receiving, with each of the plurality of microphones of the IoT human interface device, the first voice input from the user comprises utilizing at least one of far-field voice recognition, voice distinction, or a beam forming microphone array of the plurality of microphones.

11. The method of claim 1, wherein each IoT-capable device of the plurality of IoT-capable devices is assigned a unique IPv6 identifier.

12. The method of claim 1, wherein a same first voice input is received by the computing system from at least two different users, and wherein the computing system determines the implicit commands contained within the same first voice input are different based at least in part on the identity of the user inputting the first voice input and based at least in part on the previous voice inputs received from the user identified by the computing system.

13. The method of claim 1, wherein the warning command causes at least one of locking a door lock, illuminating a do-not-disturb sign, or sending a communication warning the one or more persons to one or more fourth IoT-capable devices associated with the one or more persons.

14. A system, comprising:

at least one first sensor;

an Internet of Things (“

IoT”

) human interface device, comprising;

a cylindrical base;

a plurality of microphones that are disposed within a plane that is parallel with the cylindrical base;

one or more speakers;

at least one first processor;

a data store; and

a non-transitory computer readable medium in communication with each of the at least one first processor and the data store, the non-transitory computer readable medium having stored thereon computer software comprising a first set of instructions that, when executed by the at least one first processor, causes the IoT human interface device to;

receive, with each of the plurality of microphones, a first voice input from a user; and

send the first voice input to a computing system;

the computing system, comprising;

at least one second processor; and

a non-transitory computer readable medium in communication with the at least one second processor, the non-transitory computer readable medium having stored thereon computer software comprising a second set of instructions that, when executed by the at least one second processor, causes the computing system to;

receive the first voice input from the IoT human interface device;

identify one or more explicit commands in the first voice input;

identify one or more first IoT-capable devices of a plurality of IoT-capable devices to which the one or more explicit commands are applicable;

receive one or more first sensor data from each of the at least one first sensor;

analyze the first voice input in view of previous voice inputs from the user and in view of the one or more first sensor data, to determine whether the first voice input additionally contains any implicit commands;

based on a determination that the first voice input contains at least one implicit command, identify one or more second IoT-capable devices of the one or more first IoT-capable devices to which the at least one implicit command is additionally applicable, generate second instructions for each of the one or more second IoT-capable devices, using a combination of the one or more explicit commands and the at least one implicit command, and send the generated second instructions to the one or more second IoT-capable devices; and

for each of the one or more first IoT-capable devices to which the at least one implicit command is not applicable, generate first instructions, using the one or more explicit commands, and send the generated first instructions to the one or more first IoT-capable devices to which the at least one implicit command is not applicable;

wherein analyzing the first voice input to determine whether the first voice input additionally contains any implicit commands comprises;

determining an intent of the user based at least in part on an identity of the user inputting the first voice input, based at least in part on the previous voice inputs received from the user identified by the computing system and the one or more first sensor data, and based at least in part on a detected mood of the user;

based on the detected mood of the user, determining, with the computing system, whether the at least one implicit command comprises a warning command to warn one or more persons different from the user of the detected mood of the user; and

determining a match probability that indicates a probability that the determined intent of the user matches actual intent of the user;

wherein identifying the one or more second IoT-capable devices of the one or more first IoT-capable devices to which the at least one implicit command is additionally applicable comprises;

based on a determination that the at least one implicit command comprises the warning command to warn the one or more persons of the detected mood of the user, identifying the one or more second IoT-capable devices of the one or more first IoT-capable devices to use to warn the one or more persons of the detected mood of the user;

wherein the second instructions comprise a data structure that comprises the determined intent of the user, the determined match probability, and a tagged list listing at least the one or more second IoT-capable devices; and

the plurality of IoT-capable devices, each of the one or more IoT-capable devices of the plurality of IoT-capable devices comprising;

at least one third processor; and

a non-transitory computer readable medium in communication with the at least one third processor, the non-transitory computer readable medium having stored thereon computer software comprising a third set of instructions that, when executed by the at least one third processor, causes the IoT-capable device to;

receive the generated first or second instructions;

perform one or more tasks in accordance with the received first or second instructions; and

update the computing system with progress of each of the one or more tasks.

15. The system of claim 14, wherein the second set of instructions, when executed by the at least one second processor, further causes the computing system to:

identify one or more third IoT-capable devices of the plurality of IoT-capable devices to which the at least one implicit command is applicable, the one or more third IoT-capable devices being separate from each of the one or more first IoT-capable devices and the one or more second IoT-capable devices;

generate third instructions for each of the one or more third IoT-capable devices, using the at least one implicit command; and

send the generated third instructions to the one or more third IoT-capable devices.

16. The system of claim 14, wherein each microphone of the plurality of microphones is disposed equidistant from adjacent microphones and is disposed at a predetermined radius from a vertical axis that is orthogonal to a center of the cylindrical base of the IoT human interface device.

17. The system of claim 14, wherein the plurality of IoT-capable devices are separate from the IoT human interface device, wherein the plurality of IoT-capable devices comprises at least one of one or more sensor devices, one or more household appliances, one or more kitchen appliances, one or more lighting systems, one or more automated door locking systems, one or more automated door opening or closing systems, one or more automated window locking systems, one or more automated window opening or closing systems, one or more smart windows, one or more window covering control systems, one or more solar cells, one or more customer premises security systems, one or more customer premises environmental control systems, one or more electrical outlets, one or more power strips, one or more dimmer switches, one or more data ports, one or more display devices, one or more clocks, one or more sprinkler systems, one or more vehicles, one or more mobile user devices, one or more media recording or playback devices, one or more medical devices, one or more fitness trackers, one or more exercise equipment, or the IoT human interface device.

18. The system of claim 17, wherein the one or more sensor devices comprises the at least one first sensor, and wherein the one or more sensor devices comprises at least one of one or more temperature sensors, one or more light sensors, one or more humidity sensors, one or more motion sensors, one or more air quality sensors, one or more carbon monoxide sensors, one or more smoke detectors, one or more water leak detectors, one or more contact sensors, one or more audio sensors, one or more accelerometers, one or more proximity sensors, one or more biometrics sensors, one or more location sensors, one or more radiation sensors, one or more telecommunications signal sensors, or one or more cameras.

19. The system of claim 14, wherein the IoT human interface device comprises the at least one first sensor, and wherein the at least one first sensor comprises at least one of one or more temperature sensors, one or more light sensors, one or more humidity sensors, or one or more motion sensors.

20. The system of claim 19, wherein the at least one first sensor further comprises at least one of one or more air quality sensors, one or more carbon monoxide sensors, one or more smoke detectors, one or more water leak detectors, one or more contact sensors, one or more audio sensors, one or more proximity sensors, one or more biometrics sensors, one or more location sensors, one or more radiation sensors, one or more telecommunications signal sensors, or one or more cameras.

21. The system of claim 14, wherein the computing system comprises one of a single processor disposed within the IoT human interface device, a plurality of processors disposed within the IoT human interface device, a server computer remote from the IoT human interface device, a cloud computing system, a distributed computing system that integrates computing resources from two or more IoT-capable devices of the plurality of IoT-capable devices, or a combination of two or more of these computing systems.

22. The system of claim 14, wherein receiving one or more first sensor data from each of at least one first sensor comprises receiving one or more first sensor data from each of at least one first sensor, via one or more application programming interfaces (“

APIs”

) established between the IoT human interface device and each of the at least one first sensor.

23. The system of claim 14, wherein each IoT-capable device of the plurality of IoT-capable devices is assigned a unique IPv6 identifier.

24. An Internet of Things (“

IoT”

) human interface device, comprising;

a cylindrical base;

a plurality of microphones that are disposed within a plane that is parallel with the cylindrical base;

one or more speakers;

one or more communications transceivers;

at least one processor;

a data store; and

a non-transitory computer readable medium in communication with each of the at least one processor and the data store, the non-transitory computer readable medium having stored thereon computer software comprising a set of instructions that, when executed by the at least one processor, causes the IoT human interface device to;

receive, with each of the plurality of microphones, a first voice input from a user;

identify one or more explicit commands in the first voice input;

identify one or more first IoT-capable devices of a plurality of IoT-capable devices to which the one or more explicit commands are applicable;

receive one or more first sensor data from each of the at least one first sensor;

analyze the first voice input in view of previous voice inputs from the user and in view of the one or more first sensor data, to determine whether the first voice input additionally contains any implicit commands;

based on a determination that the first voice input contains at least one implicit command, identify one or more second IoT-capable devices of the one or more first IoT-capable devices to which the at least one implicit command is additionally applicable, generate second instructions for each of the one or more second IoT-capable devices, using a combination of the one or more explicit commands and the at least one implicit command, and send the generated second instructions to the one or more second IoT-capable devices; and

for each of the one or more first IoT-capable devices to which the at least one implicit command is not applicable, generate first instructions, using the one or more explicit commands, and send the generated first instructions to the one or more first IoT-capable devices to which the at least one implicit command is not applicable;

wherein analyzing the first voice input to determine whether the first voice input additionally contains any implicit commands comprises;

determining an intent of the user based at least in part on an identity of the user inputting the first voice input, based at least in part on the previous voice inputs received from the user identified by the computing system and the one or more first sensor data, and based at least in part on a detected mood of the user;

based on the detected mood of the user, determining whether the at least one implicit command comprises a warning command to warn one or more persons different from the user of the detected mood of the user; and

determining a match probability that indicates a probability that the determined intent of the user matches actual intent of the user;

wherein identifying the one or more second IoT-capable devices of the one or more first IoT-capable devices to which the at least one implicit command is additionally applicable comprises;

based on a determination that the at least one implicit command comprises the warning command to warn the one or more persons of the detected mood of the user, identifying the one or more second IoT-capable devices of the one or more first IoT-capable devices to use to warn the one or more persons of the detected mood of the user;

wherein the second instructions comprise a data structure that comprises the determined intent of the user, the determined match probability, and a tagged list listing at least the one or more second IoT-capable devices.

25. The IoT human interface device of claim 24, wherein the set of instructions, when executed by the at least one processor, further causes the IoT human interface device to:

identify one or more third IoT-capable devices of the plurality of IoT-capable devices to which the at least one implicit command is applicable, the one or more third IoT-capable devices being separate from each of the one or more first IoT-capable devices and the one or more second IoT-capable devices;

generate third instructions for each of the one or more third IoT-capable devices, using the at least one implicit command; and

send the generated third instructions to the one or more third IoT-capable devices.

26. The IoT human interface device of claim 24, wherein each microphone of the plurality of microphones is disposed equidistant from adjacent microphones and is disposed at a predetermined radius from a vertical axis that is orthogonal to a center of the cylindrical base of the IoT human interface device.

27. The IoT human interface device of claim 24, wherein analyzing the first voice input comprises analyzing the first voice input utilizing artificial intelligence (“

AI”

) to improve interactions with the user.

28. The IoT human interface device of claim 27, wherein the AI is further utilized to improve machine-to-machine interactions between the IoT human interface device and each of the plurality of IoT-capable devices, and to improve utilization of the at least one first sensor and the plurality of IoT-capable devices.

29. The IoT human interface device of claim 24, further comprising:

the at least one first sensor, wherein the at least one first sensor comprises at least one of one or more temperature sensors, one or more light sensors, one or more humidity sensors, or one or more motion sensors.

30. The IoT human interface device of claim 29, wherein the at least one first sensor further comprises at least one of one or more air quality sensors, one or more carbon monoxide sensors, one or more smoke detectors, one or more water leak detectors, one or more contact sensors, one or more audio sensors, one or more proximity sensors, one or more biometrics sensors, one or more location sensors, one or more radiation sensors, one or more telecommunications signal sensors, or one or more cameras.