3D mapping of internet of things devices

• US 10,091,015 B2
• Filed: 12/16/2014
• Issued: 10/02/2018
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kennedy, Kevin Alan
• Primary Examiner(s): Chea, Philip
• Assistant Examiner(s): Ma, Wing
• Application Number:
  US14/572,301
• Publication Number:
  US 20160173293A1
• Time in Patent Office: 1,386 Days
• Field of Search: 709222
• CPC Class Codes: H04L 12/2814 Exchanging control software or macros for controlling appliance services in a home automation network arrangements for maintenance or administration involving configuration of the network and network elements H04L41/08
• CPC Class Codes: H04L 41/0806 for initial configuration or provisioning, e.g. plug-and-play
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

A method for mapping and controlling network-enabled appliances with a control device may comprise collecting spatial information in three dimensions as at least one sensor of the control device is moved within an environment, receiving registration signals from network-enabled appliances in the environment, and associating each network-enabled appliance with a respective 3D position in the environment based on the spatial information. The method may include generating a digital map of the environment from the spatial information and placing a representation of each network-enabled appliance on the digital map based on the respective 3D position. The method may also include generating a spatially aware rule set that applies to one or more rooms within the environment. The method may also include detecting a trigger event, and in response to the trigger event, commanding the network-enabled appliances according to the spatially aware rule set.

Claims

1. A method for mapping and controlling network-enabled appliances with a control device, the method comprising:

collecting, by at least one sensor of the control device, spatial information in three dimensions as the at least one sensor of the control device is moved within an environment containing network-enabled appliances, the at least one sensor including at least one camera;

recognizing at least one of the network-enabled appliances within the environment with machine vision techniques including applying text recognition algorithms to read physical tags to identify and recognize network enabled appliances in image data captured by the at least one camera;

receiving registration signals from at least one of the network-enabled appliances in the environment;

associating each network-enabled appliance with a respective three-dimensional (3D) position in the environment based on the spatial information;

registering each network-enabled appliance with the control device to create a registration log;

generating a digital map of the environment from the spatial information;

identifying rooms within the environment;

placing a representation of each network-enabled appliance on the digital map based on the respective 3D position;

generating a spatially aware rule set that applies to one or more of the rooms within the environment based upon the spatial information and the registration log;

detecting a trigger event; and

in response to the trigger event, commanding the network-enabled appliances within the room or rooms according to the spatially aware rule set.

2. The method of claim 1, wherein identifying the rooms comprises using object recognition to compare the spatial information to a typical room arrangement.

3. The method of claim 1,

wherein identifying the rooms comprises receiving user input in the form of a spoken room name, a typed room name, or a manually selected room name, while collecting the spatial information.

4. The method of claim 1, wherein the at least one camera comprises at least one depth camera or color camera.

5. The method of claim 1, wherein the registration signals are infrared signals.

6. The method of claim 1, wherein the registration signals are visible light signals.

7. The method of claim 1, further comprising, before the registration signals are received, soliciting the registration signals from the network-enabled appliances via near field communication, wireless Internet, or a personal area network.

8. The method of claim 1, wherein the at least one sensor includes an additional sensor that comprises an array microphone and the registration signals include audio signals.

9. The method of claim 1, further comprising reading visible tags to recognize network-enabled appliances.

10. An automation system comprising a control device for mapping and controlling network-enabled appliances, the control device comprising:

at least one sensor configured to collect spatial information in three dimensions as the at least one sensor is moved within an environment containing network-enabled appliances, wherein the at least one sensor includes at least one camera;

a communication unit, including at least the one sensor or at least one second sensor, and further including a transmitter, the communication unit being configured to receive registration signals from at least one of the network-enabled appliances in the environment; and

a processor configured to execute an automation program to;

recognize at least one of the network-enabled appliances within the environment with machine vision techniques, the machine vision techniques including applying text recognition algorithms to read physical tags to identify and recognize network enabled appliances in image data captured by the at least one camera;

associate each network-enabled appliance with a respective three-dimensional (3D) position in the environment based on the spatial information;

register each network-enabled appliance with the control device to create a registration log;

generate a digital map of the environment from the spatial information;

identify rooms within the environment;

place a representation of each network-enabled appliance on the digital map based on the respective 3D position;

generate a spatially aware rule set that applies to one or more of the rooms within the environment based upon the spatial information and the registration log; and

in response to a trigger event, command the network-enabled appliances within the room or rooms via the communication unit according to the spatially aware rule set.

11. The system of claim 10, wherein the automation program identifies the rooms by using object recognition to compare the spatial information to a typical room arrangement.

12. The system of claim 10, wherein the automation program identifies the rooms by receiving user input in the form of a spoken room name, a typed room name, or a manually selected room name, while the spatial information is collected.

13. The system of claim 10, wherein the at least one camera comprises at least one depth camera or color camera.

14. The system of claim 10, wherein the registration signals are infrared signals.

15. The system of claim 10, wherein the registration signals are visible light signals.

16. The system of claim 10, wherein the communication unit is configured to, before the registration signals are received, solicit the registration signals from the network-enabled appliances via near field communication, wireless Internet, or a personal area network.

17. The system of claim 10, wherein the at least one sensor includes an additional sensor that comprises an array microphone and the registration signals include audio signals.

18. The system of claim 10, wherein the automation program is configured to read visible tags to recognize the network-enabled appliances.

19. An automation system comprising a control device for mapping and controlling network-enabled appliances, the control device comprising:

at least one sensor configured to collect spatial information in three dimensions as the at least one sensor is moved within an environment containing network-enabled appliances, wherein the at least one sensor includes at least one camera that captures image data;

a communication unit, including at least the one sensor or at least one second sensor, and further including a transmitter, the communication unit being configured to receive registration signals from at least one of the network-enabled appliances in the environment; and

a processor configured to execute an automation program to;

use the image data captured by the at least one camera and/or an additional camera of the control device to recognize at least one of the network-enabled appliances with machine vision techniques, the machine vision techniques including text recognition algorithms to read physical tags to identify and recognize network enabled appliances in the image data captured by the at least one camera;

associate each network-enabled appliance with a respective three-dimensional (3D) position in the environment based on the spatial information;

register each network-enabled appliance with the control device to create a registration log;

generate a digital map of the environment from the spatial information;

identify rooms within the environment;

place a representation of each network-enabled appliance on the digital map based on the respective 3D position;

generate a spatially aware rule set that applies to one or more of the rooms within the environment based upon the spatial information and the registration log;

detect a trigger event; and

in response to the trigger event, command the network-enabled appliances within the room or rooms via the communication unit according to the spatially aware rule set.