Systems and methods for controlling smart devices based upon image data from image sensors

• US 10,356,303 B1
• Filed: 10/02/2015
• Issued: 07/16/2019
• Est. Priority Date: 10/07/2014
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Jordan, II, Jackie O., Donovan, John, Turrentine, David, Wollenschlager, Torri, Nussbaum, Bryan R., Stockweather, Deanna, Stoiber, Jeffrey W., Markwardt, Kerstin, Carlson, Gail L., Schiebel, Kyle C., Winslow, Troy, Harr, Joseph P., Wagner, Ellakate, Harris, Jr., Michael, Sun, Jennylind
• Primary Examiner(s): Perungavoor, Sathyanaraya V
• Assistant Examiner(s): Dang, Philip P.
• Application Number:
  US14/873,783
• Time in Patent Office: 1,383 Days
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06Q 30/01 Customer relationship services
• CPC Class Codes: G06Q 30/018 Certifying business or products
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06Q 50/163 Real estate management
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G08B 1/02 using only mechanical transmission
• CPC Class Codes: G08B 13/22 Electrical actuation
• CPC Class Codes: G08B 17/10 Actuation by presence of smoke or gases , e.g. automatic alarm devices for analysing flowing fluid materials by the use of optical means
• CPC Class Codes: G08B 21/02 Alarms for ensuring the safety of persons
• CPC Class Codes: G08B 21/043 detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0446 worn on the body to detect changes of posture, e.g. a fall, inclination, acceleration, gait
• CPC Class Codes: G08B 21/0453 worn on the body to detect health condition by physiological monitoring, e.g. electrocardiogram, temperature, breathing detecting, measuring or recording for diagnostic purposes A61B5/00
• CPC Class Codes: G08B 21/0461 integrated or attached to an item closely associated with the person but not worn by the person, e.g. chair, walking stick, bed sensor
• CPC Class Codes: G08B 21/182 Level alarms, e.g. alarms responsive to variables exceeding a threshold
• CPC Class Codes: G08B 21/185 Electrical failure alarms
• CPC Class Codes: G08B 7/062 indicating emergency exits
• CPC Class Codes: G08B 7/066 guiding along a path, e.g. evacuation path lighting strip
• CPC Class Codes: H04L 12/2818 : from a device located outside both the home and the home network access arrangements H04L12/2856; for remote control or remote monitoring of applications H04L67/025; arrangements for transmitting signals characterised by the use of a wireless eletrical link G08C17/00; telephonic communication systems adapted for combination with remote control systems H04M11/007
• CPC Class Codes: H04L 12/2825 : Reporting to a device located outside the home and the home network access arrangements H04L12/2856; for remote control or remote monitoring of applications H04L67/025; telephonic communication systems adapted for combination with telemetering systems H04M11/002
• CPC Class Codes: H04L 12/283 : Processing of data at an internetworking point of a home automation network
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/024 : Guidance services
• CPC Class Codes: H04W 4/90 : Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]

Abstract

The present embodiments relate to managing the operation of devices within a home or other property based upon received image data. According to certain aspects, a controller within the home may receive, with customer permission or consent, image data from image sensors disposed throughout the home. The controller may analyze the image data in combination with profile data for individuals associated with the home to determine that a certain individual is indicated in the image data. The controller may further determine an action to facilitate based upon the individual being detected in a certain location of the property, and may direct smart devices to perform that action. As a result, the present embodiments may facilitate providing occupant-location based (or occupant-presence or preference based) functionality, functions, or services, and/or directing operations of smart devices located about a property based upon occupant location, presence, preferences, and/or activities at the property.

Claims

1. A computer-implemented method of controlling a plurality of devices associated with a property, the property populated with a hardware controller in communication with the plurality of devices, the method comprising:

receiving, by the hardware controller via a communication network, image data from a set of image sensors disposed at a location of the property;

accessing, from memory, at least one occupant profile associated with at least one occupant of the property, the at least one occupant profile containing a set of images depicting the at least one occupant and a set of physical attributes of the at least one occupant;

analyzing the image data to detect an individual and a set of characteristics of the individual;

determining that the at least one occupant is depicted in the image data, including;

comparing the set of characteristics of the individual to the set of physical attributes of the at least one occupant, and

performing a facial recognition technique on the image data and the set of images contained in the at least one occupant profile;

determining, by the processor, a set of additional auxiliary items depicted in the image data and associated with the at least one occupant;

determining, by the processor from the image data, how the at least one occupant is interacting with the set of additional auxiliary items;

based upon the at least one occupant profile;

determining, further based upon how the at least one occupant is interacting with the set of additional auxiliary items, an action to be performed, and

determining, based upon the location of the set of image sensors, at least one of the plurality of devices to perform the action; and

directing the at least one of the plurality of devices to perform the action.

2. The computer-implemented method of claim 1, wherein accessing the at least one occupant profile comprises:

accessing a set of preferences of the at least one occupant;

and wherein determining the action to be performed comprises;

determining the action to be performed according to the set of preferences of the at least one occupant.

3. The computer-implemented method of claim 2, wherein determining the at least one of the plurality of devices to perform the action comprises:

determining the at least one of the plurality of devices that is capable of performing the action to comply with the set of preferences of the at least one occupant.

4. The computer-implemented method of claim 1, wherein directing the at least one of the plurality of devices to perform the action comprises:

directing the at least one of the plurality of devices to adjust operation.

5. The computer-implemented method of claim 1, wherein directing the at least one of the plurality of devices to perform the action comprises:

generating a command to perform the action; and

transmitting, by the hardware controller via the communication network, the command to the at least one of the plurality of devices, wherein the at least one of the plurality of devices executes the command.

6. The computer-implemented method of claim 1, further comprising:

receiving, from the at least one of the plurality of devices via the communication network, a confirmation that the action was performed.

7. The computer-implemented method of claim 1, further comprising:

determining, based upon the image data, a location of the property where the at least one occupant is located;

wherein determining the action to be performed is further based upon the location of the property where the at least one occupant is located.

8. The computer-implemented method of claim 1, further comprising:

determining, based upon the image data, an activity being performed by the at least one occupant;

wherein determining the action to be performed is further based upon the activity being performed by the at least one occupant.

15. The hardware controller of claim 7, wherein the processor is further configured to:

determine, based upon the image data, a location of the property where the at least one occupant is located;

wherein the processor determines the action to be performed further based upon the location of the property where the at least one occupant is located.

16. The hardware controller of claim 7, wherein the processor is further configured to:

determine, based upon the image data, an activity being performed by the at least one occupant;

wherein the processor determines the action to be performed further based upon the activity being performed by the at least one occupant.

9. A hardware controller for controlling a plurality of devices associated with a property, the property populated with a hardware controller in communication with the plurality of devices, comprising:

at least one transceiver configured to receive image data from a set of image sensors disposed at a location of the property;

a memory configured to store non-transitory computer executable instructions and a set of occupant profiles; and

a processor interfacing with the at least one transceiver and the memory, wherein the processor is configured to execute the non-transitory computer executable instructions to cause the processor to;

access, from the memory, at least one occupant profile of the set of occupant profiles, the at least one occupant profile associated with at least one occupant of the property, the at least one occupant profile containing a set of images depicting the at least one occupant and a set of physical attributes of the at least one occupant,

determine that the at least one occupant is depicted in the image data, including;

compare the set of characteristics of the individual to the set of physical attributes of the at least one occupant, and

perform a facial recognition technique on the image data and the set of images contained in the at least one occupant profile,

determine a set of additional auxiliary items depicted in the image data and associated with the at least one occupant,

determine, from the image data, how the at least one occupant is interacting with the set of additional auxiliary items;

based upon the data contained in the at least one occupant profile;

determine, further based upon how the at least one occupant is interacting with the set of additional auxiliary items, an action to be performed, and

determine, based upon the location of the set of image sensors, at least one of the plurality of devices to perform the action, and

direct the at least one of the plurality of devices to perform the action.

10. The hardware controller of claim 9, wherein to access the at least one occupant profile, the processor is configured to:

access a set of preferences of the at least one occupant,

and wherein to determine the action to be performed, the processor is configured to;

determine the action to be performed according to the set of preferences of the at least one occupant.

11. The hardware controller of claim 10, wherein to determine the at least one of the plurality of devices to perform the action, the processor is configured to:

determine the at least one of the plurality of devices that is capable of performing the action to comply with the set of preferences of the at least one occupant.

12. The hardware controller of claim 9, wherein to direct the at least one of the plurality of devices to perform the action, the processor is configured to:

direct the at least one of the plurality of devices to adjust operation.

13. The hardware controller of claim 9, wherein to direct the at least one of the plurality of devices to perform the action, the processor is configured to:

generate a command to perform the action, and

transmit, via the at least one transceiver, the command to the at least one of the plurality of devices, wherein the at least one of the plurality of devices executes the command.

14. The hardware controller of claim 9, wherein the processor is further configured to:

receive, from the at least one of the plurality of devices via the at least one transceiver, a confirmation that the action was performed.