Systems and methods of detecting and responding to a visitor to a smart home environment

• US 10,664,688 B2
• Filed: 11/10/2017
• Issued: 05/26/2020
• Est. Priority Date: 09/20/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Goulden, Jason Evans, Aravamudhan, Rengarajan, Jeong, Haerim, Dixon, Michael, Stewart, James Edward, Shafi, Sayed Yusef, Mysore, Sahana, Yang, Seungho, Lien, Yu-An, Burns, Christopher Charles, Nongpiur, Rajeev, Boyd, Jeffrey
• Primary Examiner(s): Yang, Qian
• Application Number:
  US15/809,924
• Publication Number:
  US 20190087646A1
• Time in Patent Office: 928 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19684 Portable terminal, e.g. mobile phone, used for viewing video remotely
• CPC Class Codes: G08B 13/19695 Arrangements wherein non-video detectors start video recording or forwarding but do not generate an alarm themselves
• CPC Class Codes: G08B 3/10 using electric transmission; using electromagnetic transmission
• CPC Class Codes: H04M 11/02 with bell or annunciator systems
• CPC Class Codes: H04M 11/025 Door telephones adapted for television H04N7/186

Abstract

A method of detecting and responding to a visitor to a smart home environment via an electronic greeting system of the smart home environment, including determining that a visitor is approaching an entryway of the smart home environment; initiating a facial recognition operation while the visitor is approaching the entryway; initiating an observation window in response to the determination that a visitor is approaching the entryway; obtaining context information from one or more sensors of the smart home environment during the observation window; and at the end of the time window, initiating a response to the detected approach of the visitor based on the context information and/or an outcome of the facial recognition operation.

Claims

1. A method comprising:

at an electronic greeting system of a smart home environment having one or more processors and memory;

determining that a visitor is approaching an entryway of the smart home environment;

in response to the determination that the visitor is approaching the entryway;

(a) initiating a facial recognition operation while the visitor is approaching the entryway; and

(b) initiating an observation window, including;

initiating the observation window before the visitor reaches a physical interaction area of the entryway and/or before the visitor initiates an announcement; and

ending the observation window at the earlier of (i) a predetermined time threshold and (ii) a detected visitor announcement event;

in accordance with the initiating of the observation window, obtaining context information from one or more sensors of the smart home environment during the observation window; and

in accordance with the ending of the observation window, initiating a response to the detected approach of the visitor based on the context information and an outcome of the facial recognition operation.

2. The method of claim 1, wherein determining that a visitor is approaching the entryway comprises obtaining and analyzing presence information indicative of a visitor approaching or in proximity to the entryway.

3. The method of claim 2, wherein the presence information is derived from video data obtained from a camera having a field of view corresponding to the entryway;

and

wherein deriving the presence information from the video data comprises analyzing a plurality of image frames to determine whether a difference in position of an object in the plurality of image frames exceeds a threshold.

4. The method of claim 2, wherein the presence information is derived from audio data obtained from an audio sensor in proximity to the entryway, and

wherein deriving the presence information from the audio data comprises identifying:

a door knock, a doorbell sound, a footstep, and/or a verbal announcement.

5. The method of claim 1, wherein detecting a visitor of the smart home environment comprises:

obtaining position data from a sensor of the smart home environment;

identifying, based on analysis of the position data, a position of the visitor with respect to the entryway; and

comparing the position of the visitor to a threshold distance from the entryway, or to a previous position of the visitor.

6. The method of claim 1, wherein determining that a visitor is approaching the entryway comprises:

obtaining a first measurement of a dimension of a characteristic of the visitor;

obtaining a subsequent measurement of the dimension of the characteristic of the visitor; and

determining whether a difference between the first measurement and the subsequent measurement exceeds a threshold.

7. The method of claim 1, wherein the context information is based on at least one of a time of day, a day of the week, audio data of the smart home environment, a time of a prescheduled event, a prescheduled status of the smart home environment, a status of a user of the electronic greeting system, a location of the user, and an identity of the user.

8. An electronic greeting system of a smart home environment, the system comprising one or more processors and memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:

determining that a visitor is approaching an entryway of the smart home environment;

in response to the determination that the visitor is approaching the entryway;

(a) initiating a facial recognition operation while the visitor is approaching the entryway; and

(b) initiating an observation window, including;

initiating the observation window before the visitor reaches a physical interaction area of the entryway and/or before the visitor initiates an announcement; and

ending the observation window at the earlier of (i) a predetermined time threshold and (ii) a detected visitor announcement event;

in accordance with the initiating of the observation window, obtaining context information from one or more sensors of the smart home environment during the observation window; and

in accordance with the ending of the observation window, initiating a response to the detected approach of the visitor based on the context information and an outcome of the facial recognition operation.

9. The electronic greeting system of claim 8, wherein the context information comprises a detected announcement event.

10. The electronic greeting system of claim 9, wherein the detected announcement event comprises:

a doorbell button press, a door knock, and/or a verbal announcement.

11. The electronic greeting system of claim 9, wherein the detected announcement event is part of a pre-assigned doorbell button press pattern, door knock pattern, or verbal announcement associated with a known visitor.

12. The electronic greeting system of claim 8, wherein the context information is based on:

a facial recognition analysis result;

one or more behavior characteristics of the visitor;

one or more physical characteristics of the visitor;

one or more clothing and/or accessory characteristics of the visitor;

a time of day during which the visitor approaches the entryway;

a day of the week during which the visitor approaches the entryway;

audio data from the smart home environment;

proximity in time to a prescheduled event;

proximity in time to a prescheduled status of the smart home environment;

a known or unknown status of a user of the electronic greeting system;

an expected or unexpected status of a user of the electronic greeting system;

a location of a user of the electronic greeting system;

an identity of a user of the electronic greeting system; and

/or

one or more detected visitor actions.

13. The electronic greeting system of claim 8, wherein the context information is based on one or more detected visitor actions including:

a doorbell activation;

a door knock;

an audio announcement; and

/or

an interaction between the visitor and the electronic greeting system.

14. The electronic greeting system of claim 8, wherein the context information is further based on a timing comparison between one or more detected visitor actions and the initial detection of the visitor.

15. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to:

determine that a visitor is approaching an entryway of a smart home environment;

in response to the determination that the visitor is approaching the entryway;

(a) initiate a facial recognition operation while the visitor is approaching the entryway; and

(b) initiate an observation window, including;

initiating the observation window before the visitor reaches a physical interaction area of the entryway and/or before the visitor initiates an announcement; and

ending the observation window at the earlier of (i) a predetermined time threshold and (ii) a detected visitor announcement event;

in accordance with the initiating of the observation window, obtain context information from one or more sensors of the smart home environment during the observation window; and

in accordance with the ending of the observation window, initiate a response to the detected approach of the visitor based on the context information and an outcome of the facial recognition operation.

16. The non-transitory computer-readable storage medium of claim 15, wherein the context information comprises an absence of a detected announcement event during a predetermined time threshold.

17. The non-transitory computer-readable storage medium of claim 15, wherein the response to the detected approach of the visitor comprises:

conveying a communication to the visitor;

initiating a security action; and

/or

transmitting a notification to a user of the computing system, to a preselected contact of the user, or to public or private law enforcement personnel.

18. The non-transitory computer-readable storage medium of claim 17, wherein the response to the detected approach of the visitor comprises conveying a communication comprising:

a status of a user of the computing system;

a request for the visitor to perform an action;

a request for the visitor to leave a message;

a preprogrammed customized communication;

a user-composed text message for conversion to an audio message;

an audio message conveyed by a synthesized voice; and

/or

a visual message presented on a display.

19. The non-transitory computer-readable storage medium of claim 17, wherein the response to the detected approach of the visitor comprises initiating a security action including:

activating a light or adjusting a lighting level of the smart home environment;

locking or unlocking a door of the smart home environment;

activating an alarm or adjusting an alarm sensitivity of the smart home environment;

activating a sprinkler system of the smart home environment;

activating a simulated dog bark;

activating a security system or adjusting a security status of the smart home environment;

transmitting a notification or an alert to public or private law enforcement personnel;

transmitting a notification or an alert to a preselected contact of the user; and

/or

recording an image or video of the visitor.

20. The non-transitory computer-readable storage medium of claim 17, wherein the response to the detected approach of the visitor comprises transmitting a notification including:

information about the visitor;

information about an outcome of the facial recognition operation;

information about a detected visitor announcement event or lack thereof; and

/or

information about the obtained context information.