Methods and systems for providing intelligent alerts for events

• US 10,192,415 B2
• Filed: 07/11/2016
• Issued: 01/29/2019
• Est. Priority Date: 07/11/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Heitz, III, George Alban, Bapat, Akshay R., Nariyawala, Mehul
• Primary Examiner(s): Atala, Jamie J
• Assistant Examiner(s): Boylan, James T
• Application Number:
  US15/207,463
• Publication Number:
  US 20180012460A1
• Time in Patent Office: 932 Days
• Field of Search: 348135, 348143
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19615 wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 13/19684 Portable terminal, e.g. mobile phone, used for viewing video remotely
• CPC Class Codes: G08B 13/19691 Signalling events for better perception by user, e.g. indicating alarms by making display brighter, adding text, creating a sound

Abstract

The various embodiments described herein include methods, devices, and systems for providing event alerts. In one aspect, a method includes: (1) receiving a plurality of video frames from a camera, the plurality of video frames including a motion event candidate; (2) categorizing the motion event candidate by processing the plurality of video frames, the categorizing including: (a) associating the motion event candidate with a first category of a plurality of motion event categories; and (b) generating a confidence level for the association of the motion event candidate with the first category; and (3) sending an alert indicative of the first category and the confidence level to a user associated with the camera.

Claims

1. A method, comprising:

at a server system having one or more processors and memory, the server system managing a plurality of smart home environments, each smart home environment of the plurality of smart home environments corresponding to a distinct set of one or more cameras, registered users, and client devices;

receiving a plurality of video frames from a camera of a first smart home environment of the plurality of smart home environments, the plurality of video frames including a motion event candidate;

categorizing the motion event candidate by processing the plurality of video frames, the categorizing including;

characterizing motion of the motion event candidate;

identifying an object involved in the motion event candidate;

associating the motion event candidate with a first category of a plurality of motion event categories based on the characterized motion and identified object; and

generating a confidence level for the association of the motion event candidate with the first category;

determining whether the confidence level meets one or more predefined criteria;

obtaining a descriptive phrase indicative of whether the confidence level meets the one or more predefined criteria;

selecting a first electronic delivery method of a plurality of electronic delivery methods for sending an alert to a registered user of the first smart home environment via a communication network, wherein the first delivery method is based at least in part on the confidence level; and

using the first electronic delivery method, sending the alert for the motion event candidate to a client device of the registered user, the alert including the obtained phrase and information regarding the first category.

2. The method of claim 1, wherein the first category indicates that the motion event involves at least one of:

a person;

a known person; and

an unknown person.

3. The method of claim 1, wherein the first category indicates that the motion event involves a particular portion of a field of view of the camera.

4. The method of claim 1, wherein categorizing the motion event candidate comprises categorizing the motion event candidate by processing the plurality of video frames and analyzing information received from a device of the first smart home environment, distinct from the camera.

5. The method of claim 1, wherein categorizing the motion event candidate by processing the plurality of video frames comprises analyzing at least one of:

total amount of motion in the video frames;

direction of motion detected in the video frames;

velocity of motion detected in the video frames; and

whether motion detected in the video frames corresponds to a recognized activity.

6. The method of claim 1, further comprising analyzing one or more audio events corresponding to the motion event candidate;

wherein generating the confidence level comprises generating the confidence level based at least in part on the analyzed one or more audio events.

7. A computing system comprising:

one or more processors; and

memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving a plurality of video frames from a camera of a first smart home environment of the plurality of smart home environments, the plurality of video frames including a motion event candidate;

categorizing the motion event candidate by processing the plurality of video frames, the categorizing including;

characterizing motion of the motion event candidate;

identifying an object involved in the motion event candidate;

associating the motion event candidate with a first category of a plurality of motion event categories based on the characterized motion and identified object; and

generating a confidence level for the association of the motion event candidate with the first category;

determining whether the confidence level meets one or more predefined criteria;

obtaining a descriptive phrase indicative of whether the confidence level meets the one or more predefined criteria; and

selecting a first electronic delivery method of a plurality of electronic delivery methods for sending an alert to a registered user of the first smart home environment via a communication network, wherein the first delivery method is based at least in part on the confidence level; and

using the first electronic delivery method, sending the alert for the motion event candidate to a client device of the registered user, the alert including the obtained phrase and information regarding the first category.

8. The computing system of claim 7, wherein the first category indicates that the motion event involves a particular portion of a field of view of the camera.

9. The computing system of claim 7, wherein categorizing the motion event candidate comprises categorizing the motion event candidate by processing the plurality of video frames and analyzing information received from a device of the first smart home environment, distinct from the camera.

10. The computing system of claim 7, the one or more programs further include instructions for analyzing one or more audio events corresponding to the motion event candidate;

and

wherein generating the confidence level comprises generating the confidence level based at least in part on the analyzed one or more audio events.

11. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to:

receive a plurality of video frames from a camera of a first smart home environment of the plurality of smart home environments, the plurality of video frames including a motion event candidate;

categorize the motion event candidate by processing the plurality of video frames, the categorizing including;

characterize motion of the motion event candidate;

identify an object involved in the motion event candidate;

associate the motion event candidate with a first category of a plurality of motion event categories based on the characterized motion and identified object; and

generate a confidence level for the association of the motion event candidate with the first category;

determine whether the confidence level meets one or more predefined criteria;

obtain a descriptive phrase indicative of whether the confidence level meets the one or more predefined criteria; and

select a first electronic delivery method of a plurality of electronic delivery methods for sending an alert to a registered user of the first smart home environment via a communication network, wherein the first delivery method is based at least in part on the confidence level; and

using the first electronic delivery method, send the alert for the motion event candidate to a client device registered user, the alert including the obtained phrase and information regarding the first category.

12. The non-transitory computer-readable storage medium of claim 11, wherein the first category indicates that the motion event involves a particular portion of a field of view of the camera.

13. The non-transitory computer-readable storage medium of claim 11, wherein categorizing the motion event candidate comprises categorizing the motion event candidate by processing the plurality of video frames and analyzing information received from a device of the first smart home environment, distinct from the camera.

14. The non-transitory computer-readable storage medium of claim 11, wherein the first electronic delivery method utilizes first display characteristics that are distinct from respective display characteristics of at least a subset of the plurality of electronic delivery methods.

15. The non-transitory computer-readable storage medium of claim 11, wherein identifying the object involved in the motion event candidate comprises identifying the object as a known person;

and

wherein the alert includes a name of the known person.

16. The non-transitory computer-readable storage medium of claim 11, wherein the information regarding the first category comprises a descriptive phrase indicative of the first category.

17. The non-transitory computer-readable storage medium of claim 11, wherein the alert further includes timing information for the motion event candidate.