Method and system for video zone monitoring

• US 9,224,044 B1
• Filed: 10/08/2014
• Issued: 12/29/2015
• Est. Priority Date: 07/07/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Laska, Jason N., Nelson, Gregory R., Duffy, Greg, Mitsuji, Hiro, Hill, Cameron, Davidsson, Martin, Montalbo, Michael D., Wan, Tung Yuen
• Primary Examiner(s): Thirugnanam, Gandhi
• Application Number:
  US14/510,040
• Time in Patent Office: 447 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24147 Distances to closest patterns, e.g. nearest neighbour classification
• CPC Class Codes: G06F 18/243 relating to the number of classes
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: H04N 21/2181 comprising remotely distributed storage units, e.g. when movies are replicated over a plurality of video servers distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: H04N 21/239 : Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests hybrid fiber coaxial [HFC] networks for upstream channel allocation for video distribution H04L12/2801; flow control in data networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/44231 : Monitoring of peripheral device or external card, e.g. to detect processing problems in a handheld device or the failure of an external recording device configuring of peripheral devices in general G06F9/4411; reporting information sensed by appliance or service execution status of appliance services in a home automation network H04L12/2823; monitoring the status of connected device in data switched networks H04L43/0817
• CPC Class Codes: H04N 23/632 : for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/633 : for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 5/144 : Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 : Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/2628 : Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A computing system receives a definition of a zone of interest within the scene depicted in the video steam. In response to receiving the definition of the zone of interest, the computing system determines, for each motion event detected in the video stream, whether a respective event mask of the motion event overlaps with the zone of interest by at least a predetermined overlap factor; and identifying the motion event as an event of interest associated with the zone of interest in accordance with a determination that the respective event mask of the motion event overlaps with the zone of interest by at least the predetermined overlap factor.

Claims

1. A method of monitoring selected zones in a scene depicted in a video stream:

receiving a definition of a zone of interest within the scene depicted in the video stream; and

in response to receiving the definition of the zone of interest;

determining, for each motion event detected in the video stream, whether a respective event mask of the motion event overlaps with the zone of interest by at least a predetermined overlap factor;

identifying the motion event as an event of interest associated with the zone of interest in accordance with a determination that the respective event mask of the motion event overlaps with the zone of interest by at least the predetermined overlap factor; and

generating the respective event mask for the motion event, wherein the generating includes;

creating binary motion pixel maps for frames of a video segment associated with the motion event; and

aggregating the binary motion pixel maps of frames of the video segment to generate the respective event mask for the motion event.

2. The method of claim 1, further comprising:

receiving a first selection input from a user to select the zone of interest as a first event filter;

receiving a second selection input from the user to select one or more object features as a second event filter to be combined with the first event filter;

determining whether the identified event of interest includes at least one object in motion satisfying the one or more object features; and

generating a real-time alert for the user in accordance with a determination that the identified event of interest includes at least one object in motion satisfying the one or more object features.

3. The method of claim 2, further comprising:

visually labeling the identified event of interest with a respective indicator associated with both the zone of interest and the one or more object features in an event review interface.

4. The method of claim 2, wherein the one or more object features are features representing a human.

5. The method of claim 1, wherein the definition of the zone of interest includes a plurality of vertices specified in the scene of the video recording.

6. The method of claim 1, wherein the video stream is a live video stream, and wherein determining whether the respective event mask of the motion event overlaps with the zone of interest by at least a predetermined overlap factor further comprises:

processing the live video stream in real-time to detect a start of a live motion event;

generating a live event mask based on respective motion pixels associated with a respective object in motion identified in the live motion event; and

determining, in real-time, whether the live event mask overlaps with the zone of interest by at least the predetermined overlap factor.

7. The method of claim 1, further comprising:

providing a composite video segment corresponding to the identified event of interest, the composite video segment including a plurality of composite frames each including a high-resolution portion covering the zone of interest, and a low-resolution portion covering regions outside of the zone of interest.

8. A computing system for monitoring selected zones in a scene depicted in a video stream, the computing system comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

receiving a definition of a zone of interest within the scene depicted in the video stream; and

in response to receiving the definition of the zone of interest;

determining, for each motion event detected in the video stream, whether a respective event mask of the motion event overlaps with the zone of interest by at least a predetermined overlap factor;

identifying the motion event as an event of interest associated with the zone of interest in accordance with a determination that the respective event mask of the motion event overlaps with the zone of interest by at least the predetermined overlap factor; and

generating the respective event mask for the motion event, wherein the generating includes;

creating binary motion pixel maps for frames of a video segment associated with the motion event; and

aggregating the binary motion pixel maps of frames of the video segment to generate the respective event mask for the motion event.

9. The computing system of claim 8, wherein the operations further comprise:

receiving a first selection input from a user to select the zone of interest as a first event filter;

receiving a second selection input from the user to select one or more object features as a second event filter to be combined with the first event filter;

determining whether the identified event of interest includes at least one object in motion satisfying the one or more object features; and

generating a real-time alert for the user in accordance with a determination that the identified event of interest includes at least one object in motion satisfying the one or more object features.

10. The computing system of claim 9, wherein the operations further comprise:

visually labeling the identified event of interest with a respective indicator associated with both the zone of interest and the one or more object features in an event review interface.

11. The computing system of claim 9, wherein the one or more object features are features representing a human.

12. The computing system of claim 8, wherein the definition of the zone of interest includes a plurality of vertices specified in the scene of the video recording.

13. The computing system of claim 8, wherein the video stream is a live video stream, and wherein determining whether the respective event mask of the motion event overlaps with the zone of interest by at least a predetermined overlap factor further comprises:

processing the live video stream in real-time to detect a start of a live motion event;

generating a live event mask based on respective motion pixels associated with a respective object in motion identified in the live motion event; and

determining, in real-time, whether the live event mask overlaps with the zone of interest by at least the predetermined overlap factor.

14. The computing system of claim 8, wherein the operations further comprise:

providing a composite video segment corresponding to the identified event of interest, the composite video segment including a plurality of composite frames each including a high-resolution portion covering the zone of interest, and a low-resolution portion covering regions outside of the zone of interest.

15. A non-transitory computer-readable medium for monitoring selected zones in a scene depicted in a video stream, the non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

receiving a definition of a zone of interest within the scene depicted in the video stream; and

in response to receiving the definition of the zone of interest;

determining, for each motion event detected in the video stream, whether a respective event mask of the motion event overlaps with the zone of interest by at least a predetermined overlap factor;

identifying the motion event as an event of interest associated with the zone of interest in accordance with a determination that the respective event mask of the motion event overlaps with the zone of interest by at least the predetermined overlap factor; and

generating the respective event mask for the motion event, wherein the generating includes;

creating binary motion pixel maps for frames of a video segment associated with the motion event; and

aggregating the binary motion pixel maps of frames of the video segment to generate the respective event mask for the motion event.

16. The non-transitory computer-readable medium of claim 15, wherein the operations further comprise:

receiving a first selection input from a user to select the zone of interest as a first event filter;

receiving a second selection input from the user to select one or more object features as a second event filter to be combined with the first event filter;

determining whether the identified event of interest includes at least one object in motion satisfying the one or more object features; and

generating a real-time alert for the user in accordance with a determination that the identified event of interest includes at least one object in motion satisfying the one or more object features.

17. The non-transitory computer-readable medium of claim 16, wherein the operations further comprise:

visually labeling the identified event of interest with a respective indicator associated with both the zone of interest and the one or more object features in an event review interface.