Method and system for cluster-based video monitoring and event categorization

• US 9,489,580 B2
• Filed: 09/24/2015
• Issued: 11/08/2016
• Est. Priority Date: 07/07/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Laska, Jason N., Nelson, Gregory R., Duffy, Greg, Mitsuji, Hiro, Hill, Cameron, Davidsson, Martin, Montalbo, Michael D., Wan, Tung Yuen
• Primary Examiner(s): Werner, David N
• Application Number:
  US14/864,584
• Publication Number:
  US 20160012609A1
• Time in Patent Office: 411 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24147 Distances to closest patterns, e.g. nearest neighbour classification
• CPC Class Codes: G06F 18/243 relating to the number of classes
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: H04N 21/2181 comprising remotely distributed storage units, e.g. when movies are replicated over a plurality of video servers distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: H04N 21/239 : Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests hybrid fiber coaxial [HFC] networks for upstream channel allocation for video distribution H04L12/2801; flow control in data networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/44231 : Monitoring of peripheral device or external card, e.g. to detect processing problems in a handheld device or the failure of an external recording device configuring of peripheral devices in general G06F9/4411; reporting information sensed by appliance or service execution status of appliance services in a home automation network H04L12/2823; monitoring the status of connected device in data switched networks H04L43/0817
• CPC Class Codes: H04N 23/632 : for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/633 : for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 5/144 : Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 : Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/2628 : Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A computing system obtains a respective motion vector for each of a series of motion event candidates in real-time as said each motion event candidate is detected in a live video stream. In response to receiving the respective motion vector for each of the series of motion event candidates, the computing system determines a spatial relationship between the respective motion vector of said each motion event candidate to one or more existing clusters established based on a plurality of previously processed motion vectors, and in accordance with a determination that the respective motion vector of a first motion event candidate of the series of motion event candidates falls within a respective range of at least a first existing cluster of the one or more existing clusters, assigns the first motion event candidate to at least a first event category associated with the first existing cluster.

Claims

1. A method of categorizing a motion event candidate, comprising:

forming one or more clusters comprising respective pluralities of motion vectors, wherein motion vectors of the respective pluralities of motion vectors include respective start locations and end locations of a respective object in motion,

the motion vectors corresponding to respective motion event candidates previously detected in a live video stream, and

the one or more clusters being associated with a respective event category and having a respective cluster creation time at which the respective cluster was formed;

in accordance with a determination that a first cluster of the one or more clusters has included fewer than a threshold number of motion vectors for at least a threshold amount of time since a respective cluster creation time of the first cluster;

deleting the first cluster including all motion vectors currently in the first cluster; and

removing the motion event candidates corresponding to the deleted motion vectors from a first event category associated with the first cluster.

2. The method of claim 1, wherein the first event category associated with the first cluster is a category of recognized events.

3. The method of claim 1, further comprising:

in accordance with the determination that the first cluster has included fewer than the threshold number of motion vectors for at least the threshold amount of time, and prior to deleting the first cluster;

presenting to a user respective motion event candidates corresponding for the first cluster.

4. The method of claim 3, further comprising:

in response to presenting the respective motion event candidates for the first cluster to the user, receiving an indication to remove the first cluster,

wherein deleting the first cluster is performed in accordance with receiving the indication to remove the first cluster.

5. A computing system for categorizing a motion event candidate, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

forming one or more clusters comprising respective pluralities of motion vectors, wherein motion vectors of the respective pluralities of motion vectors include respective start locations and end locations of a respective object in motion,

the motion vectors corresponding to respective motion event candidates previously detected in a live video stream, and

the one or more clusters being associated with a respective event category and having a respective cluster creation time at which the respective cluster was formed;

in accordance with a determination that a first cluster of the one or more clusters has included fewer than a threshold number of motion vectors for at least a threshold amount of time since a respective cluster creation time of the first cluster;

deleting the first cluster including all motion vectors currently in the first cluster; and

removing the motion event candidates corresponding to the deleted motion vectors from a first event category associated with the first cluster.

6. The computing system of claim 5, wherein the operations further comprise:

storing a respective cluster creation time, a respective current cluster weight, a respective current cluster center, and a respective current cluster radius for each of the one or more clusters.

7. The computing system of claim 6, wherein the operations further comprise:

determining a respective current cluster density for the first cluster based on a respective current cluster weight and a respective current cluster radius of the first cluster.

8. The computing system of claim 7, wherein the operations further comprise:

in accordance with a determination that the respective current cluster density of the first cluster meets a predetermined cluster promotion density threshold, promoting the first existing cluster as a dense cluster.

9. The computing system of claim 8, wherein promoting the first existing cluster further comprises:

creating a new event category for the first cluster; and

disassociating the first cluster from the first event category.

10. The computing system of claim 9, wherein the operations further comprise:

after disassociating the first cluster from the first event category, reassigning all motion vectors in the first cluster into the new event category.

11. A method of categorizing a motion event candidate, comprising:

forming one or more clusters comprising respective pluralities of motion vectors,

the motion vectors corresponding to respective motion event candidates previously detected in a live video stream, and

the one or more clusters being associated with a respective event category and having a respective cluster creation time at which the respective cluster was formed;

in accordance with a determination that a first cluster of the one or more clusters has included fewer than a threshold number of motion vectors for at least a threshold amount of time since a respective cluster creation time of the first cluster;

deleting the first cluster including all motion vectors currently in the first cluster; and

removing the motion event candidates corresponding to the deleted motion vectors from a first event category associated with the first cluster; and

removing the deleted motion vectors of first cluster from an event timeline on a review interface.

12. The method of claim 11, wherein the first event category associated with the first cluster is a category of unrecognized events.

13. The method of claim 11, wherein the first event category associated with the first cluster is a category of recognized events.

14. The method of claim 11, further comprising:

in accordance with the determination that the first cluster has included fewer than the threshold number of motion vectors for at least the threshold amount of time, and prior to deleting the first cluster;

presenting to a user respective motion event candidates corresponding for the first cluster.

15. The method of claim 14, further comprising:

in response to presenting the respective motion event candidates for the first cluster to the user, receiving an indication to remove the first cluster,

wherein deleting the first cluster is performed in accordance with receiving the indication to remove the first cluster.

16. A computing system for categorizing a motion event candidate, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

forming one or more clusters comprising respective pluralities of motion vectors,

the motion vectors corresponding to respective motion event candidates previously detected in a live video stream, and

the one or more clusters being associated with a respective event category and having a respective cluster creation time at which the respective cluster was formed;

in accordance with a determination that a first cluster of the one or more clusters has included fewer than a threshold number of motion vectors for at least a threshold amount of time since a respective cluster creation time of the first cluster;

deleting the first cluster including all motion vectors currently in the first cluster; and

removing the motion event candidates corresponding to the deleted motion vectors from a first event category associated with the first cluster; and

removing the deleted motion vectors of first cluster from an event timeline on a review interface.

17. The computing system of claim 16, wherein the operations further comprise:

storing a respective cluster creation time, a respective current cluster weight, a respective current cluster center, and a respective current cluster radius for each of the one or more clusters.

18. The computing system of claim 17, wherein the operations further comprise:

determining a respective current cluster density for the first cluster based on a respective current cluster weight and a respective current cluster radius of the first cluster.

19. The computing system of claim 18, wherein the operations further comprise:

in accordance with a determination that the respective current cluster density of the first cluster meets a predetermined cluster promotion density threshold, promoting the first existing cluster as a dense cluster.

20. The computing system of claim 19, wherein promoting the first existing cluster further comprises:

creating a new event category for the first cluster; and

disassociating the first cluster from the first event category.

21. A method of categorizing a motion event candidate, comprising:

forming one or more clusters comprising respective pluralities of motion vectors,

the motion vectors corresponding to respective motion event candidates previously detected in a live video stream, and

the one or more clusters being associated with a respective event category and having a respective cluster creation time at which the respective cluster was formed;

in accordance with a determination that a first cluster of the one or more clusters has included fewer than a threshold number of motion vectors for at least a threshold amount of time since a respective cluster creation time of the first cluster;

deleting the first cluster including all motion vectors currently in the first cluster; and

removing the motion event candidates corresponding to the deleted motion vectors from a first event category associated with the first cluster; and

assigning a different display appearance to the deleted motion vectors of first cluster in an event timeline on a review interface.

22. The method of claim 21, further comprising assigning the deleted motion vectors of first cluster to a category of rare events.

23. The method of claim 21, wherein the first event category associated with the first cluster is a category of recognized events.

24. The method of claim 21, further comprising:

in accordance with the determination that the first cluster has included fewer than the threshold number of motion vectors for at least the threshold amount of time, and prior to deleting the first cluster;

presenting to a user respective motion event candidates corresponding for the first cluster.

25. The method of claim 24, further comprising:

in response to presenting the respective motion event candidates for the first cluster to the user, receiving an indication to remove the first cluster,

wherein deleting the first cluster is performed in accordance with receiving the indication to remove the first cluster.

26. A computing system for categorizing a motion event candidate, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

forming one or more clusters comprising respective pluralities of motion vectors,

the motion vectors corresponding to respective motion event candidates previously detected in a live video stream, and

the one or more clusters being associated with a respective event category and having a respective cluster creation time at which the respective cluster was formed;

in accordance with a determination that a first cluster of the one or more clusters has included fewer than a threshold number of motion vectors for at least a threshold amount of time since a respective cluster creation time of the first cluster;

deleting the first cluster including all motion vectors currently in the first cluster; and

removing the motion event candidates corresponding to the deleted motion vectors from a first event category associated with the first cluster; and

assigning a different display appearance to the deleted motion vectors of first cluster in an event timeline on a review interface.

27. The computing system of claim 26, wherein the operations further comprise:

storing a respective cluster creation time, a respective current cluster weight, a respective current cluster center, and a respective current cluster radius for each of the one or more clusters.

28. The computing system of claim 27, wherein the operations further comprise:

determining a respective current cluster density for the first cluster based on a respective current cluster weight and a respective current cluster radius of the first cluster.

29. The computing system of claim 28, wherein the operations further comprise:

in accordance with a determination that the respective current cluster density of the first cluster meets a predetermined cluster promotion density threshold, promoting the first existing cluster as a dense cluster.

30. The computing system of claim 29, wherein promoting the first existing cluster further comprises:

creating a new event category for the first cluster; and

disassociating the first cluster from the first event category.

31. A method of categorizing a motion event candidate, comprising:

forming one or more clusters comprising respective pluralities of motion vectors,

the motion vectors corresponding to respective motion event candidates previously detected in a live video stream, and

the one or more clusters being associated with a respective event category and having a respective cluster creation time at which the respective cluster was formed;

in accordance with a determination that a first cluster of the one or more clusters has included fewer than a threshold number of motion vectors for at least a threshold amount of time since a respective cluster creation time of the first cluster;

deleting the first cluster including all motion vectors currently in the first cluster; and

removing the motion event candidates corresponding to the deleted motion vectors from a first event category associated with the first cluster;

wherein the determination that the first cluster of the one or more clusters has included fewer than the threshold number of motion vectors for at least the threshold amount of time corresponds to a scene change in the live video stream.

32. The method of claim 31, wherein the first event category associated with the first cluster is a category of unrecognized events.

33. The method of claim 31, wherein the first event category associated with the first cluster is a category of recognized events.

34. The method of claim 31, further comprising:

in accordance with the determination that the first cluster has included fewer than the threshold number of motion vectors for at least the threshold amount of time, and prior to deleting the first cluster;

presenting to a user respective motion event candidates corresponding for the first cluster.

35. The method of claim 34, further comprising:

in response to presenting the respective motion event candidates for the first cluster to the user, receiving an indication to remove the first cluster,

wherein deleting the first cluster is performed in accordance with receiving the indication to remove the first cluster.

36. A computing system for categorizing a motion event candidate, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

forming one or more clusters comprising respective pluralities of motion vectors,

the motion vectors corresponding to respective motion event candidates previously detected in a live video stream, and

the one or more clusters being associated with a respective event category and having a respective cluster creation time at which the respective cluster was formed;

in accordance with a determination that a first cluster of the one or more clusters has included fewer than a threshold number of motion vectors for at least a threshold amount of time since a respective cluster creation time of the first cluster;

deleting the first cluster including all motion vectors currently in the first cluster; and

removing the motion event candidates corresponding to the deleted motion vectors from a first event category associated with the first cluster;

wherein the determination that the first cluster of the one or more clusters has included fewer than the threshold number of motion vectors for at least the threshold amount of time corresponds to a scene change in the live video stream.

37. The computing system of claim 36, wherein the operations further comprise:

storing a respective cluster creation time, a respective current cluster weight, a respective current cluster center, and a respective current cluster radius for each of the one or more clusters.

38. The computing system of claim 37, wherein the operations further comprise:

determining a respective current cluster density for the first cluster based on a respective current cluster weight and a respective current cluster radius of the first cluster.

39. The computing system of claim 38, wherein the operations further comprise:

in accordance with a determination that the respective current cluster density of the first cluster meets a predetermined cluster promotion density threshold, promoting the first existing cluster as a dense cluster.

40. The computing system of claim 39, wherein promoting the first existing cluster further comprises:

creating a new event category for the first cluster; and

disassociating the first cluster from the first event category.