Method and system for motion vector-based video monitoring and event categorization

• US 9,158,974 B1
• Filed: 10/08/2014
• Issued: 10/13/2015
• Est. Priority Date: 07/07/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Laska, Jason N., Duffy, Greg, Nelson, Gregory R.
• Primary Examiner(s): Thirugnanam, Gandhi
• Application Number:
  US14/509,999
• Time in Patent Office: 370 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24147 Distances to closest patterns, e.g. nearest neighbour classification
• CPC Class Codes: G06F 18/243 relating to the number of classes
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: H04N 21/2181 comprising remotely distributed storage units, e.g. when movies are replicated over a plurality of video servers distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: H04N 21/239 : Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests hybrid fiber coaxial [HFC] networks for upstream channel allocation for video distribution H04L12/2801; flow control in data networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/44231 : Monitoring of peripheral device or external card, e.g. to detect processing problems in a handheld device or the failure of an external recording device configuring of peripheral devices in general G06F9/4411; reporting information sensed by appliance or service execution status of appliance services in a home automation network H04L12/2823; monitoring the status of connected device in data switched networks H04L43/0817
• CPC Class Codes: H04N 23/632 : for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/633 : for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 5/144 : Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 : Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/2628 : Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A computer system processes a video stream to detect a start of a first motion event candidate in the video stream, and in response to detecting the start of the first motion event candidate in the video stream, initiates event recognition processing on a first video segment associated with the start of the first motion event candidate. Initiating the event recognition processing further includes: determining a motion track of a first object identified in the first video segment; generating a representative motion vector for the first motion event candidate based on the motion track of the first object; and sending the representative motion vector for the first motion event candidate to an event categorizer, where the event categorizer assigns a respective motion event category to the first motion event candidate based on the representative motion vector of the first motion event candidate.

Claims

1. A method of processing a video stream, comprising:

processing the video stream to detect a start of a first motion event candidate in the video stream, wherein processing comprises;

obtaining a profile of a total motion pixel count for a current frame sequence in the video stream;

in response to determining that the obtained profile meets a predetermined trigger criterion, determining that the current frame sequence includes a motion event candidate;

identifying a beginning time for a portion of the profile meeting the predetermined trigger criterion; and

designating the identified beginning time to be the start of the first motion event candidate; and

in response to detecting the start of the first motion event candidate in the video stream, initiating event recognition processing on a first video segment associated with the start of the first motion event candidate, wherein initiating the event recognition processing further comprises;

determining a motion track of a first object identified in the first video segment;

generating a representative motion vector for the first motion event candidate based on the motion track of the first object; and

sending the representative motion vector for the first motion event candidate to an event categorizer, wherein the event categorizer assigns a respective motion event category to the first motion event candidate based on the representative motion vector of the first motion event candidate.

2. The method of claim 1, wherein a source of the video stream dynamically determines whether to locally perform the processing of the video stream, the determining of the motion track, and the generating of the representative motion vector, based on one or more predetermined distributed processing criteria.

3. The method of claim 1, further comprising:

in response to detecting the start of the first motion event candidate, uploading the first video segment from a source of the video stream to a remote server, wherein the first video segment begins at a predetermined lead time before the start of the first motion event candidate and lasts a predetermined duration.

4. The method of claim 3, wherein uploading the first video segment from the source of the video stream to the remote server further comprises:

in response to detecting the start of the first motion event candidate, uploading the first video segment at a higher quality level as compared to a normal quality level at which video data is uploaded for cloud storage.

5. The method of claim 1, further comprising:

in response to detecting the start of the first motion event candidate, extracting the first video segment from cloud storage for the video stream, wherein the first video segment begins at a predetermined lead time before the start of the first motion event candidate and lasts a predetermined duration.

6. The method of claim 1, wherein obtaining the profile of motion pixel counts for the current frame sequence in the video stream further comprises:

generating a raw profile based on the respective motion pixel count for each frame in the current frame sequence; and

generating the profile of motion pixel counts by smoothing the raw profile to remove one or more temporary dips in pixel counts in the raw profile.

7. The method of claim 1, further comprising:

receiving a respective motion pixel count for each frame of the video stream from a source of the video stream.

8. The method of claim 7, wherein the respective motion pixel count is adjusted for one or more of changes of camera states during generation of the video stream.

9. The method of claim 1, wherein determining the motion track of the object identified in the first video segment further comprises:

based on a frame sequence of the first video segment;

performing background estimation to obtain a background for the first video segment;

performing object segmentation to identify one or more foreground objects in the first video segment by subtracting the obtained background from the frame sequence, the one or more foreground object including the object; and

establishing a respective motion track for each of the one or more foreground objects by associating respective motion masks of the foreground object across multiple frames of the frame sequence.

10. The method of claim 9, further comprising:

determining a duration of the respective motion track for each of the one or more foreground objects; and

discarding zero or more respective motion tracks and corresponding foreground objects if the durations of the respective zero or more motion tracks are shorter than a predetermined duration.

11. A method of processing a video stream, comprising:

processing the video stream to detect a start of a first motion event candidate in the video stream; and

in response to detecting the start of the first motion event candidate in the video stream, initiating event recognition processing on a first video segment associated with the start of the first motion event candidate, wherein initiating the event recognition processing further comprises;

determining a motion track of a first object identified in the first video segment;

generating a representative motion vector for the first motion event candidate based on the motion track of the first object; and

sending the representative motion vector for the first motion event candidate to an event categorizer, wherein the event categorizer assigns a respective motion event category to the first motion event candidate based on the representative motion vector of the first motion event candidate,

wherein determining the motion track of the object identified in the first video segment further comprises, based on a frame sequence of the first video segment;

performing background estimation to obtain a background for the first video segment;

performing object segmentation to identify one or more foreground objects in the first video segment by subtracting the obtained background from the frame sequence, the one or more foreground object including the object; and

establishing a respective motion track for each of the one or more foreground objects by associating respective motion masks of the foreground object across multiple frames of the frame sequence, and

wherein performing the object segmentation to identify one or more foreground objects and establishing the respective motion track for each of the one or more foreground objects further comprises;

building a histogram of foreground pixels identified in the frame sequence of the first video segment, wherein the histogram specifies a frame count for each pixel location in a scene of the first video segment;

filtering the histogram to remove regions below a predetermined frame count;

segmenting the filtered histogram into the one or more motion regions; and

selecting one or more dominant motion regions from the one or more motion regions based on a predetermined dominance criterion, where each dominant motion region corresponds to the respective motion track of a corresponding one of the one or more foreground objects.

12. The method of claim 11, further comprising:

generating a respective event mask for the foreground object corresponding to a first dominant motion region of the one or more dominant regions based on the first dominant motion region.

13. The method of claim 11, wherein a source of the video stream dynamically determines whether to locally perform the processing of the video stream, the determining of the motion track, and the generating of the representative motion vector, based on one or more predetermined distributed processing criteria.

14. The method of claim 11, further comprising:

in response to detecting the start of the first motion event candidate, uploading the first video segment from a source of the video stream to a remote server, wherein the first video segment begins at a predetermined lead time before the start of the first motion event candidate and lasts a predetermined duration.

15. The method of claim 11, further comprising:

in response to detecting the start of the first motion event candidate, extracting the first video segment from cloud storage for the video stream, wherein the first video segment begins at a predetermined lead time before the start of the first motion event candidate and lasts a predetermined duration.

16. A computing system for processing a video stream, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

processing the video stream to detect a start of a first motion event candidate in the video stream, wherein processing comprises;

obtaining a profile of a total motion pixel count for a current frame sequence in the video stream;

in response to determining that the obtained profile meets a predetermined trigger criterion, determining that the current frame sequence includes a motion event candidate;

identifying a beginning time for a portion of the profile meeting the predetermined trigger criterion; and

designating the identified beginning time to be the start of the first motion event candidate; and

in response to detecting the start of the first motion event candidate in the video stream, initiating event recognition processing on a first video segment associated with the start of the first motion event candidate, wherein initiating the event recognition processing further comprises;

determining a motion track of a first object identified in the first video segment;

generating a representative motion vector for the first motion event candidate based on the motion track of the first object; and

sending the representative motion vector for the first motion event candidate to an event categorizer, wherein the event categorizer assigns a respective motion event category to the first motion event candidate based on the representative motion vector of the first motion event candidate.

17. The computing system of claim 16, wherein the computing system includes at least a source of the video stream and a remote server, and wherein the source of the video stream dynamically determines whether to locally perform the processing of the video stream, the determining of the motion track, and the generating of the representative motion vector, based on one or more predetermined distributed processing criteria.

18. The computing system of claim 16, wherein the operations further comprise:

in response to detecting the start of the first motion event candidate, uploading the first video segment from the source of the video stream to a remote server, wherein the first video segment begins at a predetermined lead time before the start of the first motion event candidate and lasts a predetermined duration.

19. The computing system of claim 18, wherein uploading the first video segment from the source of the video stream to the remote server further comprises:

in response to detecting the start of the first motion event candidate, uploading the first video segment at a higher quality level as compared to a normal quality level at which video data is uploaded for cloud storage.

20. The computing system of claim 16, wherein the operations further comprise:

in response to detecting the start of the first motion event candidate, extracting the first video segment from cloud storage for the video stream, wherein the first video segment begins at a predetermined lead time before the start of the first motion event candidate and lasts a predetermined duration.

21. The computing system of claim 16, wherein obtaining the profile of motion pixel counts for the current frame sequence in the video stream further comprises:

generating a raw profile based on the respective motion pixel count for each frame in the current frame sequence; and

generating the profile of motion pixel counts by smoothing the raw profile to remove one or more temporary dips in pixel counts in the raw profile.

22. The computing system of claim 16, wherein the operations further comprise:

receiving a respective motion pixel count for each frame of the video stream from a source of the video stream.

23. The computing system of claim 22, wherein the respective motion pixel count is adjusted for one or more of changes of camera states during generation of the video stream.

24. The computing system of claim 16, wherein determining the motion track of the object identified in the first video segment further comprises:

based on a frame sequence of the first video segment;

performing background estimation to obtain a background for the first video segment;

performing object segmentation to identify one or more foreground objects in the first video segment by subtracting the obtained background from the frame sequence, the one or more foreground object including the object; and

establishing a respective motion track for each of the one or more foreground objects by associating respective motion masks of the foreground object across multiple frames of the frame sequence.

25. The computing system of claim 24, wherein the operations further comprise:

determining a duration of the respective motion track for each of the one or more foreground objects; and

discarding zero or more respective motion tracks and corresponding foreground objects if the durations of the respective zero or more motion tracks are shorter than a predetermined duration.

26. A computing system for processing a video stream, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

processing the video stream to detect a start of a first motion event candidate in the video stream; and

in response to detecting the start of the first motion event candidate in the video stream, initiating event recognition processing on a first video segment associated with the start of the first motion event candidate, wherein initiating the event recognition processing further comprises;

determining a motion track of a first object identified in the first video segment;

generating a representative motion vector for the first motion event candidate based on the motion track of the first object; and

sending the representative motion vector for the first motion event candidate to an event categorizer, wherein the event categorizer assigns a respective motion event category to the first motion event candidate based on the representative motion vector of the first motion event candidate,

wherein determining the motion track of the object identified in the first video segment further comprises, based on a frame sequence of the first video segment;

performing background estimation to obtain a background for the first video segment;

performing object segmentation to identify one or more foreground objects in the first video segment by subtracting the obtained background from the frame sequence, the one or more foreground object including the object; and

establishing a respective motion track for each of the one or more foreground objects by associating respective motion masks of the foreground object across multiple frames of the frame sequence, and

wherein performing the object segmentation to identify one or more foreground objects and establishing the respective motion track for each of the one or more foreground objects further comprises;

building a histogram of foreground pixels identified in the frame sequence of the first video segment, wherein the histogram specifies a frame count for each pixel location in a scene of the first video segment;

filtering the histogram to remove regions below a predetermined frame count;

segmenting the filtered histogram into the one or more motion regions; and

selecting one or more dominant motion regions from the one or more motion regions based on a predetermined dominance criterion, where each dominant motion region corresponds to the respective motion track of a corresponding one of the one or more foreground objects.

27. The computing system of claim 26, wherein the operations further comprise:

generating a respective event mask for the foreground object corresponding to a first dominant motion region of the one or more dominant regions based on the first dominant motion region.

28. The computing system of claim 26, wherein a source of the video stream dynamically determines whether to locally perform the processing of the video stream, the determining of the motion track, and the generating of the representative motion vector, based on one or more predetermined distributed processing criteria.

29. The computing system of claim 26, wherein the operations further comprise:

in response to detecting the start of the first motion event candidate, uploading the first video segment from a source of the video stream to a remote server, wherein the first video segment begins at a predetermined lead time before the start of the first motion event candidate and lasts a predetermined duration.

30. The computing system of claim 26, wherein the operations further comprise:

in response to detecting the start of the first motion event candidate, extracting the first video segment from cloud storage for the video stream, wherein the first video segment begins at a predetermined lead time before the start of the first motion event candidate and lasts a predetermined duration.