Multi-mode video event indexing

• US 9,147,259 B2
• Filed: 08/21/2013
• Issued: 09/29/2015
• Est. Priority Date: 08/17/2010
• Status: Expired due to Fees

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Bobbitt, Russell P., Brown, Lisa M., Feris, Rogerio S., Hampapur, Arun, Zhai, Yun
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US13/971,964
• Publication Number:
  US 20130336535A1
• Time in Patent Office: 769 Days
• Field of Search: 382/103, 382/107, 348143-160, 348169-172
• US Class Current: 1/1
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Multi-mode video event indexing includes determining a quality of object distinctiveness with respect to images from a video stream input. A high-quality analytic mode is selected from multiple modes and applied to video input images via a hardware device to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality, else a low-quality analytic mode is selected and applied to the video input images via a hardware device to determine object activity within the video input images, wherein the low-quality analytic mode is different from the high-quality analytic mode.

Claims

1. A method for multi-mode video event indexing, the method comprising:

applying background subtraction to images from a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the video stream input images;

measuring an amount of light intensity within the video stream input images;

in response to the measured amount of light intensity meeting a threshold illumination level value, a processor selecting an object tracking based analytic mode from a plurality of video analytics modes and applying the object tracking based analytic mode to remove shadow pixels from regions of the detected foreground object activity of the video stream input images and to track remaining pixels of the detected foreground object activity within the video stream input images; and

in response to the measured amount of light intensity not meeting the threshold illumination level value, the processor selecting a non-object tracking based analytic mode from the plurality of video analytics modes and applying the non-object tracking based analytic mode to the detected foreground object activity of the video stream input images, wherein applying the non-object tracking based analytic mode comprises;

defining a virtual boundary line of a region of interest within the video input images;

establishing a plurality of overlapping ground patch regions along each side of the virtual boundary line;

analyzing in real-time each of the ground patch regions in the video input images data to identify attributes of objects detected in the region of interest; and

determining current appearance features of foreground objects identified within the region of interest; and

wherein the non-object tracking based analytic mode is different from the object tracking based analytic mode and parses video events from the detected foreground object activity of the video stream images for object activity detection without removing shadow pixels from the regions of the detected foreground object activity.

2. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processor, a computer readable memory and a computer readable storage medium, wherein the computer readable program code is embodied on the computer readable storage medium and comprises instructions that are executed by the processor via the computer readable memory to cause the processor to perform the steps of applying the background subtraction to the video stream input images, measuring the amount of light intensity, selecting and applying the object tracking based analytic mode to remove the shadow pixels and track remaining pixels of the detected foreground object activity within the video input images in response to the measured amount of light intensity meeting the threshold illumination level value, and selecting and applying the non-object tracking based analytic mode to the detected foreground object activity of the video input images in response to the measured amount of light intensity not meeting the threshold illumination level value.

3. The method of claim 1, further comprising:

computing brightness distortion values and color distortion values in red-green-blue color spaces for each of a plurality of pixels of the contextual background; and

modeling each of the contextual background pixels as 4-tuples of a first vector of means of red, green, and blue components computed over a plurality of background frames, a second vector of standard deviations of the red, green, and blue components, a variation of the computed brightness distortion, and a variation of the computing color distortion values.

4. The method of claim 3, wherein the object tracking based analytic mode removes shadow pixels from the regions of the detected foreground object activity of the video stream input images by:

calculating cross-correlated normalized intensities for each pixel of a neighborhood region of the detected foreground object activity between a current image frame and a background image as function of intensities of each of the pixels within the neighborhood region; and

removing a pixel as a shadow pixel in response to determining that its calculated normalized intensity exceeds a threshold intensity value.

5. The method of claim 1, wherein the step of applying the object tracking based analytic mode comprises:

using appearance-based modeling to group foreground regions of each video frame into a plurality of connected foreground region components that are each described by a bounding box; and

for each successive frame, attempting to associate each of the connected foreground region components with an existing foreground region component object track as a function of a distance between each of the foreground region components and each currently active existing foreground region component object track.

6. The method of claim 5, wherein the step of applying the object tracking based analytic mode comprises:

for each track, building a red-green-blue color appearance model that is representative of the appearance of each pixel of an object, and an associated probability mask that represents the likelihood of the object being observed with respect to the each pixel; and

for subsequent frames, updating the red-green-blue color appearance model by blending current image pixel data with the red-green-blue color appearance model for all foreground pixels, and updating all values of the associated probability masks.

7. The method of claim 1, further comprising:

updating a ground patch history model with attributes extracted from each of set of ground patch regions;

cross-referencing the updated patch history model against attributes from previously received video input data to determine if appearance data within each ground patch region has changed;

identifying pairs of the ground patch regions that each have similar updated appearance data that has changed;

determining moving directions for each of the paired ground patch regions; and

generating an alert for an object crossing the virtual boundary line in the region of interest in response to a match determined between a crossing direction of the virtual boundary line and the determining moving direction for one of the paired ground patch regions, in response to determining that the direction of motion for each of the pair of similar ground patch regions is substantially the same.

8. A system for multi-mode video event indexing, the computer system comprising:

a processor;

a computer readable memory; and

a computer readable storage medium;

wherein the processor executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby;

applies background subtraction to images from a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the video stream input images;

measures an amount of light intensity within the video stream input images;

in response to the measured amount of light intensity meeting a threshold illumination level value, selects an object tracking based analytic mode from a plurality of video analytics modes and applies the object tracking based analytic mode to remove shadow pixels from regions of the detected foreground object activity of the video stream input images and to track remaining pixels of the detected foreground object activity within the video stream input images; and

in response to the measured amount of light intensity not meeting the threshold illumination level value, selects a non-object tracking based analytic mode from the plurality of video analytics modes and applies the non-object tracking based analytic mode to the detected foreground object activity of the video stream input images to determine object activity within the video input images, wherein the processor applies the non-object tracking based analytic mode by;

defining a virtual boundary line of a region of interest within the video input images;

establishing a plurality of overlapping ground patch regions along each side of the virtual boundary line;

analyzing in real-time each of the ground patch regions in the video input images data to identify attributes of objects detected in the region of interest; and

determining current appearance features of foreground objects identified within the region of interest; and

wherein the non-object tracking based analytic mode is different from the object tracking based analytic mode and parses video events from the detected foreground object activity of the video stream images for object activity detection without removing shadow pixels from the regions of the detected foreground object activity.

9. The system of claim 8, wherein the processor executes the program instructions stored on the computer readable storage medium via the computer readable memory, and thereby:

computes brightness distortion values and color distortion values in red-green-blue color spaces for each of a plurality of pixels of the contextual background; and

models each of the contextual background pixels as 4-tuples of a first vector of means of red, green, and blue components computed over a plurality of background frames, a second vector of standard deviations of the red, green, and blue components, a variation of the computed brightness distortion, and a variation of the computing color distortion values.

10. The system of claim 9, wherein the processor executes the program instructions stored on the computer readable storage medium via the computer readable memory, and thereby removes shadow pixels from the regions of the detected foreground object activity of the video stream input images by:

calculating cross-correlated normalized intensities for each pixel of a neighborhood region of the detected foreground object activity between a current image frame and a background image as function of intensities of each of the pixels within the neighborhood region; and

removing a pixel as a shadow pixel in response to determining that its calculated normalized intensity exceeds a threshold intensity value.

11. The system of claim 8, wherein the processor executes the program instructions stored on the computer readable storage medium via the computer readable memory and thereby applies the object tracking based analytic mode by:

using appearance-based modeling to group foreground regions of each video frame into a plurality of connected foreground region components that are each described by a bounding box; and

for each successive frame, attempting to associate each of the connected foreground region components with an existing foreground region component object track as a function of a distance between each of the foreground region components and each currently active existing foreground region component object track.

17. The system of claim 8, wherein the processor executes the program instructions stored on the computer readable storage medium via the computer readable memory and thereby:

updates a ground patch history model with attributes extracted from each of a set of ground patch regions;

cross-references the updated patch history model against attributes from previously received video input data to determine if appearance data within each ground patch region has changed;

identifies pairs of the ground patch regions that each have similar updated appearance data that has changed;

determines moving directions for each of the paired ground patch regions; and

generates an alert for an object crossing the virtual boundary line in the region of interest in response to a match determined between a crossing direction of the virtual boundary line and the determining moving direction for one of the paired ground patch regions, in response to determining that the direction of motion for each of the pair of similar ground patch regions is substantially the same.

12. A computer program product for multi-mode video event indexing, the computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising instructions for execution by a computer processor that cause the computer processor to;

apply background subtraction to images from a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the video stream input images;

measure an amount of light intensity within the video stream input images;

in response to the measured amount of light intensity meeting a threshold illumination level value, select an object tracking based analytic mode from a plurality of video analytics modes and apply the object tracking based analytic mode to remove shadow pixels from regions of the detected foreground object activity of the video stream input images and to track remaining pixels of the detected foreground object activity within the video stream input images; and

in response to the measured amount of light intensity not meeting the threshold illumination level value, select a non-object tracking based analytic mode from the plurality of video analytics modes and apply the non-object tracking based analytic mode to the detected foreground object activity of the video input images to determine object activity within the video input images, wherein the processor is caused to apply the non-object tracking based analytic mode by;

defining a virtual boundary line of a region of interest within the video input images;

establishing a plurality of overlapping ground patch regions along each side of the virtual boundary line;

analyzing in real-time each of the ground patch regions in the video input images data to identify attributes of objects detected in the region of interest; and

determining current appearance features of foreground objects identified within the region of interest; and

wherein the non-object tracking based analytic mode is different from the object tracking based analytic mode and parses video events from the detected foreground object activity of the video stream images for object activity detection without removing shadow pixels from the regions of the detected foreground object activity.

13. The computer program product of claim 12, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to:

compute brightness distortion values and color distortion values in red-green-blue color spaces for each of a plurality of pixels of the contextual background; and

model each of the contextual background pixels as 4-tuples of a first vector of means of red, green, and blue components computed over a plurality of background frames, a second vector of standard deviations of the red, green, and blue components, a variation of the computed brightness distortion, and a variation of the computing color distortion values.

14. The computer program product of claim 13, wherein the computer-readable program code instructions for execution by the computer processor, further cause the computer processor to remove shadow pixels from the regions of the detected foreground object activity of the video stream input images by:

calculating cross-correlated normalized intensities for each pixel of a neighborhood region of the detected foreground object activity between a current image frame and a background image as function of the intensities of each of the pixels within the neighborhood region; and

removing a pixel as a shadow pixel in response to determining that its calculated normalized intensity exceeds a threshold intensity value.

15. The computer program product of claim 12, wherein the computer-readable program code instructions for execution by the computer processor, further cause the computer processor to apply the object tracking based analytic mode by:

using appearance-based modeling to group foreground regions of each video frame into a plurality of connected foreground region components that are each described by a bounding box; and

for each successive frame, attempting to associate each of the connected foreground region components with an existing foreground region component object track as a function of a distance between each of the foreground region components and each currently active existing foreground region component object track.

16. The computer program product of claim 12, wherein the computer-readable program code instructions for execution by the computer processor, further cause the computer processor to:

update a ground patch history model with attributes extracted from each of a set of ground patch regions;

cross-reference the updated patch history model against attributes from previously received video input data to determine if appearance data within each ground patch region has changed;

identify pairs of the ground patch regions that each have similar updated appearance data that has changed;

determine moving directions for each of the paired ground patch regions; and

generate an alert for an object crossing the virtual boundary line in the region of interest in response to a match determined between a crossing direction of the virtual boundary line and the determining moving direction for one of the paired ground patch regions, in response to determining that the direction of motion for each of the pair of similar ground patch regions is substantially the same.