Multi-mode video event indexing

• US 9,652,863 B2
• Filed: 08/19/2015
• Issued: 05/16/2017
• Est. Priority Date: 08/17/2010
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Brown, Lisa M., Hampapur, Arun, Bobbitt, Russell P., Feris, Rogerio S., Zhai, Yun
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/829,702
• Publication Number:
  US 20150356745A1
• Time in Patent Office: 636 Days
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Multi-mode video event indexing includes determining a quality of object distinctiveness with respect to images from a video stream input. A high-quality analytic mode is selected from multiple modes and applied to video input images via a hardware device to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality, else a low-quality analytic mode is selected and applied to the video input images via a hardware device to determine object activity within the video input images, wherein the low-quality analytic mode is different from the high-quality analytic mode.

Claims

1. A computer-implemented method for multi-mode video event indexing, the method comprising executing on a processor the steps of:

applying background subtraction to images from a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the video stream input images;

measuring an amount of light intensity within the video stream input images;

in response to the measured amount of light intensity not meeting a threshold illumination level value, defining a virtual boundary line of a region of interest within the video input images;

establishing a plurality of overlapping ground patch regions along each side of the virtual boundary line;

updating a ground patch history model with attributes extracted from each of set of ground patch regions;

cross-referencing the updated ground patch history model against attributes from previously received video input data to determine if appearance data within each ground patch region has changed;

identifying pairs of the ground patch regions that each have similar updated appearance data that has changed; and

determining moving directions for each of the paired ground patch regions.

2. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processor, a computer readable memory and a computer readable storage medium, wherein the computer readable program code is embodied on the computer readable storage medium and comprises instructions that are executed by the processor via the computer readable memory to cause the processor to perform the steps of applying the background subtraction to the video stream input images to detect foreground object activity, measuring the amount of light intensity within the video stream input images, defining the virtual boundary line of the region of interest within the video input images in response to the measured amount of light intensity not meeting the threshold illumination level value, establishing the plurality of overlapping ground patch regions along each side of the virtual boundary line, updating the ground patch history model with attributes extracted from each of set of ground patch regions, cross-referencing the updated ground patch history model against attributes from previously received video input data to determine if appearance data within each ground patch region has changed, identifying the pairs of the ground patch regions that each have similar updated appearance data that has changed, and determining moving directions for each of the paired ground patch regions.

3. The method of claim 1, further comprising executing on the processor the step of:

generating an alert for an object crossing the virtual boundary line in the region of interest in response to a match determined between a crossing direction of the virtual boundary line and the determined moving direction for one of the paired ground patch regions, in response to determining that the direction of motion for each of the pair of similar ground patch regions is substantially the same.

4. The method of claim 3, further comprising executing on the processor the steps of:

computing brightness distortion values and color distortion values in red-green-blue color spaces for each of a plurality of pixels of the contextual background; and

modeling each of the contextual background pixels as 4-tuples of a first vector of means of red, green, and blue components computed over a plurality of background frames, a second vector of standard deviations of the red, green, and blue components, a variation of the computed brightness distortion, and a variation of the computing color distortion values.

5. The method of claim 4, further comprising:

in response to the measured amount of light intensity meeting the threshold illumination level value, applying an object tracking based analytic mode to remove shadow pixels from regions of the detected foreground object activity of the video stream input images and to track remaining pixels of the detected foreground object activity within the video stream input images;

wherein the object tracking based analytic mode removes shadow pixels from the regions of the detected foreground object activity of the video stream input images by;

calculating cross-correlated normalized intensities for each pixel of a neighborhood region of the detected foreground object activity between a current image frame and a background image as function of intensities of each of the pixels within the neighborhood region; and

removing a pixel as a shadow pixel in response to determining that its calculated normalized intensity exceeds a threshold intensity value.

6. The method of claim 5, further comprising executing on the processor the steps of:

using appearance-based modeling to group foreground regions of each video frame into a plurality of connected foreground region components that are each described by a bounding box; and

for each successive frame, attempting to associate each of the connected foreground region components with an existing foreground region component object track as a function of a distance between each of the foreground region components and each currently active existing foreground region component object track.

7. The method of claim 6, further comprising executing on the processor the steps of:

for each track, building a red-green-blue color appearance model that is representative of the appearance of each pixel of an object, and an associated probability mask that represents the likelihood of the object being observed with respect to the each pixel; and

for subsequent frames, updating the red-green-blue color appearance model by blending current image pixel data with the red-green-blue color appearance model for all foreground pixels, and updating all values of the associated probability masks.

8. A system, comprising:

a processor;

a computer readable memory in circuit communication with the processor; and

a computer readable storage medium in circuit communication with the processor;

wherein the processor executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby;

applies background subtraction to images from a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the video stream input images;

measures an amount of light intensity within the video stream input images;

in response to the measured amount of light intensity not meeting a threshold illumination level value, defines a virtual boundary line of a region of interest within the video input images;

establishes a plurality of overlapping ground patch regions along each side of the virtual boundary line;

updates a ground patch history model with attributes extracted from each of set of ground patch regions;

cross-references the updated ground patch history model against attributes from previously received video input data to determine if appearance data within each ground patch region has changed;

identifies pairs of the ground patch regions that each have similar updated appearance data that has changed; and

determines moving directions for each of the paired ground patch regions.

9. The system of claim 8, wherein the processor executes the program instructions stored on the computer readable storage medium via the computer readable memory, and thereby:

generates an alert for an object crossing the virtual boundary line in the region of interest in response to a match determined between a crossing direction of the virtual boundary line and the determined moving direction for one of the paired ground patch regions, in response to determining that the direction of motion for each of the pair of similar ground patch regions is substantially the same.

10. The system of claim 9, wherein the processor executes the program instructions stored on the computer readable storage medium via the computer readable memory, and thereby:

computes brightness distortion values and color distortion values in red-green-blue color spaces for each of a plurality of pixels of the contextual background; and

models each of the contextual background pixels as 4-tuples of a first vector of means of red, green, and blue components computed over a plurality of background frames, a second vector of standard deviations of the red, green, and blue components, a variation of the computed brightness distortion, and a variation of the computing color distortion values.

11. The system of claim 10, wherein the processor executes the program instructions stored on the computer readable storage medium via the computer readable memory and thereby:

in response to the measured amount of light intensity meeting the threshold illumination level value, applies an object tracking based analytic mode to remove shadow pixels from regions of the detected foreground object activity of the video stream input images and to track remaining pixels of the detected foreground object activity within the video stream input images;

wherein application of the object tracking based analytic mode removes shadow pixels from the regions of the detected foreground object activity of the video stream input images by;

calculating cross-correlated normalized intensities for each pixel of a neighborhood region of the detected foreground object activity between a current image frame and a background image as function of intensities of each of the pixels within the neighborhood region; and

removing a pixel as a shadow pixel in response to determining that its calculated normalized intensity exceeds a threshold intensity value.

12. The system of claim 11, wherein the processor executes the program instructions stored on the computer readable storage medium via the computer readable memory and thereby:

uses appearance-based modeling to group foreground regions of each video frame into a plurality of connected foreground region components that are each described by a bounding box; and

for each successive frame, attempts to associate each of the connected foreground region components with an existing foreground region component object track as a function of a distance between each of the foreground region components and each currently active existing foreground region component object track.

13. The system of claim 12, wherein the processor executes the program instructions stored on the computer readable storage medium via the computer readable memory and thereby:

for each track, builds a red-green-blue color appearance model that is representative of the appearance of each pixel of an object, and an associated probability mask that represents the likelihood of the object being observed with respect to the each pixel; and

for subsequent frames, updates the red-green-blue color appearance model by blending current image pixel data with the red-green-blue color appearance model for all foreground pixels, and updating all values of the associated probability masks.

14. A computer program product for multi-mode video event indexing, the computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising instructions for execution by a computer processor that cause the computer processor to;

apply background subtraction to images from a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the video stream input images;

measure an amount of light intensity within the video stream input images;

in response to the measured amount of light intensity not meeting a threshold illumination level value, define a virtual boundary line of a region of interest within the video input images;

establish a plurality of overlapping ground patch regions along each side of the virtual boundary line;

update a ground patch history model with attributes extracted from each of set of ground patch regions;

cross-reference the updated ground patch history model against attributes from previously received video input data to determine if appearance data within each ground patch region has changed;

identify pairs of the ground patch regions that each have similar updated appearance data that has changed; and

determine moving directions for each of the paired ground patch regions.

15. The computer program product of claim 14, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to:

generate an alert for an object crossing the virtual boundary line in the region of interest in response to a match determined between a crossing direction of the virtual boundary line and the determined moving direction for one of the paired ground patch regions, in response to determining that the direction of motion for each of the pair of similar ground patch regions is substantially the same.

16. The computer program product of claim 15, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to:

compute brightness distortion values and color distortion values in red-green-blue color spaces for each of a plurality of pixels of the contextual background; and

model each of the contextual background pixels as 4-tuples of a first vector of means of red, green, and blue components computed over a plurality of background frames, a second vector of standard deviations of the red, green, and blue components, a variation of the computed brightness distortion, and a variation of the computing color distortion values.

17. The computer program product of claim 16, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to:

in response to the measured amount of light intensity meeting the threshold illumination level value, apply an object tracking based analytic mode to remove shadow pixels from regions of the detected foreground object activity of the video stream input images and to track remaining pixels of the detected foreground object activity within the video stream input images;

wherein application of the object tracking based analytic mode removes shadow pixels from the regions of the detected foreground object activity of the video stream input images by;

calculating cross-correlated normalized intensities for each pixel of a neighborhood region of the detected foreground object activity between a current image frame and a background image as function of intensities of each of the pixels within the neighborhood region; and

removing a pixel as a shadow pixel in response to determining that its calculated normalized intensity exceeds a threshold intensity value.

18. The computer program product of claim 17, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to:

use appearance-based modeling to group foreground regions of each video frame into a plurality of connected foreground region components that are each described by a bounding box; and

for each successive frame, attempt to associate each of the connected foreground region components with an existing foreground region component object track as a function of a distance between each of the foreground region components and each currently active existing foreground region component object track.

19. The computer program product of claim 18, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to:

for each track, build a red-green-blue color appearance model that is representative of the appearance of each pixel of an object, and an associated probability mask that represents the likelihood of the object being observed with respect to the each pixel; and

for subsequent frames, update the red-green-blue color appearance model by blending current image pixel data with the red-green-blue color appearance model for all foreground pixels, and updating all values of the associated probability masks.