MULTI-MODE VIDEO EVENT INDEXING

• US 20130336535A1
• Filed: 08/21/2013
• Published: 12/19/2013
• Est. Priority Date: 08/17/2010
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Bobbitt, Russell P., Brown, Lisa M., Hampapur, Arun, Zhai, Yun, Feris, Rogerio S.
• Granted Patent: US 9,147,259 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Multi-mode video event indexing includes determining a quality of object distinctiveness with respect to images from a video stream input. A high-quality analytic mode is selected from multiple modes and applied to video input images via a hardware device to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality, else a low-quality analytic mode is selected and applied to the video input images via a hardware device to determine object activity within the video input images, wherein the low-quality analytic mode is different from the high-quality analytic mode.

Claims

1. A method for multi-mode video event indexing, the method comprising:

determining by a processor a quality of object distinctiveness with respect to images from a video stream input;

in response to the determined level of detected quality of object distinctiveness meeting a threshold level of quality, the processor selecting a high-quality analytic mode from a plurality of video analytics modes and applying the high-quality analytic mode to the video input images to determine object activity within the video input images; and

in response to the determined level of detected quality of object distinctiveness not meeting the threshold level of quality, the processor selecting a low-quality analytic mode from the plurality of video analytics modes and applying the low-quality analytic mode to the video input images to determine object activity within the video input images; and

wherein the determining the quality of object distinctiveness with respect to the video input images comprises determining an amount of scene illumination in the video input images, the threshold level of quality is an amount of scene illumination, the high-quality analytic mode is a strong-shadow analytic mode that provides for object activity detection by removing strong shadows from the scene images, and the low-quality analytic mode is a low-light analytic mode that is different from the high-quality analytic mode and parses the video images for object activity detection without removing shadows from the scene images.

2. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processor, a computer readable memory and a computer readable storage medium, wherein the computer readable program code is embodied on the computer readable storage medium and comprises instructions that, when executed by the processor via the computer readable memory, cause the processor to perform the steps of determining the quality of object distinctiveness with respect to the video input images from the video stream input, selecting and applying the high-quality analytic mode to the video input images to determine object activity within the video input images in response to the determined level of detected quality of the detected foreground object activity meeting the threshold level of quality, and selecting and applying the low-quality analytic mode to the video input images to determine object activity within the video input images in response to the determined level of detected quality of the detected foreground object activity not meeting the threshold level of quality.

3. The method of claim 1, wherein the determining the amount of scene illumination comprises determining that a weather condition of the scene is sunny, raining, snowing, foggy, cloudy or nighttime, and wherein the threshold amount of illumination is sunny.

4. The method of claim 3, wherein the high-quality analytic mode is an object tracking based mode, and the low quality mode is a non-object tracking based mode.

5. The method of claim 4, wherein the threshold level of quality is a threshold level of background activity, the method further comprising:

applying background subtraction to the video stream input to detect the foreground object activity as a difference between current frame image information and static information of a contextual background of video input images from the video stream input; and

determining a level of the detected foreground object activity; and

wherein the step of selecting the high-quality analytic mode is responsive to the determined level of the detected foreground object activity being lower than the threshold level of background activity, and the step of selecting the low-quality analytic mode is responsive to the determined level of the detected foreground object activity not being lower than the threshold level of background activity.

6. The method of claim 5, wherein the step of applying the high-quality, object tracking based analytic mode comprises:

using appearance-based modeling to group foreground regions of each video frame into a plurality of connected foreground region components that are each described by a bounding box; and

for each successive frame, attempting to associate each of the connected foreground region components with an existing foreground region component object track as a function of a distance between each of the foreground region components and each currently active existing foreground region component object track.

7. The method of claim 6, wherein the step of applying the high-quality, object tracking based analytic mode comprises:

for each track, building a red-green-blue color appearance model that is representative of the appearance of each pixel of an object, and an associated probability mask that represents the likelihood of the object being observed with respect to the each pixel; and

for subsequent frames, updating the red-green-blue color appearance model by blending current image pixel data with the red-green-blue color appearance model for all foreground pixels, and updating all values of the associated probability masks.

8. The method of claim 5, wherein the step of applying the low-quality, non-object tracking based analytic mode comprises:

defining a virtual boundary line of a region of interest within the video input images;

establishing a plurality of overlapping ground patch regions along each side of the virtual boundary line;

analyzing in real-time each of the ground patch regions in the video input images data to identifying attributes of objects detected in the region of interest; and

determining current appearance features of foreground objects identified within the region of interest.

9. The method of claim 8, further comprising:

updating a ground patch history model with attributes extracted from each of set of ground patch regions;

cross-referencing the updated patch history model against attributes from previously received video input data to determine if appearance data within each ground patch region has changed;

identifying pairs of the ground patch regions that each have similar updated appearance data that has changed; and

determining moving directions for each of the paired ground patch regions; and

generating an alert for an object crossing the virtual boundary line in the region of interest in response to a match determined between a crossing direction of the virtual boundary line and the determining moving direction for one of the paired ground patch regions, if the direction of motion for each of the pair of similar ground patch regions is substantially the same.

10. A system for multi-mode video event indexing, the computer system comprising:

a processor;

a computer readable memory;

a computer readable storage medium;

wherein the processor, when executing program instructions stored on the computer-readable storage medium via the computer readable memory;

determines a quality of object distinctiveness with respect to images from a video stream input;

in response to the determined level of detected quality of object distinctiveness meeting a threshold level of quality, selects a high-quality analytic mode from a plurality of video analytics modes and applies the high-quality analytic mode to the video input images to determine object activity within the video input images; and

in response to the determined level of detected quality of object distinctiveness not meeting the threshold level of quality, selects a low-quality analytic mode from the plurality of video analytics modes and applies the low-quality analytic mode to the video input images to determine object activity within the video input images; and

wherein the determining the quality of object distinctiveness with respect to the video input images comprises determining an amount of scene illumination in the video input images, the threshold level of quality is an amount of scene illumination, the high-quality analytic mode is a strong-shadow analytic mode that provides for object activity detection by removing strong shadows from the scene images, and the low-quality analytic mode is a low-light analytic mode that is different from the high-quality analytic mode and parses the video images for object activity detection without removing shadows from the scene images.

11. The system of claim 10, wherein the processor determines the amount of scene illumination by determining that a weather condition of the scene is sunny, raining, snowing, foggy, cloudy or nighttime, and wherein the threshold amount of illumination is sunny.

12. The system of claim 11, wherein the high-quality analytic mode is an object tracking based mode, and the low quality mode is a non-object tracking based mode.

13. The system of claim 12, wherein the threshold level of quality is a threshold level of background activity;

and

wherein the processor, when executing the program instructions stored on the computer readable storage medium via the computer readable memory, further;

applies background subtraction to the video stream input to detect the foreground object activity as a difference between current frame image information and static information of a contextual background of video input images from the video stream input; and

determines a level of the detected foreground object activity; and

wherein the processor selects the high-quality analytic mode in response to the determined level of the detected foreground object activity being lower than the threshold level of background activity, and selects the low-quality analytic mode in response to the determined level of the detected foreground object activity not being lower than the threshold level of background activity.

14. The system of claim 13, wherein the processor, when executing the program instructions stored on the computer readable storage medium via the computer readable memory, applies the high-quality, object tracking based analytic mode by:

using appearance-based modeling to group foreground regions of each video frame into a plurality of connected foreground region components that are each described by a bounding box; and

for each successive frame, attempting to associate each of the connected foreground region components with an existing foreground region component object track as a function of a distance between each of the foreground region components and each currently active existing foreground region component object track.

15. The system of claim 13, wherein the processor, when executing the program instructions stored on the computer readable storage medium via the computer readable memory, applies the low-quality, non-object tracking based analytic mode by:

defining a virtual boundary line of a region of interest within the video input images;

establishing a plurality of overlapping ground patch regions along each side of the virtual boundary line;

analyzing in real-time each of the ground patch regions in the video input images data to identifying attributes of objects detected in the region of interest; and

determining current appearance features of foreground objects identified within the region of interest.

16. A computer program product for multi-mode video event indexing, the computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising instructions that, when executed by a computer processor, cause the computer processor to;

determine a quality of object distinctiveness with respect to images from a video stream input;

in response to the determined level of detected quality of object distinctiveness meeting a threshold level of quality, select a high-quality analytic mode from a plurality of video analytics modes and apply the high-quality analytic mode to the video input images to determine object activity within the video input images; and

in response to the determined level of detected quality of object distinctiveness not meeting the threshold level of quality, select a low-quality analytic mode from the plurality of video analytics modes and apply the low-quality analytic mode to the video input images to determine object activity within the video input images; and

wherein the determining the quality of object distinctiveness with respect to the video input images comprises determining an amount of scene illumination in the video input images, the threshold level of quality is an amount of scene illumination, the high-quality analytic mode is a strong-shadow analytic mode that provides for object activity detection by removing strong shadows from the scene images, and the low-quality analytic mode is a low-light analytic mode that is different from the high-quality analytic mode and parses the video images for object activity detection without removing shadows from the scene images.

17. The computer program product of claim 16, wherein the high-quality analytic mode is an object tracking based mode, and the low quality mode is a non-object tracking based mode.

18. The computer program product of claim 17, wherein the threshold level of quality is a threshold level of background activity;

and

wherein the computer-readable program code instructions, when executed by the computer processor, further cause the computer processor to;

apply background subtraction to the video stream input to detect the foreground object activity as a difference between current frame image information and static information of a contextual background of video input images from the video stream input; and

determine a level of the detected foreground object activity; and

wherein the processor selects the high-quality analytic mode in response to the determined level of the detected foreground object activity being lower than the threshold level of background activity, and selects the low-quality analytic mode in response to the determined level of the detected foreground object activity not being lower than the threshold level of background activity.

19. The computer program product of claim 18, wherein the computer-readable program code instructions, when executed by the computer processor, further cause the computer processor to apply the high-quality, object tracking based analytic mode by:

using appearance-based modeling to group foreground regions of each video frame into a plurality of connected foreground region components that are each described by a bounding box; and

for each successive frame, attempting to associate each of the connected foreground region components with an existing foreground region component object track as a function of a distance between each of the foreground region components and each currently active existing foreground region component object track.

20. The computer program product of claim 18, wherein the computer-readable program code instructions, when executed by the computer processor, further cause the computer processor to apply the low-quality, non-object tracking based analytic mode by:

defining a virtual boundary line of a region of interest within the video input images;

establishing a plurality of overlapping ground patch regions along each side of the virtual boundary line;

analyzing in real-time each of the ground patch regions in the video input images data to identifying attributes of objects detected in the region of interest; and

determining current appearance features of foreground objects identified within the region of interest.