MULTI-MODE VIDEO EVENT INDEXING

• US 20120045090A1
• Filed: 08/17/2010
• Published: 02/23/2012
• Est. Priority Date: 08/17/2010
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated (Kyndryl Holdings, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Bobbitt, Russell P., Brown, Lisa M., Feris, Rogerio S., Hampapur, Arun, Zhai, Yun
• Granted Patent: US 8,532,336 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Multi-mode video event indexing includes determining a quality of object distinctiveness with respect to images from a video stream input. A high-quality analytic mode is selected from multiple modes and applied to video input images via a hardware device to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality, else a low-quality analytic mode is selected and applied to the video input images via a hardware device to determine object activity within the video input images, wherein the low-quality analytic mode is different from the high-quality analytic mode.

Claims

1. A method for multi-mode video event indexing, the method comprising:

determining a quality of object distinctiveness with respect to images from a video stream input;

selecting a high-quality analytic mode from a plurality of video analytics modes and applying the high-quality analytic mode to the video input images via a hardware device to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality; and

selecting a low-quality analytic mode from the plurality of video analytics modes and applying the low-quality analytic mode to the video input images via a hardware device to determine object activity within the video input images if the determined level of detected quality of object distinctiveness does not meet the threshold level of quality, wherein the low-quality analytic mode is different from the high-quality analytic mode.

2. The method of claim 1, wherein the determining the quality of object distinctiveness with respect to the video input images comprises determining whether a camera providing the video input is moving or stationary relative to a scene within the video input, and the threshold level of quality of the scene is stationary.

3. The method of claim 1, wherein the determining the quality of object distinctiveness with respect to the video input images comprises determining an amount of scene illumination in the video input images, and the threshold level of quality is an amount of scene illumination;

wherein the high-quality analytic mode is a strong-shadow analytic mode that provides for object activity detection by removing strong shadows from the scene images; and

wherein the low-quality analytic mode is a low-light analytic mode that parses the video images for object activity detection without removing shadows from the scene images.

4. The method of claim 3, wherein the determining the amount of scene illumination comprises determining that a weather condition of the scene is sunny, raining, snowing, foggy, cloudy or nighttime, and wherein the threshold amount of illumination is sunny.

5. The method of claim 1, further comprising:

applying background subtraction via a programmable hardware device to the video input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the image;

wherein the determining the quality of object distinctiveness comprises determining a level of the detected foreground object activity;

wherein the selecting and applying the high-quality analytic mode comprises selecting and applying an object tracking-based mode to the detected foreground object activity of the video input to track a foreground object;

wherein the selecting and applying the low-quality analytic mode comprises selecting and applying a non-object tracking-based mode to the detected foreground object activity of the video input to determine object movement from extracted foreground object appearance attributes without tracking the foreground object; and

further comprising writing results of the tracking the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index.

6. The method of claim 5, wherein the determining the level of the detected foreground object activity comprises:

determining a density of the foreground relative to an entirety of the image; and

wherein the threshold level of activity is a density value.

7. The method of claim 6, wherein the selecting the object tracking-based mode further comprises:

determining if a first density of the foreground relative to the entirety of the image is greater than or equal to the threshold density value at a first time;

determining if a second density of the foreground relative to the entirety of the image is less than a minimum value that is less than the threshold density value at a second time subsequent to the first time;

determining an elapsed time between the first and the second times;

comparing the elapsed time to a stability time period; and

selecting the object tracking-based mode if the elapsed time is less than the stability time period.

8. The method of claim 6, wherein the selecting the object tracking-based mode further comprises:

determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity;

comparing the time duration to a specified time period;

selecting the object tracking-based mode if the time duration is less than the specified time period.

9. The method of claim 8, wherein the density of the foreground is an average of a plurality of individual densities over the specified time period.

10. The method of claim 9, further comprising:

determining the average of the plurality of individual densities over the specified time period by weighting a more recent density higher than an older density.

11. A computer system for multi-mode video event indexing, the computer system comprising:

a processing unit, computer readable memory and a computer readable storage system;

first program instructions to determine a quality of object distinctiveness with respect to images from a video stream input;

second program instructions to select a high-quality analytic mode from a plurality of video analytics modes and apply the high-quality analytic mode to the video input images to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality; and

third program instructions to select a low-quality analytic mode from the plurality of video analytics modes and apply the low-quality analytic mode to the video input images to determine object activity within the video input images if the determined level of detected quality of object distinctiveness does not meet the threshold level of quality, wherein the low-quality analytic mode is different from the high-quality analytic mode;

wherein the first, second and third program instructions are stored on the computer readable storage system for execution by the processing unit via the computer readable memory.

12. The computer system of claim 11, wherein the first program instructions are further to determine the quality of object distinctiveness with respect to the video input images by determining an amount of scene illumination in the video input images, and the threshold level of quality of the scene is an amount of illumination;

wherein the high-quality analytic mode is a strong-shadow analytic mode that provides for object activity detection by removing strong shadows from the scene images; and

wherein the low-quality analytic mode is a low-light analytic mode that that parses the video images for object activity detection without removing shadows from the scene images.

13. The computer system of claim 11, further comprising:

fourth program instructions to apply background subtraction to the video input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the image;

wherein the first program instructions are further to determine the quality of object distinctiveness by determining a level of the detected foreground object activity;

wherein the second program instructions are to select and apply an object tracking-based mode to the detected foreground object activity of the video input to track a foreground object;

wherein the third program instructions are to select and apply a non-object tracking-based mode and apply the non-object tracking-based mode to the detected foreground object activity of the video input to determine object movement from extracted foreground object appearance attributes without tracking the foreground object; and

fifth program instructions to write results of the tracking the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index; and

wherein the fourth and fifth program instructions are stored on the computer readable storage system for execution by the processing unit via the computer readable memory.

14. The computer system of claim 13, wherein the first program instructions are further to determine the level of the detected foreground object activity as a density of the foreground relative to an entirety of the image;

and

wherein the threshold level of activity is a density value.

15. The computer system of claim 14, wherein the second program instructions are to select and apply the object tracking-based mode by:

determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity;

comparing the time duration to a specified time period; and

selecting the object tracking-based mode if the time duration is less than the specified time period.

16. A computer program product for multi-mode video event indexing, the computer program product comprising:

a computer readable storage medium;

first program instructions to determine a quality of object distinctiveness with respect to images from a video stream input;

second program instructions to select a high-quality analytic mode from a plurality of video analytics modes and apply the high-quality analytic mode to the video input images to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality; and

third program instructions to select a low-quality analytic mode from the plurality of video analytics modes and apply the low-quality analytic mode to the video input images to determine object activity within the video input images if the determined level of detected quality of object distinctiveness does not meet the threshold level of quality, wherein the low-quality analytic mode is different from the high-quality analytic mode; and

wherein the first, second and third program instructions are stored on the computer readable storage medium.

17. The computer program product of claim 16, wherein the first program instructions are further to determine the quality of object distinctiveness with respect to the video input images by determining an amount of scene illumination in the video input images, and the threshold level of quality of the scene is an amount of illumination;

wherein the high-quality analytic mode is a strong-shadow analytic mode that provides for object activity detection by removing strong shadows from the scene images; and

wherein the low-quality analytic mode is a low-light analytic mode that parses the video images for object activity detection without removing shadows from the scene images.

18. The computer program product of claim 16, further comprising:

fourth program instructions to apply background subtraction to the video input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the image;

wherein the first program instructions are further to determine the quality of object distinctiveness by determining a level of the detected foreground object activity;

wherein the second program instructions are to select and apply an object tracking-based mode to the detected foreground object activity of the video input to track a foreground object;

wherein the third program instructions are to select and apply a non-object tracking-based mode to the detected foreground object activity of the video input to determine object movement from extracted foreground object appearance attributes without tracking the foreground object; and

fifth program instructions to write results of the tracking the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index; and

wherein the fourth and fifth program instructions are stored on the computer readable storage medium.

19. The computer program product of claim 18, wherein the first program instructions are further to determine the level of the detected foreground object activity as a density of the foreground relative to an entirety of the image;

and

wherein the threshold level of activity is a density value.

20. The computer program product of claim 19, wherein the second program instructions are to select and apply the object tracking-based mode by:

determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity;

comparing the time duration to a specified time period; and

selecting the object tracking-based mode if the time duration is less than the specified time period.

21. A service for multi-mode video event indexing, the service comprising:

providing a computer infrastructure that;

determines a quality of object distinctiveness with respect to images from a video stream input;

selects a high-quality analytic mode from a plurality of video analytics modes and applies the high-quality analytic mode to the video input images to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality; and

selects a low-quality analytic mode from the plurality of video analytics modes and applies the low-quality analytic mode to the video input images to determine object activity within the video input images if the determined level of detected quality of object distinctiveness does not meet the threshold level of quality, wherein the low-quality analytic mode is different from the high-quality analytic mode.

22. The service of claim 21, wherein the computer infrastructure further determines the quality of object distinctiveness with respect to the video input images by determining an amount of scene illumination in the video input images, and the threshold level of quality of the scene is an amount of illumination;

wherein the high-quality analytic mode is a strong-shadow analytic mode that provides for object activity detection by removing strong shadows from the scene images; and

wherein the low-quality analytic mode is a low-light analytic mode that that parses the video images for object activity detection without removing shadows from the scene images.

23. The service of claim 21, the computer infrastructure further to:

apply background subtraction to the video input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the image;

determine the quality of object distinctiveness by determining a level of the detected foreground object activity;

select and apply an object tracking-based mode to the detected foreground object activity of the video input to track a foreground object;

select and apply a non-object tracking-based mode to the detected foreground object activity of the video input to determine object movement from extracted foreground object appearance attributes without tracking the foreground object; and

write results of the tracking the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index.

24. The service of claim 23, the computer infrastructure further to determine the level of the detected foreground object activity as a density of the foreground relative to an entirety of the image;

and

wherein the threshold level of activity is a density value.

25. The service of claim 24, the computer infrastructure further to select and apply the object tracking-based mode by:

determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity;

comparing the time duration to a specified time period; and

selecting the object tracking-based mode if the time duration is less than the specified time period.