Multi-mode video event indexing

• US 9,123,129 B2
• Filed: 08/21/2013
• Issued: 09/01/2015
• Est. Priority Date: 08/17/2010
• Status: Expired due to Fees

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Bobbitt, Russell P., Brown, Lisa M., Feris, Rogerio S., Hampapur, Arun, Zhai, Yun
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US13/971,984
• Publication Number:
  US 20140050356A1
• Time in Patent Office: 741 Days
• Field of Search: 382/103, 382/107, 348143-160, 348169-172
• US Class Current: 1/1
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Multi-mode video event indexing includes determining a quality of object distinctiveness with respect to images from a video stream input. A high-quality analytic mode is selected from multiple modes and applied to video input images via a hardware device to determine object activity within the video input images if the determined level of detected quality of object distinctiveness meets a threshold level of quality, else a low-quality analytic mode is selected and applied to the video input images via a hardware device to determine object activity within the video input images, wherein the low-quality analytic mode is different from the high-quality analytic mode.

Claims

1. A method for multi-mode video event indexing, the method comprising:

applying background subtraction to video input images from a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the video input images from the video stream input;

determining by a processor a level of density of the foreground object activity detected with respect to the video input images from the video stream input relative to an entirety of the image;

in response to the determined level of density of the detected foreground object activity meeting a threshold density value, the processor selecting from a plurality of video analytics modes and applying an object tracking-based analytic mode to the detected foreground object activity of the video input images to track a foreground object; and

in response to the determined level of density of the detected foreground object activity not meeting the threshold density value, the processor selecting from the plurality of video analytics modes and applying a non-object tracking-based analytic mode to the detected foreground object activity of the video input images to determine object movement from extracted foreground object appearance attributes without tracking the foreground object.

2. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processor, a computer readable memory and a computer readable storage medium, wherein the computer readable program code is embodied on the computer readable storage medium and comprises instructions for execution by the processor via the computer readable memory that cause the processor to perform the steps of applying the background subtraction to the video stream input images to detect the foreground object activity as the difference between the current frame image information and the static information of the contextual background of the video input images from the video stream input, determining the level of density of foreground object activity detected with respect to the video input images from the video stream input, selecting from the plurality of video analytics modes and applying the object tracking-based analytic mode to the detected foreground object activity of the video input images to track a foreground object in response to the determined level of density of the detected foreground object activity meeting the threshold density value, and selecting from the plurality of video analytics modes and applying the non-object tracking-based analytic mode to the detected foreground object activity of the video input images to determine object movement from extracted foreground object appearance attributes without tracking the foreground object in response to the determined level of density of the detected foreground object activity not meeting the threshold density value.

3. The method of claim 1, further comprising:

writing results of the tracking the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index.

4. The method of claim 1, wherein the selecting the object tracking-based mode further comprises:

determining if a first density of the foreground relative to the entirety of the image is greater than or equal to the threshold density value at a first time;

determining if a second density of the foreground relative to the entirety of the image is less than a minimum value that is less than the threshold density value at a second time subsequent to the first time;

determining an elapsed time between the first and the second times;

comparing the elapsed time to a stability time period; and

selecting the object tracking-based mode if the elapsed time is less than the stability time period.

5. The method of claim 4, wherein the selecting the object tracking-based mode further comprises:

determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity;

comparing the time duration to a specified time period;

selecting the object tracking-based mode if the time duration is less than the specified time period.

6. The method of claim 5, wherein the density of the foreground is an average of a plurality of individual densities over the specified time period.

7. The method of claim 6, further comprising:

determining the average of the plurality of individual densities over the specified time period by weighting a more recent density higher than an older density.

8. A system for multi-mode video event indexing, the computer system comprising:

a processor;

a computer-readable memory;

a computer-readable storage medium;

fourth program instructions to apply background subtraction to video input images from a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the video input images from the video stream input;

first program instructions to determine a level of density of the foreground object activity detected with respect to the video input images from the video stream input relative to an entirety of the image;

second program instructions to select from a plurality of video analytics modes and apply an object tracking-based mode to the detected foreground object activity of the video input to track a foreground object, in response to the determined level of density of the detected foreground object activity meeting a threshold density value; and

third program instructions to select from the plurality of video analytics modes and apply a non-object tracking-based mode to the detected foreground object activity of the video input images to determine object movement from extracted foreground object appearance attributes without tracking the foreground object, in response to the determined level of density of the detected foreground object activity not meeting the threshold density value; and

wherein the first, second, third and fourth program instructions are stored on the computer-readable storage medium for execution by the processor via the computer-readable memory.

9. The system of claim 8, further comprising:

fifth program instructions stored on the computer-readable storage medium for execution by the processor via the computer-readable memory to write results of the tracking the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index.

10. The system of claim 8, wherein the second program instructions are to select and apply the object tracking-based mode by:

determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity;

comparing the time duration to a specified time period; and

selecting the object tracking-based mode if the time duration is less than the specified time period.

11. The system of claim 10, wherein the second program instructions are to select and apply the object tracking-based mode by:

determining if a first density of the foreground relative to the entirety of the image is greater than or equal to the threshold density value at a first time;

determining if a second density of the foreground relative to the entirety of the image is less than a minimum value that is less than the threshold density value at a second time subsequent to the first time;

determining an elapsed time between the first and the second times;

comparing the elapsed time to a stability time period; and

selecting the object tracking-based mode in response to determining that the elapsed time is less than the stability time period.

12. The system of claim 11, wherein the second program instructions are further to determine the density of the foreground as an average of a plurality of individual densities over the specified time period.

13. The system of claim 12, wherein the second program instructions are further to determine the average of the plurality of individual densities over the specified time period by weighting a more recent density higher than an older density.

14. A computer program product for multi-mode video event indexing, the computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising instructions for execution by a computer processor that cause the computer processor to;

apply background subtraction to video input images from a video stream input to detect foreground object activity as a difference between current frame image information and static information of a contextual background of the video input images from the video stream input;

determine a level of density of the foreground object activity detected with respect to the video input images from the video stream input relative to an entirety of the image;

select from a plurality of video analytics modes and apply an object tracking-based mode to the detected foreground object activity of the video input to track a foreground object, in response to the determined level of density of the detected foreground object activity meeting a threshold density value; and

select from the plurality of video analytics modes and apply a non-object tracking-based mode to the detected foreground object activity of the video input images to determine object movement from extracted foreground object appearance attributes without tracking the foreground object, in response to the determined level of density of the detected foreground object activity not meeting the threshold density value.

15. The computer program product of claim 14, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to:

write results of the tracking the foreground object and the object movement determination from the extracted foreground object appearance attributes to an index.

16. The computer program product of claim 14, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to select and apply the object tracking-based mode by:

determining a time duration that the density of the foreground is greater than or equal to the threshold level of activity;

comparing the time duration to a specified time period; and

selecting the object tracking-based mode if the time duration is less than the specified time period.

17. The computer program product of claim 16, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to select and apply the object tracking-based mode by:

determining if a first density of the foreground relative to the entirety of the image is greater than or equal to the threshold density value at a first time;

determining if a second density of the foreground relative to the entirety of the image is less than a minimum value that is less than the threshold density value at a second time subsequent to the first time;

determining an elapsed time between the first and the second times;

comparing the elapsed time to a stability time period; and

selecting the object tracking-based mode in response to determining that the elapsed time is less than the stability time period.

18. The computer program product of claim 17, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to determine the density of the foreground as an average of a plurality of individual densities over the specified time period.

19. The computer program product of claim 18, wherein the computer-readable program code instructions for execution by the computer processor further cause the computer processor to determine the average of the plurality of individual densities over the specified time period by weighting a more recent density higher than an older density.