Systems and methods for detecting and tracking objects in a video stream

• US 9,501,701 B2
• Filed: 02/02/2015
• Issued: 11/22/2016
• Est. Priority Date: 01/31/2014
• Status: Expired due to Fees

• Current Assignee: Charles Stark Draper Laboratory Incorporated
• Original Assignee: Charles Stark Draper Laboratory Incorporated
• Inventors: Wood, Richard J., Irvine, John M., Reed, David
• Primary Examiner(s): Ansari, Tahmina
• Application Number:
  US14/611,788
• Publication Number:
  US 20150220789A1
• Time in Patent Office: 659 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

This disclosure provides systems, methods, and apparatus for analyzing a video are disclosed. An input video signal can be received and a quality metric associated with the input video signal can be determined. A parameter to be used in a first object tracking process can be selected, based in part on the quality metric associated with the input video signal. A background model can be extracted from the input video signal based on an online median estimate of the intensity values of the pixels in the image frames. The online median estimate can be calculated using a quantile function.

Claims

1. A system for tracking foreground objects represented in an input video signal, the system comprising:

an image quality assessment module configured to determine at least one image quality metric associated with the input video signal, wherein the image quality metric includes at least one of a jitter metric representing a statistical description of the autocorrelation across sequential frames of the input video signal, a metric relating to the sharpness of edges of background and foreground objects within the input video signal, a metric associated with the local clutter present in the video signal, and an entropy metric;

a parameter selection module configured to select at least one parameter of a background modeling process based on the at least one image quality metric;

a background identification module configured to update a background model for the input video signal using the at least one parameter of the background modeling process;

an object detection module configured to identify at least one foreground object present in the input video signal, wherein the at least one object is not included in the background model; and

an object tracking module configured to track a position of the at least one object within frames of the input video signal, wherein the background model includes an estimate of a median pixel value for each pixel corresponding to a scene imaged in the input video signal and is further configured to estimate the median pixel value for each pixel in the imaged scene using a quantile function.

2. The system of claim 1, wherein, for each pixel corresponding to the scene, the median pixel value is estimated by updating a stored median estimate for the pixel value based on the difference between a value of a corresponding pixel in a current image frame and the stored median estimate for the pixel.

3. The system of claim 1, wherein the background identification module is further configured to update the stored median value for each pixel at a frequency that is less than a frame rate of the input video signal.

4. The system of claim 1, wherein the parameter selection module is further configured to:

reduce a value of a parameter corresponding to a frequency with which the background identification module updates the stored median value for each pixel, based on an increase in the value of an image quality metric associated with a noise level of the input video signal.

5. The system of claim 1, wherein the object detection module is further configured to:

calculate, for each pixel of a current image frame corresponding to a pixel in the imaged scene, a first difference between the pixel value and the stored median estimate for the pixel;

compare the first difference to a first threshold; and

identify the pixel as representing a portion of a background of the imaged scene, responsive to determining that the first difference does not exceed the first threshold.

6. The system of claim 5, wherein the object detection module is further configured to:

calculate, responsive to a determination that the first difference exceeds the first threshold value, a second difference between a value of a neighboring pixel and the stored median estimate for the neighboring pixel; and

identify the pixel as representing an object not included within the background of the imaged scene, responsive to a determination that at least one of the first difference and the second difference exceeds a second threshold value.

7. The system of claim 6, wherein the parameter selection module is further configured to:

calculate the first threshold value by multiplying a coefficient parameter by a quantile range associated with each pixel of the current image frame; and

modify a value of at least one of the coefficient parameter and the quantile range used for calculating the first threshold value based on a change in the value of an image quality metric associated with a noise level or jitter level in the input video signal.

8. The system of claim 1, wherein the image quality metric determined by the image quality assessment module includes at least one of a jitter metric and a local clutter metric.

9. The system of claim 1, further comprising a video pre-processing module configured to:

select at least one image filtering process based on the at least one image quality metric; and

apply the at least one selected image filtering process to the input video signal prior to the background identification module updating the background based on the input video signal.

18. The system of claim 1, wherein the quantile function used to estimate the median pixel value for each pixel in the imaged scene is:

Q_t+1=Q_t+ε

(sgn(Pt−

Q_t))

wherein;

Q is the quantile estimate being calculated;

P_t is a current pixel intensity value;

sgn is a sign function;

and ε

is a learning rate.

19. The system of claim 18, wherein the learning rate ε

is defined as;

ε

=1.5C/k²

wherein k is a number of frames observed and C is a cumulative absolute deviation defined as;

C_t+1=C_t+|P_t−

Q_t|.

10. A system for detecting foreground objects represented in an input video signal, the system comprising:

a background identification module configured to generate a background model for the input video signal, wherein;

the background model includes an estimate of a median pixel value for each pixel corresponding to a scene imaged in the input video signal, wherein the estimate of the median pixel value for each pixel in the image frame of the input video signal is calculated using a quantile function; and

for each pixel of a current image frame corresponding to the scene, the median pixel value is estimated by updating a stored median estimate for the pixel value based on the difference between a value of a corresponding pixel in a current image frame and the stored median estimate for the pixel; and

an object detection module configured to;

calculate, for each pixel of the current image frame corresponding to an pixel in the imaged scene, a first difference between the pixel value and the stored median estimate for the pixel;

compare the first difference to a first threshold; and

identify the pixel as representing a portion of a background of the imaged scene, responsive to determining that the first difference does not exceed the first threshold.

11. The system of claim 10, wherein the object detection module is further configured to:

calculate, responsive to a determination that the first difference exceeds the first threshold value, a second difference between a value of a neighboring pixel and the stored median estimate for the neighboring pixel; and

identify the pixel as representing an object not included within the background of the imaged scene, responsive to a determination that at least one of the first difference and the second difference exceeds a second threshold value.

12. The system of claim 10, wherein the background identification module is further configured to update the stored median value for each pixel at a frequency that is less than a frame rate of the input video signal.

13. The system of claim 12, wherein a sampling interval for updating the stored median value for each pixel is in the range of about 100 milliseconds to about 300 milliseconds.

20. The system of claim 10, wherein the quantile function used to estimate the median pixel value for each pixel in the imaged scene is:

Q_t+1=Q_t+ε

(sgn(P_t−

Q_t))

wherein;

Q is the quantile estimate being calculated;

P_t is a current pixel intensity value;

sgn is a sign function; and

ε

is a learning rate.

21. The system of claim 20, wherein the learning rate E is defined as:

ε

=1.5C/k²

wherein k is a number of frames observed and C is a cumulative absolute deviation defined as;

C_t+1=C_t+|P_t−

Q_t|.

14. A system for tracking foreground objects represented in an input video signal, the system comprising:

an image quality assessment module configured to determine an image quality metric associated with the input video signal, wherein the image quality metric includes at least one of a jitter metric representing a statistical description of the autocorrelation across sequential frames of the input video signal and a local clutter metric;

a parameter selection module configured to determine a set of parameters for a first tracking algorithm, the set of parameters including at least one parameter selected based on the at least one image quality metric;

an object detection module configured to identify at least one foreground object present in the input video signal; and

an object tracking module configured to track a position of the at least one object within frames of the input video signal, based on the first object tracking algorithm and the set of parameters.

15. The system of claim 14, wherein the parameter selection module is further configured to predict a performance metric based on the set of parameters, the first object tracking algorithm, and the at least one image quality metric.

16. The system of claim 15, wherein the parameter selection module is further configured to update the at least one parameter of the first object tracking algorithm used by the object tracking module, based on a determination that the performance metric is below a threshold value.

17. The system of claim 15, wherein the parameter selection module is further configured to select a second object tracking algorithm to be used by the object tracking module, based on a determination that the performance metric is below a threshold value.

22. The system of claim 14, wherein the at least one parameter includes at least one of motion sensitivity, a number of frames used in a frame differencing process, a skipped frame parameter, a resurrection window parameter, and an advanced histogram parameter.