Foreground detector for video analytics system

• US 9,349,054 B1
• Filed: 10/29/2014
• Issued: 05/24/2016
• Est. Priority Date: 10/29/2014
• Status: Active Grant

• Current Assignee: Intellective Ai, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Cobb, Wesley Kenneth, Saitwal, Kishor Adinath, RISINGER, Lon
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US14/526,756
• Publication Number:
  US 20160125245A1
• Time in Patent Office: 573 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats

Abstract

Techniques are disclosed for creating a background model of a scene using both a pixel based approach and a context based approach. The combined approach provides an effective technique for segmenting scene foreground from background in frames of a video stream. Further, this approach can scale to process large numbers of camera feeds simultaneously, e.g., using parallel processing architectures, while still generating an accurate background model. Further, using both a pixel based approach and context based approach ensures that the video analytics system can effectively and efficiently respond to changes in a scene, without overly increasing computational complexity. In addition, techniques are disclosed for updating the background model, from frame-to-frame, by absorbing foreground pixels into the background model via an absorption window, and dynamically updating background/foreground thresholds.

Claims

1. A computer-implemented method for generating a background model of a scene depicted in a sequence of video frames captured by a video camera, the method comprising:

receiving a video frame, wherein the video frame includes one or more appearance values for each of a plurality of pixels;

classifying each pixel as depicting either foreground or background by comparing the one or more appearance values of each pixel to a background model of the scene; and

performing one or more context based evaluations on one or more of the pixels classified as depicting foreground, wherein the context based evaluations selectively reclassifies one or more of the pixels as depicting foreground or background based on the classification of other pixels in the video frame as depicting either foreground or background.

2. The method of claim 1, wherein comparing the one or more appearance values of each pixel to a background model of the scene comprises:

for each pixel of the plurality of pixels;

determining a distance between the pixel and a corresponding pixel in the background model,

upon determining the distance exceeds a threshold, classifying the pixel as depicting foreground, and

upon determining the distance does not exceed the threshold, classifying the pixel as depicting background.

3. The method of claim 2, wherein the threshold is dynamically determined for each pixel based on a camera noise model.

4. The method of claim 2, wherein the background model provides, for each pixel in the video frame, a distribution modeling each of the one or more appearance values for the given pixel in the sequence of video frames and a respective mean and variance of each distribution.

5. The method of claim 4, further comprising, for each pixel classified as depicting background, updating the mean and variance of the distribution modeling the appearance values of that pixel based on the appearance values of the pixel in the video frame.

6. The method of claim 4, further comprising, for each pixel classified as depicting foreground, updating the mean of the distribution modeling the appearance values of that pixel based on the appearance values of the pixel in the video frame.

7. The method of claim 4, wherein the distance is a Mahalanobis distance.

8. The method of claim 1, wherein one of the context based evaluations comprises:

dilating one or more of the pixels classified as depicting foreground and reclassifying, as foreground, one or more of the pixels classified as background with overlap with a specified number of the dilated pixels.

9. The method of claim 1, wherein one of the context based evaluations comprises eroding one or more contiguous groups of foreground pixels by reclassifying the pixels in the one or more contiguous groups as background.

10. The method of claim 1, wherein one of the context based evaluations comprises:

identifying a contiguous region of pixels classified as foreground;

determining a normalized cross correlation score between the contiguous region of pixels in the video game classified as foreground and a corresponding region of pixels in a background image of the background model; and

upon determining the normalized cross correlation score satisfies a specified threshold, reclassifying the pixels in the contiguous region of pixels as depicting background.

11. A computer-readable storage medium containing a program, which when executed on a processor, performs an operation for generating a background model of a scene depicted in a sequence of video frames captured by a video camera, the operation comprising:

receiving a video frame, wherein the video frame includes one or more appearance values for each of a plurality of pixels;

classifying each pixel as depicting either foreground or background by comparing the one or more appearance values of each pixel to a background model of the scene; and

performing one or more context based evaluations on one or more of the pixels classified as depicting foreground, wherein the context based evaluations selectively reclassifies one or more of the pixels as depicting foreground or background based on the classification of other pixels in the video frame as depicting either foreground or background.

12. The computer-readable storage medium of claim 11, wherein comparing the one or more appearance values of each pixel to a background model of the scene comprises:

for each pixel of the plurality of pixels;

determining a distance between the pixel and a corresponding pixel in the background model,

upon determining the distance exceeds a threshold, classifying the pixel as depicting foreground, and

upon determining the distance does not exceed the threshold, classifying the pixel as depicting background.

13. The computer-readable storage medium of claim 12, wherein the threshold is dynamically determined for each pixel based on a camera noise model.

14. The computer-readable storage medium of claim 12, wherein the background model provides, for each pixel in the video frame, a distribution modeling each of the one or more appearance values for the given pixel in the sequence of video frames and a respective mean and variance of each distribution.

15. The computer-readable storage medium of claim 14, wherein the operation further comprises, for each pixel classified as depicting background, updating the mean and variance of the distribution modeling the appearance values of that pixel based on the appearance values of the pixel in the video frame.

16. The computer-readable storage medium of claim 14, wherein the operation further comprises, for each pixel classified as depicting foreground, updating the mean of the distribution modeling the appearance values of that pixel based on the appearance values of the pixel in the video frame.

17. The computer-readable storage medium of claim 14, wherein the distance is a Mahalanobis distance.

18. The computer-readable storage medium of claim 11, wherein one of the context based evaluations comprises:

dilating one or more of the pixels classified as depicting foreground and reclassifying, as foreground, one or more of the pixels classified as background with overlap with a specified number of the dilated pixels.

19. The computer-readable storage medium of claim 11, wherein one of the context based evaluations comprises eroding one or more contiguous groups of foreground pixels by reclassifying the pixels in the one or more contiguous groups as background.

20. The computer-readable storage medium of claim 11, wherein one of the context based evaluations comprises:

identifying a contiguous region of pixels classified as foreground;

determining a normalized cross correlation score between the contiguous region of pixels in the video game classified as foreground and a corresponding region of pixels in a background image of the background model; and

upon determining the normalized cross correlation score satisfies a specified threshold, reclassifying the pixels in the contiguous region of pixels as depicting background.

21. A system, comprising:

a video input source configured to provide a sequence of video frames, each depicting a scene;

a central processing unit (CPU);

a graphics processing unit (GPU); and

a memory containing a program, which, when executed on the CPU and GPU is configured to perform an operation for generating a background model of a scene depicted in a sequence of video frames captured by a video camera, the operation comprising;

receiving a video frame, wherein the video frame includes one or more appearance values for each of a plurality of pixels,

classifying each pixel as depicting either foreground or background by comparing the one or more appearance values of each pixel to a background model of the scene, and

performing one or more context based evaluations on one or more of the pixels classified as depicting foreground, wherein the context based evaluations selectively reclassifies one or more of the pixels as depicting foreground or background based on the classification of other pixels in the video frame as depicting either foreground or background.

22. The system medium of claim 21, wherein comparing the one or more appearance values of each pixel to a background model of the scene comprises:

for each pixel of the plurality of pixels;

determining a distance between the pixel and a corresponding pixel in the background model,

upon determining the distance exceeds a threshold, classifying the pixel as depicting foreground, and

upon determining the distance does not exceed the threshold, classifying the pixel as depicting background.

23. The system of claim 22, wherein the threshold is dynamically determined for each pixel based on a camera noise model.

24. The system of claim 22, wherein the background model provides, for each pixel in the video frame, a distribution modeling each of the one or more appearance values for the given pixel in the sequence of video frames and a respective mean and variance of each distribution.

25. The system of claim 24, wherein the operation further comprises, for each pixel classified as depicting background, updating the mean and variance of the distribution modeling the appearance values of that pixel based on the appearance values of the pixel in the video frame.

26. The system of claim 24, wherein the operation further comprises, for each pixel classified as depicting foreground, updating the mean of the distribution modeling the appearance values of that pixel based on the appearance values of the pixel in the video frame.

27. The system of claim 24, wherein the distance is a Mahalanobis distance.

28. The system of claim 21, wherein one of the context based evaluations comprises:

dilating one or more of the pixels classified as depicting foreground and reclassifying, as foreground, one or more of the pixels classified as background with overlap with a specified number of the dilated pixels.

29. The system of claim 21, wherein one of the context based evaluations comprises eroding one or more contiguous groups of foreground pixels by reclassifying the pixels in the one or more contiguous groups as background.

30. The system of claim 21, wherein one of the context based evaluations comprises:

identifying a contiguous region of pixels classified as foreground;

determining a normalized cross correlation score between the contiguous region of pixels in the video game classified as foreground and a corresponding region of pixels in a background image of the background model; and

upon determining the normalized cross correlation score satisfies a specified threshold, reclassifying the pixels in the contiguous region of pixels as depicting background.