VIDEO SEGMENTATION USING STATISTICAL PIXEL MODELING

• US 20090297023A1
• Filed: 03/01/2007
• Published: 12/03/2009
• Est. Priority Date: 03/23/2001
• Status: Active Grant

• Current Assignee: Avigilon Fortress Corporation
• Original Assignee: ObjectVideo Incorporated (Alarm.com Holdings, Inc.)
• Inventors: Rasheed, Zeeshan, Myers, Gary W., Haering, Niels, Zhang, Zhong, Venetianer, Peter L., Lipton, Alan J., Javed, Omar, Yin, Wei Hong
• Granted Patent: US 8,457,401 B2
• Time in Patent Office: Days
• US Class Current: 382/164
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method for segmenting video data into foreground and background (324) portions utilizes statistical modeling of the pixels Λ statistical model of the background is built for each pixel, and each pixel in an incoming video frame is compared (326) with the background statistical model for that pixel. Pixels are determined to be foreground or background based on the comparisons. The method for segmenting video data may be further incorporated into a method for implementing an intelligent video surveillance system The method for segmenting video data may be implemented in hardware.

Claims

1. Circuitry adapted to perform a two-pass method of video segmentation for differentiating between foreground and background portions of video, the method comprising the steps of:

obtaining a frame sequence from an input video stream;

executing a first-pass method for each frame of the frame sequence, the first-pass method comprising the steps of;

aligning the frame with a scene model; and

updating a background statistical model; and

finalizing the background statistical model;

executing a second-pass method for each frame of the frame sequence, the second-pass method comprising the steps of;

labeling each region of the frame; and

performing spatial/temporal filtering of the regions of the frame.

2. Circuitry according to claim 1, wherein the step of updating a background statistical model comprises the steps of:

compiling values corresponding to the regions of the frames; and

computing a standard deviation for each region of the frames.

3. Circuitry according to claim 1, wherein the step of finalizing the background statistical model comprises the step of:

for each region of the frames, determining a statistical mode based on the compiled values; and

assigning the statistical mode of each region as its value in the background statistical model.

4. Circuitry according to claim 1, wherein the step of labeling each region of the frame comprises the steps of:

determining whether or not the region matches the background statistical model; and

assigning a label based on the result of the determining step.

5. Circuitry according to claim 4, wherein the step of determining comprises the step of determining whether a value of the region of the frame falls within some range about a corresponding value of the region in the background statistical model;

and

wherein the step of assigning a label comprises the steps of;

assigning a first label if the value of the region of the frame does fall within the range; and

assigning a second label if the value of the region of the frame does not fall within the range.

6. Circuitry according to claim 5, wherein said first label is background and said second label is foreground.

7. Circuitry according to claim 4, wherein the step of determining comprises the steps of:

determining whether a value of the region of the frame falls within a first range about a corresponding value of the region in the background statistical model; and

if the value of the region of the frame does not fall within the first range, determining if it falls within a second range about said corresponding value of the region in the background statistical model; and

wherein the step of assigning a label comprises the steps of;

assigning a first label if the value of the region of the frame falls within the first range;

assigning a second label if the value of the region of the frame does not fall within the first range and does lie within the second range; and

assigning a third label if the value of the region of the frame does not fall within the second range.

8. Circuitry according to claim 7, wherein said first label is background, said second label is foreground, and said third label is definite foreground.

9. Circuitry according to claim 1, wherein the step of labeling each region of the frame comprises the steps of:

determining whether or not the region matches the background statistical model, comprising the step of performing determinations as to whether or not a value of the region of the frame falls within each of an arbitrary number of ranges about a corresponding value of the region in the background statistical model; and

assigning a label based on the result of the determining step.

10. Circuitry according to claim 1, wherein the step of spatial/temporal filtering comprises the steps of, for each region of the frame:

performing neighborhood filtering; and

performing spatial filtering, temporal filtering, or a combination thereof.

11. Circuitry according to claim 10, wherein the step of performing spatial filtering, temporal filtering, or a combination thereof comprises the step of performing morphological erosions and dilations, wherein the morphological erosions and dilations are performed in two dimensions or in three dimensions.

12. Circuitry according to claim 11, wherein said step of performing morphological erosions and dilations comprises the steps of:

performing a number, n, of erosions, wherein any region labeled definite foreground is not eroded;

following the step of performing n erosions, performing n dilations; and

following the step of performing n dilations, performing a second number, m, of dilations.

13. Circuitry according to claim 10, wherein the step of performing neighborhood filtering comprises the steps of, for each region of the frame, P_i, not labeled background:

determining a location, P_m, in the scene model corresponding to P_i;

determining a neighborhood of regions, P′

_m around P_m;

determining if any of the P′

_m matches P_i;

if any of the P′

_m matches P_i, determining a neighborhood of regions P′

_i around P_i;

determining if any of the P′

_i matches P_m; and

if any of the P′

_m matches P_i, relabeling P_i background.

66. A computer-readable medium comprising software implementing the method of claim 1.

14. Circuitry adapted to perform a one-pass method of video segmentation for differentiating between foreground and background portions of video, the method comprising the steps of:

obtaining a frame sequence from a video stream; and

for each frame in the frame sequence, performing the following steps;

aligning the frame with a scene model;

building a background statistical model;

labeling the regions of the frame; and

performing spatial/temporal filtering.

15. Circuitry according to claim 14, wherein the step of spatial/temporal filtering comprises the steps of, for each region of the frame:

performing neighborhood filtering; and

performing spatial filtering, temporal filtering, or a combination thereof.

16. Circuitry according to claim 15, wherein the step of performing spatial filtering, temporal filtering, or a combination thereof comprises the step of performing morphological erosions and dilations, wherein the morphological erosions and dilations are performed in two dimensions or in three dimensions.

17. Circuitry according to claim 16, wherein said step of performing morphological erosions and dilations comprises the steps of:

performing a number, n, of erosions, wherein any region labeled definite foreground is not eroded;

following the step of performing n erosions, performing n dilations; and

following the step of performing n dilations, performing a second number, m, of dilations.

18. Circuitry according to claim 15, wherein the step of performing neighborhood filtering comprises the steps of, for each region of the frame, P_i, not labeled background:

determining a location, P_m, in the scene model corresponding to P_i;

determining a neighborhood of regions, P′

_m around P_m;

determining if any of the P′

_m matches P_i;

if any of the P′

_m matches P_i, determining a neighborhood of regions P′

_i around P_i;

determining if any of the P′

_i matches P_m; and

if any of the P′

_m matches P_i, relabeling P_i background.

19. Circuitry according to claim 14, wherein said step of building a background statistical model comprises the steps of:

for each region of the frame, performing the steps of;

determining if the background statistical model is mature and stable;

if the background statistical model is mature and stable, determining if a value of the region matches a corresponding value of the background statistical model; and

if the value of the region matches the value of the background statistical model, updating the background statistical model using the value of the region.

20. Circuitry according to claim 19, wherein the step of determining if the background statistical model is mature and stable comprises the steps of:

determining if the background statistical model is mature, and if it is not, updating the background statistical model using the value of the region;

if the background statistical model is determined to be mature, determining if the background statistical model is stable;

if the background statistical model is determined not to be stable, updating the background statistical model using the value of the region.

21. Circuitry according to claim 20, wherein each step of updating the background statistical model using the value of the region comprises the steps of:

updating a running mean value for the region; and

updating a running standard deviation value for the region.

22. Circuitry according to claim 19, wherein the step of updating the background statistical model using the value of the region comprises the steps of:

updating a running mean value for the region; and

updating a running standard deviation value for the region.

23. Circuitry according to claim 19, wherein said step of building a background statistical model further comprises the step of finalizing the background statistical model.

24. The method according to claim 14, wherein said step of labeling the regions of the frame comprises, for each region of the frame, the steps of:

making determinations as to the maturity and stability of the background statistical model for that region;

determining if a value of the background statistical model for that region matches a value of the scene model for that region;

determining if a value of the region matches the value of the scene model for that region; and

determining to what extent the value of the region matches the value of the background statistical model for that region;

and wherein, based on the results of the making and determining steps, a label is assigned to the region.

25. Circuitry according to claim 24, wherein the step of making determinations as to the maturity and stability of the background statistical model comprises the steps of:

determining whether or not the background statistical model is mature;

determining whether or not the background statistical model is stable; and

if the background statistical model is not stable, determining if it was ever previously stable.

26. Circuitry according to claim 25, wherein the step of determining if the background statistical model is stable is executed only if the background statistical model is determined to be mature;

and wherein, if the background statistical model is determined not to be mature, the step of determining if a value of the region matches the value of the scene model for that region is executed.

27. Circuitry according to claim 25, wherein, if the background statistical model is determined to be stable, the step of determining if a value of the background statistical model for that region matches a value of the scene model for that region is executed.

28. Circuitry according to claim 25, wherein, if the step of determining if the background statistical model was ever previously stable returns an affirmative result, the step of determining if a value of the region matches the value of the scene model for that region is executed;

and wherein, if the step of determining if the background statistical model was ever previously stable returns a negative result, then the region is labeled definite foreground.

29. Circuitry according to claim 24, wherein if the step of determining if a value of the region matches the value of the scene model for that region returns an affirmative result, the region is labeled background, and if it returns a negative result, the region is labeled foreground.

30. Circuitry according to claim 24, wherein, if the step of determining if a value of the background statistical model for that region matches a value of the scene model for that region returns an affirmative result, the step of determining to what extent the value of the region matches the value of the background statistical model for that region is executed;

and wherein, if the step of determining if a value of the background statistical model for that region matches a value of the scene model for that region returns a negative result, then the step of determining if a value of the region matches the value of the scene model for that region is executed.

31. Circuitry according to claim 24, wherein the step of determining to what extent the value of the region matches the value of the background statistical model for that region comprises the steps of:

determining if the value of the region matches the value of the background statistical model for that region;

if the step of determining if the value of the region matches the value of the background statistical model for that region, labeling the region background; and

if the step of determining if the value of the region matches the value of the background statistical model for that region returns a negative result, determining if the value of the region falls far from the background statistical model.

32. Circuitry according to claim 31, wherein the step of determining if the value of the region falls far from the background statistical model comprises the step of performing a threshold comparison;

and wherein;

if the step of determining if the value of the region falls far from the background statistical model returns an affirmative result, labeling the region definite foreground, and

if the step of determining if the value of the region falls far from the background statistical model returns a negative result, labeling the region foreground.

33. Circuitry adapted to perform a one-pass method of video segmentation for differentiating between foreground and background portions of video, the method comprising the steps of:

obtaining a frame sequence from a video stream; and

for each frame in the frame sequence, performing the following steps;

aligning the frame with a scene model;

building a background statistical model and a secondary statistical model;

labeling the regions of the frame; and

performing spatial/temporal filtering.

34. Circuitry according to claim 33, wherein the step of spatial/temporal filtering comprises the steps of, for each region of the frame:

performing neighborhood filtering; and

performing spatial filtering, temporal filtering, or a combination thereof.

35. Circuitry according to claim 34, wherein the step of performing spatial filtering, temporal filtering, or a combination thereof comprises the step of performing morphological erosions and dilations, wherein the morphological erosions and dilations are performed in two dimensions or in three dimensions.

36. Circuitry according to claim 35, wherein said step of performing morphological erosions and dilations comprises the steps of:

performing a number, n, of erosions, wherein any region labeled definite foreground is not eroded;

following the step of performing n erosions, performing n dilations; and

following the step of performing n dilations, performing a second number, m, of dilations.

37. Circuitry according to claim 34, wherein the step of performing neighborhood filtering comprises the steps of, for each region of the frame, P_i, not labeled background:

determining a location, P_m, in the scene model corresponding to P_i;

determining a neighborhood of regions, P′

_m around P_m;

determining if any of the P′

_m matches P_i;

if any of the P′

_m matches P_i, determining a neighborhood of regions P′

_i around P_i;

determining if any of the P′

_i matches P_m; and

if any of the P′

_m matches P_i, relabeling P_i background.

38. Circuitry according to claim 33, wherein said step of building a background statistical model comprises the steps of:

for each region of the frame, performing the steps of;

determining if the background statistical model is mature and stable;

if the background statistical model is mature and stable, determining if a value of the region matches a corresponding value of the background statistical model;

if the value of the region matches the value of the background statistical model, updating the background statistical model using the value of the region; and

if the value of the region does not match the value of the background statistical model, updating the secondary statistical model using the value of the region.

39. Circuitry according to claim 38, wherein the step of determining if the background statistical model is mature and stable comprises the steps of:

determining if the background statistical model is mature, and if it is not, updating the background statistical model using the value of the region;

if the background statistical model is determined to be mature, determining if the background statistical model is stable;

if the background statistical model is determined not to be stable, updating the background statistical model using the value of the region.

40. Circuitry according to claim 39, wherein each step of updating the background statistical model using the value of the region comprises the steps of:

updating a running mean value for the region; and

updating a running standard deviation value for the region.

41. Circuitry according to claim 38, wherein the step of updating the background statistical model using the value of the region comprises the steps of:

updating a running mean value for the region; and

updating a running standard deviation value for the region.

42. Circuitry according to claim 38, wherein said step of updating the secondary statistical model comprises the steps of:

determining if the secondary statistical model is mature;

if the secondary statistical model is not mature, updating the secondary statistical model using the value of the region;

if the secondary statistical model is mature, determining if the secondary statistical model is stable;

if the secondary statistical model is not stable, updating the secondary statistical model using the value of the region; and

if the secondary statistical model is stable, replacing the background statistical model for the region with the secondary statistical model for the region.

43. Circuitry according to claim 38, wherein said step of building a background statistical model further comprises the step of finalizing the background statistical model.

44. The method according to claim 33, wherein said step of labeling the regions of the frame comprises, for each region of the frame, the steps of:

making determinations as to the maturity and stability of the background statistical model for that region;

determining if the value of a background statistical model for that region matches a value of the scene model for that region;

determining if a value of the region matches the value of the scene model for that region; and

determining to what extent the value of the region matches the value of the background statistical model for that region;

and wherein, based on the results of the making and determining steps, a label is assigned to the region.

45. Circuitry according to claim 44, wherein the step of making determinations as to the maturity and stability of the background statistical model comprises the steps of:

determining whether or not the background statistical model is mature;

determining whether or not the background statistical model is stable; and

if the background statistical model is not stable, determining if it was ever previously stable.

46. Circuitry according to claim 45, wherein the step of determining if the background statistical model is stable is executed only if the background statistical model is determined to be mature;

and wherein, if the background statistical model is determined not to be mature, the step of determining if a value of the region matches the value of the scene model for that region is executed.

47. Circuitry according to claim 45, wherein, if the background statistical model is determined to be stable, the step of determining if a value of the background statistical model for that region matches a value of the scene model for that region is executed.

48. Circuitry according to claim 45, wherein, if the step of determining if the background statistical model was ever previously stable returns an affirmative result, the step of determining if a value of the region matches the value of the scene model for that region is executed;

and wherein, if the step of determining if the background statistical model was ever previously stable returns a negative result, then the region is labeled definite foreground.

49. Circuitry according to claim 44, wherein if the step of determining if a value of the region matches the value of the scene model for that region returns an affirmative result, the region is labeled background, and if it returns a negative result, the region is labeled foreground.

50. Circuitry according to claim 44, wherein, if the step of determining if a value of the background statistical model for that region matches a value of the scene model for that region returns an affirmative result, the step of determining to what extent the value of the region matches the value of the background statistical model for that region is executed;

and wherein, if the step of determining if a value of the background statistical model for that region matches a value of the scene model for that region returns a negative result, then the step of determining if a value of the region matches the value of the scene model for that region is executed.

51. Circuitry according to claim 44, wherein the step of determining to what extent the value of the region matches the value of the background statistical model for that region comprises the steps of:

determining if the value of the region matches the value of the background statistical model for that region;

if the step of determining if the value of the region matches the value of the background statistical model for that region, labeling the region background; and

if the step of determining if the value of the region matches the value of the background statistical model for that region returns a negative result, determining if the value of the region falls far from the background statistical model.

52. Circuitry according to claim 51, wherein the step of determining if the value of the region falls far from the background statistical model comprises the step of performing a threshold comparison;

and wherein;

if the step of determining if the value of the region falls far from the background statistical model returns an affirmative result, labeling the region definite foreground, and

if the step of determining if the value of the region falls far from the background statistical model returns a negative result, labeling the region foreground.

53. Circuitry adapted to perform a two-pass method of video segmentation for differentiating between foreground and background portions of video, the method comprising the steps of:

obtaining a frame sequence from an input video stream;

executing a first-pass method for each frame of the frame sequence, the first-pass method comprising the steps of;

aligning the frame with a scene model; and

updating a background statistical model, the background statistical model comprising values corresponding to regions of frames of the frame sequence and variances for the regions;

finalizing the background statistical model; and

executing a second-pass method for each frame of the frame sequence, the second-pass method comprising the steps of;

labeling each region of the frame; and

performing spatial/temporal filtering of the regions of the frame.

54. Circuitry adapted to perform a one-pass method of video segmentation for differentiating between foreground and background portions of video, the method comprising the steps of:

obtaining a frame sequence from a video stream; and

for each frame in the frame sequence, performing the following steps;

aligning the frame with a scene model;

building a background statistical model, the background statistical model comprising values corresponding to regions of frames of the frame sequence and variances for the regions;

labeling the regions of the frame; and

performing spatial/temporal filtering.

55. A one-pass method of video segmentation, for differentiating between foreground and background portions of video, comprising the steps of:

obtaining a real-time video stream; and

for each frame in the real-time frame stream, performing the following steps;

labeling pixels in the frame;

performing spatial/temporal filtering;

updating a background statistical model, after the pixels are labeled; and

building and/or updating at least one foreground statistical model, after the pixels are labeled.

56. A method according to claim 55, wherein for each frame in the real-time frame stream, further performing the following step:

inserting an object into the background statistical model.

57. A method according to claim 55, wherein labeling pixels in the frame comprises:

compensating for global lighting change to the frame.

58. A method according to claim 55, wherein updating the background statistical model and the foreground statistical model comprises:

compensating for global lighting change to the frame.

59. A method according to claim 55, wherein pixels are labeled in the frame for an area of interest in the frame.

60. A method according to claim 55, wherein the background statistical model and the foreground statistical model pixels are built and/or updated for an area of interest in the frame.

61. A method according to claim 55, wherein the background statistical model is updated based on previous statistical values of the background statistical model.

62. A method according to claim 61, wherein the background statistical model is updated based on an infinite impulse response filter applied to previous statistical values of the background statistical model.

63. A method according to claim 55, wherein the foreground statistical model is updated based on previous statistical values of the foreground statistical model.

64. A method according to claim 63, wherein the foreground statistical model is updated based on an infinite impulse response filter applied to previous statistical values of the foreground statistical model.

65. A computer system comprising:

a computer; and

a computer-readable medium containing software implementing the method of claim 55.

67. Circuitry adapted to perform a one-pass method of video segmentation for differentiating between foreground and background portions of video, the method comprising the steps of:

obtaining a real-time video stream; and

for each frame in the real-time frame stream, performing the following steps;

labeling pixels in the frame;

performing spatial/temporal filtering;

updating a background statistical model, after the pixels are labeled; and

building and/or updating at least one foreground statistical model, after the pixels are labeled.

68. A method according to claim 67, wherein for each frame in the real-time frame stream, further performing the following step:

inserting an object into the background statistical model.

69. A method according to claim 67, wherein labeling pixels in the frame comprises:

compensating for global lighting change to the frame.

70. A method according to claim 67, wherein updating the background statistical model and the foreground statistical model comprises:

compensating for global lighting change to the frame.

71. A method according to claim 67, wherein pixels are labeled in the frame for an area of interest in the frame.

72. A method according to claim 67, wherein the background statistical model and the foreground statistical model pixels are built and/or updated for an area of interest in the frame.

73. A method according to claim 67, wherein the background statistical model is updated based on previous statistical values of the background statistical model.

74. A method according to claim 73, wherein the background statistical model is updated based on an infinite impulse response filter applied to previous statistical values of the background statistical model.

75. A method according to claim 67, wherein the foreground statistical model is updated based on previous statistical values of the foreground statistical model.

76. A method according to claim 75, wherein the foreground statistical model is updated based on an infinite impulse response filter applied to previous statistical values of the foreground statistical model.