Video surveillance system

• US 7,956,889 B2
• Filed: 06/04/2003
• Issued: 06/07/2011
• Est. Priority Date: 06/04/2003
• Status: Active Grant

• Current Assignee: Model Software Corporation
• Original Assignee: Model Software Corporation
• Inventors: Miller, J. Matthew III, Kelly, John J. III
• Primary Examiner(s): CZEKAJ, DAVID J
• Application Number:
  US10/453,700
• Publication Number:
  US 20040246336A1
• Time in Patent Office: 2,925 Days
• Field of Search: 348/143, 348151-155, 348/169
• US Class Current: 348/143
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Viewing a three dimensional area from numerous camera angles at different exposures using intensity and chromaticity data at the different exposures to create a pixel model for each pixel. A current image is compared with the background model to find pixels that have changed from their pixel model. These novel pixels are processed using contiguous region detection and grouped into foreground figures. For each camera, software extracts features from each foreground figure. A central processor maintains an object model for each foreground figure. A graphical user interface displays the relative locations of foreground figures in a world model. The location and identification of the foreground figures is checked against a table of permissions so as to selectively generate an alert. If a figure leaves or is about to leave a cell, the invention accounts for its approximate position and notifies adjacent cells of the expected arrival of the foreground figure.

Claims

1. A method for video surveillance, the method comprising the following steps:

storing video data, in a physical computer readable medium, for at least two images at more than one exposure time;

analyzing, in a computer, the video data for the at least two images at more than one exposure time to create a pixel model for each pixel in the at least two images;

storing video data, in a physical computer readable medium, for a current image at a current exposure time; and

comparing, in a computer, each pixel of said current image with a corresponding pixel model so as to detect novel pixels.

2. A method according to claim 1, the method further comprising the step of grouping the novel pixels into foreground figures.

3. A method according to claim 2, wherein the step of grouping the novel pixels into foreground figures comprises the following steps:

labeling a pixel as being in an existing region, no region, or a new region; and

joining two regions if the addition of the pixel makes the two regions adjacent;

and calculating the average grid location of the pixels in each region.

4. A method according to claim 2, the method further comprising the step of extracting features from said foreground figure.

5. A method according to claim 4, the method further comprising the step of comparing said features with object models in a world model.

6. A method according to claim 4, wherein the step of grouping the novel pixels into foreground figures comprises the following steps:

generating a list of colors; and

sorting the colors by frequency of appearance; and

grouping the pixels based upon the list of colors.

7. A method according to claim 6, wherein the step of grouping the novel pixels into foreground figures comprises the following steps:

labeling a pixel as being in an existing region, no region, or a new region; and

joining two regions if the addition of the pixel makes the two regions adjacent and the regions are the same color.

8. A method according to claim 7, wherein the step of grouping the novel pixels into foreground figures further comprises the step of connecting two regions if the addition of the pixel makes the two regions adjacent and the regions are not the same color.

9. A method according to claim 8, wherein the step of grouping the novel pixels into foreground figures further comprises the step of calculating the average grid location of the pixels in each region.

10. A method according to claim 5, the method further comprising the step of determining whether to update one of the object models or add an object model to the world model.

11. A method according to claim 1, wherein the video data comprises intensity data and chromaticity data.

12. A method according to claim 11, wherein the intensity data comprises red, green, and blue intensities for each pixel.

13. A method according to claim 12, wherein the chromaticity data comprises the ratio of the red intensity to the green intensity for each pixel.

14. A method according to claim 13, wherein the chromaticity data further comprises the ratio of the blue intensity to the green intensity for each pixel.

15. A method according to claim 1, wherein the step of storing video data for at least two images comprises the following steps:

storing video data for a plurality of images at a minimum exposure time;

storing video data for a plurality of images at a plurality of intermediate exposure times; and

storing video data for a plurality of images at a maximum exposure time.

16. A method according to claim 15, wherein the plurality of intermediate exposure times are equally spaced between the minimum exposure time and the maximum exposure time.

17. A method according to claim 15, wherein the number of images stored at the minimum exposure time is greater than the number of exposures stored at any of the intermediate exposure times.

18. A method according to claim 15, further comprising the step of selecting a best exposure time.

19. A method according to claim 18, the current exposure time equal to the best exposure time.

20. A method according to claim 1, further comprising the step of normalizing the video data for the current image for exposure time.

21. A method according to claim 1, further comprising the step of storing video data for a current image at a current exposure time and gain setting.

22. A method according to claim 21, further comprising the step of normalizing the video data for the current image for exposure time and gain setting.

23. A method according to claim 1, wherein the step of analyzing the video data comprises the step of counting the number of times that each pixel in the at least two images is underexposed.

24. A method according to claim 23, wherein the step of analyzing the video data further comprises the step of counting the number of times that each pixel in the at least two images is overexposed.

25. A method according to claim 24, wherein the step of analyzing the video data further comprises the step of counting the number of times that each pixel in the at least two images is correctly exposed.

26. A method according to claim 25, wherein the step of analyzing the video data further comprises the step of saving video data in the pixel model if the pixel is correctly exposed.

27. A method according to claim 26, wherein the step of analyzing the video data further comprises the step of calculating an average for the video data in the pixel model.

28. A method according to claim 27, wherein the step of analyzing the video data further comprises the step of calculating a standard deviation for the video data in the pixel model.

29. A method according to claim 1, the video data comprising red, green, and blue intensities.

30. A method according to claim 29, wherein the step of analyzing the video data comprises the step of calculating an average red intensity, an average green intensity, and an average blue intensity.

31. A method according to claim 29, wherein the step of analyzing the video data comprises the step of saving red, green, and blue intensities in the pixel models.

32. A method according to claim 31, wherein the step of analyzing the video data further comprises the step of calculating an average red intensity, an average green intensity, and an average blue intensity for the red, green, and blue intensities in each pixel model.

33. A method according to claim 32, wherein the step of comparing comprises the steps of:

calculating the sum of the red, green, and blue intensities in each pixel model;

calculating the sum of the red, green, and blue intensities for each pixel in the current image; and

comparing the sum for each pixel in the current image with the sum for the corresponding pixel model.

34. A method according to claim 33, wherein the step of analyzing the video data further comprises the step of saving chromaticity data in the pixel model.

35. A method according to claim 34, wherein the step of comparing further comprises the step of comparing the chromaticity of a pixel in the current image with the chromaticity data saved in a corresponding pixel model.

36. A method according to claim 35, wherein the step of comparing further comprises the step of treating said pixel as in said pixel model if the chromaticity of said pixel is consistent with said chromaticity data.

37. A method according to claim 35, wherein the step of comparing further comprises the step of comparing the red intensity of said pixel in the current image with red intensity data saved in said corresponding pixel model.

38. A method according to claim 37, wherein the step of comparing further comprises the step of treating said pixel as novel if the red intensity is inconsistent with the red intensity data saved in said corresponding pixel model.

39. A method according to claim 38, wherein the step of comparing further comprises the step of comparing the green intensity of said pixel in the current image with green intensity data saved in said corresponding pixel model.

40. A method according to claim 39, wherein the step of comparing further comprises the step of treating said pixel as novel if the green intensity is inconsistent with the green intensity data saved in said corresponding pixel model.

41. A method according to claim 40, wherein the step of comparing further comprises the step of comparing the blue intensity of said pixel in the current image with blue intensity data saved in said corresponding pixel model.

42. A method according to claim 41, wherein the step of comparing further comprises the step of treating said pixel as novel if the blue intensity is inconsistent with the blue intensity data saved in said corresponding pixel model.

43. A method according to claim 42, wherein the step of comparing further comprises the step of treating said pixel as in said pixel model if the blue intensity is consistent with the blue intensity data saved in said corresponding pixel model.

44. A method according to claim 30, the video data further comprising the ratio of the red intensity to the green intensity.

45. A method according to claim 44, wherein the step of analyzing the video data further comprises the step of calculating an average ratio of the red intensity to the green intensity.

46. A method according to claim 45, the video data further comprising the ratio of the blue intensity to the green intensity.

47. A method according to claim 46, wherein the step of analyzing the video data further comprises the step of calculating an average ratio of the blue intensity to the green intensity.

48. A method according to claim 1, wherein the step of analyzing the video data comprises the step of saving video data in the pixel model.

49. A method according to claim 48, wherein the step of analyzing the video data further comprises the step of calculating an average for the video data in the pixel model.

50. A method according to claim 49, wherein the step of analyzing the video data further comprises the step of adjusting the average if the average is near an underexposed threshold.

51. A method according to claim 50, wherein the step of analyzing the video data further comprises the step of adjusting the average if the average is near an overexposed threshold.

52. A method according to claim 49, wherein the step of analyzing the video data further comprises the step of calculating a standard deviation for the video data in the pixel model.

53. A method according to claim 1, wherein the step of analyzing the video data comprises the step of saving first color ratios and second color ratios in the pixel models.

54. A method according to claim 53, wherein the step of comparing comprises the following steps:

determining whether the numerator of a first color ratio of a pixel in the current image is incorrectly exposed;

determining whether the numerator of a second color ratio of a pixel in the current image is incorrectly exposed; and

treating the pixel as in the model if the numerator of the first color ratio is incorrectly exposed in the way that it was expected to be incorrectly exposed and the numerator of the second color ratio is incorrectly exposed in the way that it was expected to be incorrectly exposed.

55. A method according to claim 53, wherein the step of comparing comprises the following steps:

determining whether the denominator of a first color ratio and a second color ratio of a pixel in the current image is incorrectly exposed; and

treating the pixel as in the model if the denominator is incorrectly exposed in the way that it was expected to be incorrectly exposed.

56. A method according to claim 1, wherein the step of analyzing the video data comprises the step of saving a color intensity in the pixel models.

57. A method according to claim 56, wherein the step of analyzing the video data further comprises the step of calculating the average color intensity for the color intensities in each pixel model.

58. A method according to claim 57, wherein the step of comparing comprises the step of comparing the color intensity for a pixel in the current image with the average color intensity for the corresponding pixel model.

59. A method according to claim 56, wherein the step of comparing comprises the following steps:

determining whether a pixel in the current image was ever correctly exposed;

determining whether the number of values that were underexposed is greater than the number of values that were overexposed; and

determining whether the color intensity of the pixel in the current image is less than or equal to the color intensity of the underexposed threshold.

60. A method according to claim 59, wherein the step of comparing further comprises the following steps:

determining whether the number of values that were overexposed is greater than the number of values that were underexposed; and

determining whether the color intensity of the pixel in the current image is greater than or equal to the color intensity of the overexposed threshold.

61. A method according to claim 56, wherein the step of comparing comprises the following steps:

calculating the predicted color intensity for a pixel in the current image; and

determining whether the color intensity of the pixel in the current image is less than or equal to the color intensity for the underexposed threshold; and

determining whether the predicted color intensity for the pixel in the current image is less than or equal to the color intensity for the underexposed threshold.

62. A method according to claim 61, wherein the step of comparing further comprises:

determining whether the color intensity of the pixel in the current image is greater than or equal to the color intensity for the overexposed threshold; and

determining whether the predicted color intensity for the pixel in the current image is greater than or equal to the color intensity for the overexposed threshold.

63. A method according to claim 53, wherein the step of analyzing the video data further comprises the step of calculating an average first color ratio and an average second color ratio for the first color ratios and second color ratios in each pixel model.

64. A method according to claim 63, wherein the step of comparing comprises:

calculating a first difference between the first color ratio of a pixel in the current image and the average first color ratio in a corresponding pixel model in standard deviations;

calculating a second difference between the second color ratio of said pixel and the average second color ratio in said corresponding pixel model in standard deviations; and

calculating the square root of the sum of first difference squared and the second difference squared.

65. A method according to claim 63, wherein the step of comparing comprises the step of determining whether the first color ratio is normal, undefined, approaches zero, or approaches infinity.

66. A method according to claim 65, wherein the step of comparing further comprises the step of determining whether the second color ratio is normal, undefined, approaches zero, or approaches infinity.

67. A method according to claim 66, wherein the step of comparing further comprises the step of selecting a Boolean statement based upon whether the first color ratio is normal, approaches zero, or approaches infinity and whether the second color ratio is normal, approaches zero, or approaches infinity.

68. A method according to claim 67, where the Boolean statement involves comparing the first color ratio with the average first color ratio and comparing the second color ratio with the average second color ratio.

69. A method according to claim 68, wherein the step of comparing further comprises the step of comparing the second color ratio with the average second color ratio if the first color ratio is undefined.

70. A method according to claim 69, wherein the step of comparing further comprises the step of comparing the first color ratio with the average first color ratio if the second color ratio is undefined.

71. A method for video surveillance, the method comprising the following steps:

storing video data, in a physical computer readable medium, for at least two images at more than one exposure time;

analyzing, in a computer, the video data for the at least two images at more than one exposure time to create a pixel model for each pixel in the at least two images;

storing video data, in a physical computer readable medium, for a current image at a current exposure time; and

comparing, in a computer, each pixel of said current image with a corresponding pixel model so as to detect novel pixels, wherein the step of storing video data for at least two images comprises the following steps;

storing video data for a plurality of images at a minimum exposure time;

storing video data for a plurality of images at a plurality of intermediate exposure times; and

storing video data for a plurality of images at a maximum exposure time,

wherein the number of images stored at the minimum exposure time is greater than the number of exposures stored at any of the intermediate exposure times

wherein the number of images stored at the maximum exposure time is greater than the number of exposures stored at any of the intermediate exposure times.

72. A method for video surveillance, the method comprising the following steps:

storing video data, in a physical computer readable medium, for at least two images at more than one exposure time;

analyzing, in a computer, the video data for the at least two images at more than one exposure time to create a pixel model for each pixel in the at least two images;

storing video data, in a physical computer readable medium, for a current image at a current exposure time; and

comparing, in a computer, each pixel of said current image with a corresponding pixel model so as to detect novel pixels, wherein the step of storing video data for at least two images comprises the following steps;

storing video data for a plurality of images at a minimum exposure time;

storing video data for a plurality of images at a plurality of intermediate exposure times; and

storing video data for a plurality of images at a maximum exposure time,

wherein the number of images stored at the minimum exposure time is greater than the number of exposures stored at any of the intermediate exposure times

wherein the step of analyzing the video data comprises the step of saving video data in the pixel model,

wherein the step of analyzing the video data further comprises the step of calculating an average for the video data in the pixel mode,

wherein the step of analyzing the video data further comprises the step of calculating a standard deviation for the video data in the pixel model, and

wherein the step of analyzing the video data further comprises the step of adjusting the average if the sum of the average and the product of a constant and the standard deviation is less than the sum of an underexposed threshold and the product of the constant and a predetermined standard deviation.

73. A method according to claim 72, wherein the step of analyzing the video data further comprises the step of adjusting the average if the difference of the average and the product of the constant and the standard deviation is greater than the difference of an overexposed threshold and the product of the constant and the predetermined standard deviation.