Video surveillance system

• US 20040246336A1
• Filed: 06/04/2003
• Published: 12/09/2004
• Est. Priority Date: 06/04/2003
• Status: Active Grant

• Current Assignee: Model Software Corporation
• Original Assignee: Model Software Corporation
• Inventors: Miller, J. Matthew III, Kelly, John J. III
• Granted Patent: US 7,956,889 B2
• Time in Patent Office: Days
• US Class Current: 348/143
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Viewing a three dimensional area from numerous camera angles at different exposures using intensity and chromaticity data at the different exposures to create a pixel model for each pixel. A current image is compared with the background model to find pixels that have changed from their pixel model. These novel pixels are processed using contiguous region detection and grouped into foreground figures. For each camera, software extracts features from each foreground figure. A central processor maintains an object model for each foreground figure. A graphical user interface displays the relative locations of foreground figures in a world model. The location and identification of the foreground figures is checked against a table of permissions so as to selectively generate an alert. If a figure leaves or is about to leave a cell, the invention accounts for its approximate position and notifies adjacent cells of the expected arrival of the foreground figure.

Claims

1. A method for video surveillance, the method comprising the following steps:

storing video data for at least two images at more than one exposure time;

analyzing the video data for the at least two images at more than one exposure time to create a pixel model for each pixel in the at least two images;

storing video data for a current image at a current exposure time; and

comparing each pixel of said current image with a corresponding pixel model so as to detect novel pixels.

2. A method according to claim 1, the method further comprising the step of grouping the novel pixels into foreground figures.

3. A method according to claim 2, the method further comprising the step of extracting features from said foreground figure.

4. A method according to claim 3, the method further comprising the step of comparing said features with object models in a world model.

5. A method according to claim 4, the method further comprising the step of determining whether to update one of the object models or add an object model to the world model.

6. A method according to claim 1, wherein the video data comprises intensity data and chromaticity data.

7. A method according to claim 6, wherein the intensity data comprises red, green, and blue intensities for each pixel.

8. A method according to claim 7, wherein the chromaticity data comprises the ratio of the red intensity to the green intensity for each pixel.

9. A method according to claim 8, wherein the chromaticity data further comprises the ratio of the blue intensity to the green intensity for each pixel.

10. A method according to claim 1, wherein the step of storing video data for at least two images comprises the following steps:

storing video data for a plurality of images at a minimum exposure time;

storing video data for a plurality of images at a plurality of intermediate exposure times; and

storing video data for a plurality of images at a maximum exposure time.

11. A method according to claim 10, wherein the plurality of intermediate exposure times are equally spaced between the minimum exposure time and the maximum exposure time.

12. A method according to claim 10, wherein the number of images stored at the minimum exposure time is greater than the number of exposures stored at any of the intermediate exposure times.

13. A method according to claim 12, wherein the number of images stored at the maximum exposure time is greater than the number of exposures stored at any of the intermediate exposure times.

14. A method according to claim 10, further comprising the step of selecting a best exposure time.

15. A method according to claim 14, the current exposure time equal to the best exposure time.

16. A method according to claim 1, further comprising the step of normalizing the video data for the current image for exposure time.

17. A method according to claim 1, further comprising the step of storing video data for a current image at a current exposure time and gain setting.

18. A method according to claim 17, further comprising the step of normalizing the video data for the current image for exposure time and gain setting.

19. A method according to claim 1, wherein the step of analyzing the video data comprises the step of counting the number of times that each pixel in the at least two images is underexposed.

20. A method according to claim 19, wherein the step of analyzing the video data further comprises the step of counting the number of times that each pixel in the at least two images is overexposed.

21. A method according to claim 20, wherein the step of analyzing the video data further comprises the step of counting the number of times that each pixel in the at least two images is correctly exposed.

22. A method according to claim 21, wherein the step of analyzing the video data further comprises the step of saving video data in the pixel model if the pixel is correctly exposed.

23. A method according to claim 22, wherein the step of analyzing the video data further comprises the step of calculating an average for the video data in the pixel model.

24. A method according to claim 23, wherein the step of analyzing the video data further comprises the step of calculating a standard deviation for the video data in the pixel model.

25. A method according to claim 1, the video data comprising red, green, and blue intensities.

26. A method according to claim 25, wherein the step of analyzing the video data comprises the step of calculating an average red intensity, an average green intensity, and an average blue intensity.

27. A method according to claim 26, the video data further comprising the ratio of the red intensity to the green intensity.

28. A method according to claim 27, wherein the step of analyzing the video data further comprises the step of calculating an average ratio of the red intensity to the green intensity.

29. A method according to claim 28, the video data further comprising the ratio of the blue intensity to the green intensity.

30. A method according to claim 29, wherein the step of analyzing the video data further comprises the step of calculating an average ratio of the blue intensity to the green intensity.

31. A method according to claim 1, wherein the step of analyzing the video data comprises the step of saving video data in the pixel model.

32. A method according to claim 31, wherein the step of analyzing the video data further comprises the step of calculating an average for the video data in the pixel model.

33. A method according to claim 32, wherein the step of analyzing the video data further comprises the step of adjusting the average if the average is near an underexposed threshold.

34. A method according to claim 33, wherein the step of analyzing the video data further comprises the step of adjusting the average if the average is near an overexposed threshold.

35. A method according to claim 32, wherein the step of analyzing the video data further comprises the step of calculating a standard deviation for the video data in the pixel model.

36. A method according to claim 35, wherein the step of analyzing the video data further comprises the step of adjusting the average if the sum of the average and the product of a constant and the standard deviation is less than the sum of an underexposed threshold and the product of the constant and a predetermined standard deviation.

37. A method according to claim 36, wherein the step of analyzing the video data further comprises the step of adjusting the average if the difference of the average and the product of the constant and the standard deviation is greater than the difference of an overexposed threshold and the product of the constant and the predetermined standard deviation.

38. A method according to claim 25, wherein the step of analyzing the video data comprises the step of saving red, green, and blue intensities in the pixel models.

39. A method according to claim 38, wherein the step of analyzing the video data further comprises the step of calculating an average red intensity, an average green intensity, and an average blue intensity for the red, green, and blue intensities in each pixel model.

40. A method according to claim 39, wherein the step of comparing comprises the steps of:

calculating the sum of the red, green, and blue intensities in each pixel model;

calculating the sum of the red, green, and blue intensities for each pixel in the current image; and

comparing the sum for each pixel in the current image with the sum for the corresponding pixel model.

41. A method according to claim 40, wherein the step of analyzing the video data further comprises the step of saving chromaticity data in the pixel model.

42. A method according to claim 41, wherein the step of comparing further comprises the step of comparing the chromaticity of a pixel in the current image with the chromaticity data saved in a corresponding pixel model.

43. A method according to claim 42, wherein the step of comparing further comprises the step of treating said pixel as in said pixel model if the chromaticity of said pixel is consistent with said chromaticity data.

44. A method according to claim 42, wherein the step of comparing further comprises the step of comparing the red intensity of said pixel in the current image with red intensity data saved in said corresponding pixel model.

45. A method according to claim 44, wherein the step of comparing further comprises the step of treating said pixel as novel if the red intensity is inconsistent with the red intensity data saved in said corresponding pixel model.

46. A method according to claim 45, wherein the step of comparing further comprises the step of comparing the green intensity of said pixel in the current image with green intensity data saved in said corresponding pixel model.

47. A method according to claim 46, wherein the step of comparing further comprises the step of treating said pixel as novel if the green intensity is inconsistent with the green intensity data saved in said corresponding pixel model.

48. A method according to claim 47, wherein the step of comparing further comprises the step of comparing the blue intensity of said pixel in the current image with blue intensity data saved in said corresponding pixel model.

49. A method according to claim 48, wherein the step of comparing further comprises the step of treating said pixel as novel if the blue intensity is inconsistent with the blue intensity data saved in said corresponding pixel model.

50. A method according to claim 49, wherein the step of comparing further comprises the step of treating said pixel as in said pixel model if the blue intensity is consistent with the blue intensity data saved in said corresponding pixel model.

51. A method according to claim 1, wherein the step of analyzing the video data comprises the step of saving first color ratios and second color ratios in the pixel models.

52. A method according to claim 51, wherein the step of analyzing the video data further comprises the step of calculating an average first color ratio and an average second color ratio for the first color ratios and second color ratios in each pixel model.

53. A method according to claim 52, wherein the step of comparing comprises the following steps:

calculating a first difference between the first color ratio of a pixel in the current image and the average first color ratio in a corresponding pixel model in standard deviations;

calculating a second difference between the second color ratio of said pixel and the average second color ratio in said corresponding pixel model in standard deviations; and

calculating the square root of the sum of first difference squared and the second difference squared.

54. A method according to claim 52, wherein the step of comparing comprises the step of determining whether the first color ratio is normal, undefined, approaches zero, or approaches infinity.

55. A method according to claim 54, wherein the step of comparing further comprises the step of determining whether the second color ratio is normal, undefined, approaches zero, or approaches infinity.

56. A method according to claim 55, wherein the step of comparing further comprises the step of selecting a Boolean statement based upon whether the first color ratio is normal, approaches zero, or approaches infinity and whether the second color ratio is normal, approaches zero, or approaches infinity.

57. A method according to claim 56, where the Boolean statement involves comparing the first color ratio with the average first color ratio and comparing the second color ratio with the average second color ratio.

58. A method according to claim 57, wherein the step of comparing further comprises the step of comparing the second color ratio with the average second color ratio if the first color ratio is undefined.

59. A method according to claim 58, wherein the step of comparing further comprises the step of comparing the first color ratio with the average first color ratio if the second color ratio is undefined.

60. A method according to claim 51, wherein the step of comparing comprises the following steps:

determining whether the numerator of a first color ratio of a pixel in the current image is incorrectly exposed;

determining whether the numerator of a second color ratio of a pixel in the current image is incorrectly exposed; and

treating the pixel as in the model if the numerator of the first color ratio is incorrectly exposed in the way that it was expected to be incorrectly exposed and the numerator of the second color ratio is incorrectly exposed in the way that it was expected to be incorrectly exposed.

61. A method according to claim 51, wherein the step of comparing comprises the following steps:

determining whether the denominator of a first color ratio and a second color ratio of a pixel in the current image is incorrectly exposed; and

treating the pixel as in the model if the denominator is incorrectly exposed in the way that it was expected to be incorrectly exposed.

62. A method according to claim 1, wherein the step of analyzing the video data comprises the step of saving a color intensity in the pixel models.

63. A method according to claim 62, wherein the step of analyzing the video data further comprises the step of calculating the average color intensity for the color intensities in each pixel model.

64. A method according to claim 63, wherein the step of comparing comprises the step of comparing the color intensity for a pixel in the current image with the average color intensity for the corresponding pixel model.

65. A method according to claim 62, wherein the step of comparing comprises the following steps:

determining whether a pixel in the current image was ever correctly exposed;

determining whether the number of values that were underexposed is greater than the number of values that were overexposed; and

determining whether the color intensity of the pixel in the current image is less than or equal to the color intensity of the underexposed threshold.

66. A method according to claim 65, wherein the step of comparing further comprises the following steps:

determining whether the number of values that were overexposed is greater than the number of values that were underexposed; and

determining whether the color intensity of the pixel in the current image is greater than or equal to the color intensity of the overexposed threshold.

67. A method according to claim 62, wherein the step of comparing comprises the following steps:

calculating the predicted color intensity for a pixel in the current image; and

determining whether the color intensity of the pixel in the current image is less than or equal to the color intensity for the underexposed threshold; and

determining whether the predicted color intensity for the pixel in the current image is less than or equal to the color intensity for the underexposed threshold.

68. A method according to claim 67, wherein the step of comparing further comprises the following steps:

determining whether the color intensity of the pixel in the current image is greater than or equal to the color intensity for the overexposed threshold; and

determining whether the predicted color intensity for the pixel in the current image is greater than or equal to the color intensity for the overexposed threshold.

69. A method according to claim 2, wherein the step of grouping the novel pixels into foreground figures comprises the following steps:

labeling a pixel as being in an existing region, no region, or a new region;

joining two regions if the addition of the pixel makes the two regions adjacent; and

calculating the average grid location of the pixels in each region.

70. A method according to claim 3, wherein the step of grouping the novel pixels into foreground figures comprises the following steps:

generating a list of colors;

sorting the colors by frequency of appearance; and

grouping the pixels based upon the list of colors.

71. A method according to claim 70, wherein the step of grouping the novel pixels into foreground figures comprises the following steps:

labeling a pixel as being in an existing region, no region, or a new region;

joining two regions if the addition of the pixel makes the two regions adjacent and the regions are the same color.

72. A method according to claim 71, wherein the step of grouping the novel pixels into foreground figures further comprises the step of connecting two regions if the addition of the pixel makes the two regions adjacent and the regions are not the same color.

73. A method according to claim 72, wherein the step of grouping the novel pixels into foreground figures further comprises the step of calculating the average grid location of the pixels in each region.

74. A method for video surveillance, the method comprising the following steps:

analyzing video data from a first perspective to create a first background model;

analyzing video data from a second perspective to create a second background model comparing video data from the first perspective with the first background model so as to detect a first set of novel pixels;

comparing video data from the second perspective with the second background model so as to detect a second set of novel pixels;

placing features from the first set of novel pixels into a queue;

placing features from the second set of novel pixels into said queue;

comparing the features from the first set of novel pixels with object models in a world model; and

comparing the features from the second set of novel pixel with object models in the world model.

75. A method according to claim 74, wherein the world model is updated before comparing the features from the second set of novel pixel with object models in the world model.

76. A method according to claim 74, wherein the queue is a time priority queue.

77. A method according to claim 76, wherein a feature from the first set of novel pixels is not processed until a feature from the second set of novel pixels is available.

78. A method according to claim 77, the method further comprising the step of determining whether to update one of the object models or add an object model to the world model.

79. A method according to claim 74, wherein a feature from the first set of novel pixels is processed before a feature from the second set of novel pixels is processed and the world model is reverted before the second set of novel pixels is processed.

80. A method according to claim 79, the method further comprising the step of determining whether to update one of the object models or add an object model to the world model.

81. A video surveillance system comprising:

a first camera station comprising one or more cameras;

a second camera station comprising one or more cameras;

a first region, the first camera station and the second camera station each providing at least one camera to survey said first region;

a third camera station comprising one or more cameras;

a fourth camera station comprising one or more cameras;

a second region, the third camera station and the fourth camera station each providing at least one camera to survey said second region;

a control unit, the control unit tracking a figure as it exits the first region and enters the second region, the control unit updating an object model in a world model to reflect the change in regions.

82. A system according to claim 81, wherein the first camera station calculates a first position estimate based upon a first vector in real space and the second camera station calculates a second position estimate based upon a second vector in real space.

83. A system according to claim 82, the control unit comparing the first position estimate with the second position estimate so as to update the position of the figure.

84. A system according to claim 83, the control unit graphically displaying an icon on a video interface based upon the position of the figure.

85. A system according to claim 84, the control unit checking the position of the figure against a table of permissions.

86. A system according to claim 85, the control unit generating an alarm based upon the position of the figure.

87. A system according to claim 81, the control unit assigning the figure a global id and a local id, the local id corresponding to the first region.

88. A system in accordance with claim 87, the first region a closed are entry point, the control unit assigning the figure an old global id and an old local id.

89. A system in accordance with claim 87, the control unit assigning the figure a second global id and a second local id, the second local id corresponding to the second region.

90. A system in accordance with claim 89, the control unit connecting the first global id with the second global id once the figure in the second region is determined to be the same figure as the figure in the first region.