Method and apparatus for detecting object movement within an image sequence

• US 5,847,755 A
• Filed: 12/11/1996
• Issued: 12/08/1998
• Est. Priority Date: 01/17/1995
• Status: Expired due to Term

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Wixson, Lambert Ernest, Hsu, Stephen Charles
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): Philippe, Gims S.
• Application Number:
  US08/755,376
• Time in Patent Office: 727 Days
• Field of Search: 348/149, 348/155, 348/169, 348/148, 348/161, 348/143, 348/154, 348/699, 382/103, 382/104
• US Class Current: 348/149
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors

Abstract

Method and apparatus for processing a sequence of images to detect object movement within the sequence. Specifically, the method comprises the steps of: (a) supplying a sequence of image frames; (b) initializing a reference image that contains image information regarding stationary objects within a scene represented by the sequence of images; (c) supplying a next image frame which temporally follows the sequence of image frames; (d) comparing the next image to the reference image to produce a motion image representing motion information regarding movement of objects within the scene; (e) updating the reference image with information within the next image that represents stationary objects within the scene; and (f) repeating steps (c), (d), and (e) for each next image supplied. The method is implemented by image processing apparatus. A specific embodiment of the method and apparatus is a traffic monitoring system that identifies vehicles in varying illumination levels and eliminates erroneous identification of non-physical movement in the scene such as shadows and headlight reflections.

Claims

1. In a vehicular traffic monitoring system including a video camera having a given field of view for recording successive image frames of road traffic within its field of view, and digital image processing means responsive to pixel information defined by each of said successive image frames;

the improvement wherein said digital image processing means comprises;

first means responsive to an initial train of said successive image frames for deriving a stored initial reference image defining only stationary objects within said field of view and thereafter updating said stored initial reference image with a reference image derived from an image frame recorded later than said initial train, with each pixel'"'"'s digital amplitude level of each of said reference images being determined by illumination conditions existing when said initial train and when said later recorded frame were recorded;

second means for modifying each pixel'"'"'s digital amplitude level of one of a current image frame and said stored reference image then being stored to make their corresponding pixels defining stationary objects substantially equal to one another;

third means responsive to the digital amplitude-level difference between corresponding pixels of each of successively-occurring ones of said successive image frames and the then stored reference image for deriving successive images defining only moving objects within said field of view;

fourth means for discriminating between those moving objects that remain substantially fixed in position with respect to one another in each of said successively-occurring ones of said successively-occurring images and those moving objects that substantially change in position with respect to one another in each of said successively-occurring ones of said successively-occurring images; and

a fifth means responsive to the variance of the digital amplitude levels of the pixels of those ones of said objects that remain substantially fixed in position with respect to one another for distinguishing those ones of said moving objects that remain substantially fixed in position with respect to one another that define non-physical moving objects, from said moving objects that remain substantially fixed in position with respect to one another that define physical moving objects, and then eliminating those ones of said moving objects that define non-physical moving objects.

2. The vehicular traffic monitoring system defined in claim 1, wherein said video camera is subject to camera sway, and wherein said digital image processing means further comprises:

electronic image stabilization means for compensating for image translation due to camera sway.

3. The vehicular traffic monitoring system defined in claim 2, wherein:

said electronic image stabilization means employs two known landmarks in each successive image frame to shift the current image that first number of rows and that second number of columns which result in said current image remaining substantially in alignment with said reference image.

4. The vehicular traffic monitoring system defined in claim 1, wherein said video camera records successive two-dimensional image frames of road traffic within its field of view, and wherein said digital image processing means further comprises:

sixth means employing pixel integration windows for converting each frame derived by said third means of said successive images that define only moving objects within said field of view from a two-dimensional image frame to a one-dimensional image frame that is substantially oriented in the direction of said road traffic.

5. The vehicular traffic monitoring system defined in claim 1, wherein:

said first means and said second means cooperate during the occurrence of said initial train of said successive image frames in accordance with a first set of equations ##EQU3## and thereafter said first and second means cooperate in accordance with a second set of equations ##EQU4## where (x,y) represents the coordinates of each pixel in an image frame, r_t represents the reference image after frame t, and i_t represents the t'"'"'th frame of the successive image frames, D represents a predetermined difference between the amplitude level of each pair of corresponding pixels of i_t and r_t, k and c_t respectively represent a single pair of estimated gain and offset values in amplitude level for the pixels of one of i_t and r_t which results in corresponding pixels of i_t and r_t which reduces the statistical error in the respective differences in amplitude levels between corresponding pixels of i_t and r_t, and γ

is a constant that determines the responsiveness of the cooperation of said first and second means both during the occurrence of said initial train of said successive image frames and thereafter.

6. The vehicular traffic monitoring system defined in claim 5, wherein:

k and c_t respectively represent that single pair of estimated gain and offset in amplitude level of the pixels of one of i_t and r_t which minimizes the statistical error in the respective differences in amplitude levels between corresponding pixels of i_t and r_t.

7. A method of image processing comprising the steps of:

(a) supplying a first sequence of image frames;

(b) initializing a reference image that contains image information regarding stationary objects within a scene represented by said first sequence of images;

(c) supplying a next image frame which temporally follows said first sequence of image frames;

(d) comparing said next image frame to said reference image to produce a two-dimensional motion image representing motion information regarding movement of objects within the scene;

(e) updating said reference image with information within said next image frame, where said information used for updating the reference image only represents stationary objects within the scene and substantially disregards moving objects and temporarily stationary objects within the scene; and

(f) repeating steps (c), (d), and (e) for each next image supplied.

8. The method of claim 7 further comprising the step of stabilizing each image frame to remove any motion within said image frames that occurs due to imaging sensor movement rather than object movement within the scene.

9. The method of claim 7 wherein said updating step comprises the steps of:

selecting a pixel value in the reference image and a pixel value in the next image frame having the same pixel location as the selected pixel value in the reference image;

computing a difference factor equal to the absolute value of the difference between said selected pixel value in the next image frame and said pixel value in the reference image;

comparing the difference value to a first threshold value;

if said difference value is less than said first threshold value, replacing the pixel value in the reference image with an updated reference pixel value equaling the selected reference image pixel value altered by an update factor; and

if said difference value is greater than or equal to said first threshold value, the pixel value in said reference image remains the same.

10. The method of claim 9 wherein said update factor is the difference between said selected pixel value in the next image and said pixel value in the reference image multiplied by a responsiveness factor, where said responsiveness factor limits the amount of the update to ensure that the updated reference image only contains information regarding information within said next image frame that represents stationary objects within the scene.

11. The method of claim 9 wherein said first threshold value is a function of the difference factor.

12. The method of claim 7 wherein said method further comprises the step of modifying said reference image using image information within said next image frame regarding changes to illumination of the scene.

13. The method of claim 12 wherein said modifying step comprises the steps of:

selecting a pixel value in the reference image and a pixel value in the next image frame having the same pixel location as the selected reference image pixel value;

computing a difference factor equal to the absolute value of the difference between said selected pixel value in the next image and said pixel value in the reference image;

comparing the difference value to a second threshold value;

if said difference value is less than said second threshold value, replacing the pixel value in the reference image with a modified reference pixel value equaling the selected reference image pixel value altered by a modifying factor; and

if said difference value is greater than or equal to said second threshold value, the selected reference image pixel value is replaced with a new reference image pixel value equal to a weighted and scaled selected reference image pixel value.

14. The method of claim 13 wherein said modifying factor is the difference between said selected pixel value in the next image frame and said selected pixel value in the reference image multiplied by a responsiveness factor, where said responsiveness factor limits the amount of the modification to ensure that the modified reference image only contains information regarding information within said next image frame that represents stationary objects within the scene and where said weighted and scaled selected reference image pixel value represents illumination changes within said scene.

15. The method of claim 13 wherein said second threshold value is a function of the difference factor.

16. The method of claim 7 further comprising the step of converting the two-dimensional motion image into a one-dimensional data set.

17. The method of claim 16 wherein the converting step further comprises the steps of:

defining a zone within said two-dimensional motion image containing a plurality of rows of pixel values;

integrating the pixel values across each said row of pixel values within said zone to produce an integrated pixel value for each of said rows of pixel values; and

subsampling said integrated pixel values to produce said one-dimensional data set.

18. The method of claim 17 further comprising the steps of:

detecting object fragments within said one dimensional data set, where said object fragments are a group of significantly-valued integrated pixel values that are adjacent one another;

estimating object motion of said object fragments between two consecutive next image frames; and

identifying objects within the zone in response to the estimated object motion of said object fragments.

19. The method of claim 18 further comprising the step of filtering said identified objects to eliminate erroneous moving object identifications.

20. The method of claim 19 wherein said filtering step comprises:

identifying the pixels within the zone having a brightness value greater than a brightness threshold as bright pixels;

computing a maximum absolute value of the pixel values within the zone;

identifying the adjacent pixels within the zone having an absolute difference of pixel values that exceeds an absolute difference threshold as intense valued pixels;

using the bright pixels, the maximum absolute value and the intense values pixels to eliminate moving object identifications that represent movement of non-physical objects within the zone.

21. The method of claim 7 further comprising the steps of:

detecting object fragments within said two-dimensional motion image, where said object fragments are a group of significantly-valued integrated pixel values that are adjacent one another;

estimating object motion of said object fragments between two consecutive next image frames; and

identifying objects within the scene in response to the estimated object motion of said object fragments.

22. The method of claim 21 further comprising the step of filtering said identified objects to accurately identify physical moving objects and eliminate non-physical moving object identifications.

23. The method of claim 22 wherein said filtering step comprises:

identifying the pixels within the two-dimensional motion image having a brightness value greater than a brightness threshold as bright pixels;

computing a maximum absolute value of the pixel values within the two-dimensional motion image;

identifying the adjacent pixels within the two-dimensional motion image having an absolute difference of pixel values that exceeds an absolute difference threshold as intense valued pixels;

using the bright pixels, the maximum absolute value and the intense valued pixels to eliminate non-physical moving objects from the identified moving objects.

24. Apparatus for image processing comprising:

imaging means for supplying a continuous sequence of image frames representing a scene;

reference image initializing means, connected to said imaging means, for initializing a reference image that contains image information regarding stationary objects within the scene;

comparator means, connected to said imaging means and said reference image initializing means, for comparing an image frame supplied by said imaging means to said reference image to produce a two-dimensional motion image representing motion information regarding movement of objects within the scene;

means for updating said reference image with information within said image frame, where said information used for updating the reference image only represents stationary objects within the scene and substantially disregards moving objects and temporarily stationary objects within the scene.

25. The apparatus of claim 24 further comprising stabilization means, connected to said imaging means, for stabilizing each image frame to remove any motion within said image frames that occurs due to imaging means movement rather than object movement within the scene.

26. The apparatus of claim 24 wherein said updating means comprises:

means for selecting a pixel value in the reference image and a pixel value in the next image frame having the same pixel location as the selected pixel value in the reference image;

means for computing a difference factor equal to the absolute value of the difference between said selected pixel value in the next image frame and said pixel value in the reference image;

means for comparing the difference value to a first threshold value; and

means for replacing reference image pixel, where if said difference value is less than said first threshold value, the pixel value in the reference image is replaced with an updated reference pixel value equaling the selected reference image pixel value altered by an update factor, and if said difference value is greater than or equal to said first threshold value, the pixel value in said reference image remains the same.

27. The apparatus of claim 26 wherein said update factor is the difference between said selected pixel value in the next image and said pixel value in the reference image multiplied by a responsiveness factor, where said responsiveness factor limits the amount of the update to ensure that the updated reference image only contains information regarding information within said image frame that represents stationary objects within the scene.

28. The apparatus of claim 26 wherein said first threshold value is a function of the difference factor.

29. The apparatus of claim 24 further comprising means for modifying said reference image using image information within said image frame regarding changes to illumination of the scene.

30. The apparatus of claim 29 wherein said modifying means comprises:

means for selecting a pixel value in the reference image and a pixel value in the image frame having the same pixel location as the selected reference image pixel value;

means for computing a difference factor equal to the absolute value of the difference between said selected pixel value in the next image and said pixel value in the reference image;

means for comparing the difference value to a second threshold value; and

means for replacing the selected reference image pixel, where if said difference value is less than said second threshold value, the pixel value in the reference image is replaced with a modified reference pixel value equaling the selected reference image pixel value altered by a modifying factor, and if said difference value is greater than or equal to said second threshold value, the selected reference image pixel value is replaced with a new reference image pixel value equal to a weighted and scaled selected reference image pixel value.

31. The apparatus of claim 30 wherein said modifying factor is the difference between said selected pixel value in the image frame and said selected pixel value in the reference image multiplied by a responsiveness factor, where said responsiveness factor limits the amount of the modification to ensure that the modified reference image only contains information regarding information within said next image frame that represents stationary objects within the scene and where said weighted and scaled selected reference image pixel value represents illumination changes within said scene.

32. The apparatus of claim 30 wherein said second threshold value is a function of the difference factor.

33. The apparatus of claim 24 further comprising means, connected to said comparator means, for converting the two-dimensional motion image into a one-dimensional data set.

34. The apparatus of claim 33 wherein the converting means further comprises:

means for defining a zone within said two-dimensional motion image containing a plurality of rows of pixel values;

means for integrating the pixel values across each said row of pixel values within said zone to produce an integrated pixel value for each of said rows of pixel values; and

means for subsampling said integrated pixel values to produce said one-dimensional data set.

35. The apparatus of claim 34 further comprising:

means for detecting object fragments within said one dimensional data set, where said object fragments are a group of significantly-valued integrated pixel values that are adjacent one another;

means for estimating object motion of said object fragments between two consecutive next image frames; and

means for identifying objects within the zone in response to the estimated object motion of said object fragments.

36. The apparatus of claim 35 further comprising filter means for filtering said identified objects to eliminate erroneous moving object identifications.

37. The apparatus of claim 36 wherein said filter means comprises:

means for identifying the pixels within the zone having a brightness value greater than a brightness threshold as bright pixels;

means for computing a maximum absolute value of the pixel values within the zone;

means for identifying the adjacent pixels within the zone having an absolute difference of pixel values that exceeds an absolute difference threshold as intense valued pixels;

means for using the bright pixels, the maximum absolute value and the intense values pixels to eliminate moving object identifications that represent movement of non-physical objects within the zone.

38. The apparatus of claim 24 further comprising:

means for detecting object fragments within said two-dimensional motion image, where said object fragments are a group of significantly-valued integrated pixel values that are adjacent one another;

means for estimating object motion of said object fragments between two consecutive next image frames; and

means for identifying objects within the scene in response to the estimated object motion of said object fragments.

39. The apparatus of claim 38 further comprising filter means for filtering said identified objects to accurately identify physical moving objects and eliminate non-physical moving object identifications.

40. The apparatus of claim 39 wherein said filter means comprises:

means for identifying the pixels within the two-dimensional motion image having a brightness value greater than a brightness threshold as bright pixels;

means for computing a maximum absolute value of the pixel values within the two-dimensional motion image;

means for identifying the adjacent pixels within the two-dimensional motion image having an absolute difference of pixel values that exceeds an absolute difference threshold as intense valued pixels; and

means for using the bright pixels, the maximum absolute value and the intense valued pixels to eliminate non-physical moving objects from the identified moving objects.