Object detection based on determination of pixel state

• US 8,311,273 B2
• Filed: 06/04/2007
• Issued: 11/13/2012
• Est. Priority Date: 06/05/2006
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Ikeda, Hiroo
• Primary Examiner(s): Chen, Wenpeng
• Application Number:
  US11/806,778
• Publication Number:
  US 20070280540A1
• Time in Patent Office: 1,989 Days
• Field of Search: 382/103, 382/173
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Abstract

An object detecting apparatus and method includes a pixel state determining unit that derives variance value for temporal properties of pixel characteristics of an input image, background model generating unit that adaptively generates a background model from characteristics in the characteristic storing unit and characteristic storing unit for background model generation using the characteristic distance and the pixel state determined as conditions, and an object judging unit that judges an object based on a characteristic distance indicative of a degree of similarity between a generated background model and pixel characteristics of an input image.

Claims

1. An object detecting apparatus comprising:

a pixel state determining unit that determines a pixel state indicative of temporal properties of pixel characteristics of an input image;

a background model generating unit that adaptively generates a background model from said pixel state and a first characteristic distance as conditions to process per each pixel unit, the first characteristic distance being indicative of a degree of similarity between pixel characteristics of the input image and a preceding background model; and

an object judging unit that judges an object based on a second characteristic distance indicative of a degree of similarity between a generated background model and pixel characteristics of the input image,

wherein said pixel state has the temporal properties of pixel characteristics of the input image, and is selected in a state from among the following states;

a stationary state that is a stable state over a short period of time, a moving state that is an unstable state over a short period of time as compared to a long period of time, a continuously stationary state that is a stable state over a long period of time, and a continuously moving state that is an unstable state over a long period of time.

2. The object detecting apparatus as defined in claim 1, wherein said background model is constituted by a mean of characteristics and a variance of characteristics.

3. The object detecting apparatus as defined in claim 1, wherein a background model is constituted by a weighted mean of characteristics and a weighted variance of characteristics, and

wherein the weights in said weighted mean and said weighted variance are set so that the latest value is a maximum value, and reduces over elapsed time.

4. The object detecting apparatus as defined in claim 1, wherein a background model is constituted by a plurality of units of characteristics, i.e., a plurality of means, a plurality of variances and a plurality of weighting coefficients under assumption of mixture of normal distributions;

background models having weighting coefficients not less than a threshold value are selected from background models; and

a smallest characteristic distance among a plurality of characteristic distances derived from characteristics extracted from an input image and a plurality of means and a plurality of variances in the characteristics of said selected background model is recognized as the characteristic distance.

5. The object detecting apparatus as defined in claim 1, wherein characteristics are constituted by multi-dimensional vector;

a background model is constituted by multi-dimension of a mean vector and a covariance matrix; and

a variance value is a sum of diagonal components or a maximum value of the diagonal components of covariance matrices of characteristics (from a latest value to a value a short while ago) stored in characteristic storing unit.

6. The object detecting apparatus according to claim 1, wherein the pixel state determining unit determines a pixel state by selecting from a plurality of pixel states, each indicating a different state of pixel characteristics over a period of time.

7. The object detecting apparatus according to claim 1, wherein the background model generating unit adaptively generates the background model directly from said pixel state and a characteristic distance as conditions to process per each pixel unit.

8. The object detecting apparatus according to claim 1, wherein the temporal properties of the pixel characteristics are determined from time-sequential characteristics of the input image over time.

9. An object detecting apparatus comprising:

a pixel state determining unit that determines a pixel state indicative of temporal properties of pixel characteristics of an input image;

a background model generating unit that adaptively generates a background model from said pixel state and a first characteristic distance as conditions to process per each pixel unit, the first characteristic distance being indicative of a degree of similarity between pixel characteristics of an input image and a preceding background model; and

an object judging unit that judges an object based on a second characteristic distance indicative of a degree of similarity between a generated background model and pixel characteristics of an input image,

wherein said pixel state determining unit determines a pixel state from variance values based on characteristics (from a latest value to a value a short while ago) stored in characteristic storing unit that time-sequentially stores pixel characteristics of an input image and a ratio R_S between the numbers of variance values not more than a threshold value for a stable state in characteristic variance value storing unit that time-sequentially stores said variance values and all stored data.

10. The object detecting apparatus as defined in claim 9, wherein said pixel state determining unit determines that said pixel state is continuously stationary when the ratio R_S is not less than a threshold for a continuously stationary state, and a continuously moving state when said ratio R_S is not more than a threshold value for a continuously moving state, and

wherein, when said pixel state is neither continuously stationary nor continuously moving, said pixel state determining unit determines that said pixel state is stationary when a variance value is not more than the threshold value for a stable state, and moving when said variance value is more than said threshold value for a stable state.

11. The object detecting apparatus as defined in claim 9, wherein:

said background model generating unit judges that said pixel state is a stationary background when the pixel state is continuously stationary, generates a background model from characteristics stored in characteristic storing unit that time-sequentially stores pixel characteristics of an input image, and copies all characteristics stored in said characteristic storing unit into characteristic storing unit for background model generation,

said background model generating unit judges that said pixel state is a moving background when the pixel state is continuously moving, generates a background model from characteristics stored in said characteristic storing unit, and copies all characteristics stored in said characteristic storing unit into said characteristic storing unit for background model generation,

said background model generating unit judges that pixel characteristics of an input image are a background when said pixel state is stationary and the first characteristic distance that indicates a degree of similarity between pixel characteristics of said input image and the preceding background model is not more than a threshold value, indicating a high degree of similarity, stores latest characteristics stored in said characteristic storing unit into said characteristic storing unit for background model generation, and generates the background model from characteristics stored in said characteristic storing unit for background model generation, and

no background model is generated and the preceding background model is utilized by the object judging unit when a pixel state does not meet any of the above conditions.

12. An object detecting apparatus that judges an object by dividing an image, comprising:

an image dividing unit that divides an input image;

a temporal characteristic determining unit that determines temporal characteristics of an image per divided image unit; and

an image unit specifying unit that analyzes temporal characteristics of a plurality of image units and specifies an image unit in which an object exists,

wherein, when analyzing temporal characteristics of a plurality of image units, said image unit specifying unit extracts distinctive features of parts with similarity in temporal properties to each other from a time sequence of characteristic information obtained per image unit, and

wherein the temporal characteristics have one of following characteristics selected from among the following states;

a stable state over a short period of time, an unstable state over a short period of time as compared to a long period of time, a stable state over a long period of time, and an unstable state over a long period of time.

13. The object detecting apparatus as defined in claim 12, wherein the temporal characteristic determining unit derives the temporal characteristics from temporal transition information in a state determined by the relationship between a degree of temporal stability of characteristic information obtained per image unit and a continuity degree how long said degree of temporal stability continues.

14. The object detecting apparatus as defined in claim 13, wherein the characteristic information includes at least one of the following information of an image unit selected from the group consisting of the intensity, hue, each value of RGB (red green blue color model), edge value, optical flow value, and the distance value in a distance image.

15. The object detecting apparatus as defined in claim 12, wherein, when analyzing temporal characteristics of a plurality of image units, said image unit specifying unit recognizes characteristic information different from said distinctive features as an image unit in which an object exists.

16. The object detecting apparatus as defined in claim 12, wherein said image unit is a pixel.

17. The object detecting apparatus according to claim 12, wherein the temporal characteristics are selected from among a choice of the stable state over a short period of time, the unstable state over the short period of time as compared to the long period of time, the stable state over the long period of time, and the unstable state over the long period of time.

18. A method for detecting an object, the method comprising:

a pixel state having temporal properties of pixel characteristics of an input image;

generating adaptively a background model using said pixel state and a characteristic distance that indicates a degree of similarity between pixel characteristics of an input image and a preceding background model as conditions per pixel unit; and

judging an object based on a characteristic distance that indicates a degree of similarity between a generated background model and the pixel characteristics of the input image,

wherein a pixel state has the temporal properties of the pixel characteristics of the input image, and is selected from the following states;

a stationary state that is a stable state over a short period of time as compared to a long period of time, a moving state that is an unstable state over a short period of time, a continuously stationary state that is a stable state over a long period of time, and a continuously moving state that is an unstable state over a long period of time.

19. The method for detecting an object as defined in claim 18, wherein the background model is constituted by a mean of characteristics and a variance of characteristics.

20. The method for detecting an object as defined in claim 18, wherein the background model is constituted by a weighted mean of characteristics and a weighted variance of characteristics;

and the weights in said weighted mean and said weighted variance are set so that the latest value is largest, and the older a value is, the smaller it becomes.

21. The method for detecting an object as defined in claim 18, wherein a background model is constituted by a plurality of units of characteristics i.e., a plurality of means, a plurality of variances, and a plurality of weighting coefficients under assumption of mixture of normal distributions;

a background model having weighting coefficients not less than a threshold value is selected from the background models; and

a smallest characteristic distance among a plurality of characteristic distances derived from characteristics extracted from an input image and a plurality of characteristic distance obtained from a plurality of means and variances of said selected background model is recognized as the characteristic distance.

22. The method for detecting an object as defined in claim 18 wherein characteristics are constituted by multi-dimensional vector;

a background model is constituted as multi-dimension by a mean vector and a covariance matrix; and

a variance value is a sum of diagonal components of covariance matrices or the maximum value of the diagonal components of characteristics (from a latest value to a value a short while ago) stored in characteristic storing unit.

23. The method according to claim 18, wherein the pixel state is selected from among the following four pixel states:

the stationary state that is the stable state over the short period of time as compared to the long period of time, the moving state that is the unstable state over the short period of time, the continuously stationary state that is the stable state over the long period of time, and the continuously moving state that is the unstable state over the long period of time.

24. A method for detecting an object, the method comprising:

a pixel state having temporal properties of pixel characteristics of an input image;

generating adaptively a background model using said pixel state and a characteristic distance that indicates a degree of similarity between pixel characteristics of an input image and a preceding background model as conditions per pixel unit; and

judging an object based on a characteristic distance that indicates a degree of similarity between a generated background model and the pixel characteristics of the input image, wherein a pixel state is determined from variance values based on characteristics (from a latest value to a value a short while ago) stored in characteristic storing unit that time-sequentially stores pixel characteristics of an input image and a ratio R_s between the numbers of variance values not more than a threshold value for a stable state in characteristic variance value storing unit that time-sequentially stores said variance values and all stored data in the determining of the pixel state.

25. The method for detecting an object as defined in claim 24, wherein, in the determining of the pixel state, a pixel state is determined to be continuously stationary when the ratio R_S is not less than a threshold for a continuously stationary state, and continuously moving when said ratio R_S is not more than a threshold value for a continuously moving state;

and

when a pixel state is determined to be neither continuously stationary nor continuously moving, a pixel state is determined to be stationary when a variance value is not more than a threshold value for a stable state, and moving when said variance value is more than said threshold value for a stable state.

26. The method for detecting an object as defined in claim 24, wherein the generating of the background model comprises a pixel state is judged to be a stationary background when the pixel state is continuously stationary, a background model is generated from characteristics stored in characteristic storing unit that time-sequentially stores pixel characteristics of an input image, and all characteristics stored in said characteristic storing unit are copied into characteristic storing unit for background model generation,

wherein the pixel state is judged to be a moving background when the pixel state is continuously moving, a background model is generated from characteristics stored in said characteristic storing unit, and all characteristics stored in said characteristic storing unit are copied into said characteristic storing unit for background model generation, and

wherein the pixel characteristics of the input image are judged to be a background when a pixel state is stationary and a characteristic distance that indicates a degree of similarity between pixel characteristics of an input image and a preceding background model is not more than a threshold value, indicating a high degree of similarity, latest characteristics stored in said characteristic storing unit are stored into said characteristic storing unit for background model generation, and a background model is generated from characteristics stored in said characteristic storing unit for background model generation;

and no background model is generated and a preceding background model is utilized when a pixel state does not meet any of the above conditions.

27. A method for detecting an object, in which an object is judged by dividing an image, comprising:

dividing an input image;

determining temporal characteristics of an image per divided image unit; and

specifying an image unit by analyzing temporal characteristics of a plurality of image units and specifying image units in which an object exists per image unit,

wherein the temporal characteristics is selected from following characteristics;

a stable state over a short period of time, an unstable state over a short period of time as compared to a long period of time, a stable state over a long period of time, and an unstable state over a long period of time.

28. The method for detecting an object as defined in claim 27, wherein the temporal characteristic determining comprising temporal characteristics derived from temporal transition information in a state determined by a relationship between a degree of temporal stability of characteristic information obtained per image unit and a continuity degree how long said degree of temporal stability continues.

29. The method for detecting an object as defined in claim 27, wherein, when the temporal characteristics of a plurality of image units are analyzed in the specifying of the image unit, distinctive features of parts with similarity in temporal properties to each other are extracted from the time sequence of characteristic information obtained per image unit, and

wherein the characteristic information different from said distinctive features is recognized as an image unit in which an object exists.

30. The method for detecting an object as defined in claim 27, wherein the image unit is a pixel.

31. An object detection program, stored in a non-transitory computer readable medium, having a computer execute a method comprising:

a pixel state determination processing, executable by a processing unit of the computer, that determines a pixel state having temporal properties of pixel characteristics of an input image for display on a display device;

a background model generation processing, executable by the processing unit, that adaptively generates a background model using said pixel state and a characteristic distance that indicates a degree of similarity between pixel characteristics of said input image and a preceding background model as conditions per pixel unit; and

an object judgment processing, executable by the processing unit, that judges an object based on a characteristic distance that indicates a degree of similarity between a generated background model and pixel characteristics of an input image,

wherein the temporal properties is selected from following properties;

a stable state over a short period of time, an unstable state over a short period of time as compared to a long period of time, a stable state over a long period of time, and an unstable state over a long period of time.

32. An object detection program, stored in a non-transitory computer readable medium, having a computer execute a method comprising:

a pixel state determination processing, executable by a processing unit of the computer, that determines a pixel state having temporal properties of pixel characteristics of an input image for display on a display device;

a background model generation processing, executable by the processing unit, that adaptively generates a background model using said pixel state and a characteristic distance that indicates a degree of similarity between pixel characteristics of said input image and a preceding background model as conditions per pixel unit; and

an object judgment processing, executable by the processing unit, that judges an object based on a characteristic distance that indicates a degree of similarity between a generated background model and pixel characteristics of an input image, wherein the pixel state determination processing comprises the pixel state being determined from variance values based on characteristics (from a latest value to a value a short while ago) stored in characteristic storing unit that time-sequentially stores pixel characteristics of an input image and a ratio R_S between the numbers of variance values not greater than a threshold value for a stable state in characteristic variance value storing unit that time-sequentially stores said variance values and all stored data.

33. The object detection program as defined in claim 32, wherein the pixel state determination processing comprises a pixel state being determined to be continuously stationary when a ratio R_S is not less than a threshold for a continuously stationary state, and continuously moving when said ratio R_S is not more than a threshold value for a continuously moving state;

and when a pixel state is determined to be neither continuously stationary nor continuously moving, a pixel state is determined to be stationary when a variance value is not more than a threshold value for a stable state, and moving when said variance value is more than said threshold value for a stable state.

34. The object detection program as defined in claim 32, wherein the background model generation processing comprises:

when a pixel state is continuously stationary, the background model is generated from characteristics stored in characteristic storing unit that time-sequentially stores pixel characteristics of an input image, and all characteristics stored in said characteristic storing unit are copied into characteristic storing unit for the background model generation;

when a pixel state is continuously moving, the background model is generated from characteristics stored in said characteristic storing unit, and all characteristics stored in said characteristic storing unit are copied into said characteristic storing unit for background model generation;

when the pixel state is stationary and a characteristic distance that indicates a degree of similarity between pixel characteristics of an input image and a preceding background model is not more than a threshold value, indicating a high degree of similarity, latest characteristics stored in said characteristic storing unit are stored into said characteristic storing unit for background model generation, and a background model is generated from characteristics stored in said characteristic storing unit for background model generation; and

no background model is generated and a preceding background model is utilized when the pixel state does not meet any of the above conditions.

35. An object detection program, stored in a non-transitory computer readable medium, having a computer that judges an object by dividing an image execute a method, comprising:

an image division processing, executable on a processor of the computer, that divides an input image to display on a display device;

a temporal characteristic determination processing, executable on the processor, that determines temporal characteristics of an image per divided image unit; and

an image unit specification processing, executable on the processor, that analyzes temporal characteristics of a plurality of image units and that specifies an image unit in which an object exists per image unit,

wherein the temporal characteristics is selected from following characteristics;

a stable state over a short period of time, an unstable state over a short period of time as compared to a long period of time, a stable state over a long period of time, and an unstable state over a long period of time.

36. The object detection program as defined in claim 35, wherein the temporal characteristic determination processing comprises the temporal characteristics being derived from temporal transition information in a state determined by the relationship between a degree of temporal stability of characteristic information obtained per image unit and a continuity degree of a length of time that said degree of temporal stability continues.

37. The object detection program as defined in claim 35, wherein the image unit specification processing comprises, when temporal characteristics of a plurality of image units are analyzed, distinctive features of parts with similarity in temporal properties to each other are extracted from the time sequence of characteristic information obtained per image unit;

and characteristic information different from said extracted distinctive features is recognized as an image unit in which an object exists.

38. An object detecting apparatus comprising:

a pixel state determining unit that determines a pixel state indicative of temporal properties of pixel characteristics of an input image;

a background model generating unit that adaptively generates a background model from said pixel state and a first characteristic distance as conditions to process per each pixel unit, the first characteristic distance being indicative of a degree of similarity between pixel characteristics of an input image and a preceding background model; and

an object judging unit that judges an object based on a second characteristic distance indicative of a degree of similarity between a generated background model and pixel characteristics of an input image, wherein;

the background model generating unit generates the background model comprising an updated background model when the pixel state is stationary and the first characteristic distance is equal to or less than a characteristic distance threshold, and

the object judging unit judges the object based on the second characteristic distance indicative of the degree of similarity between the generated background model and the pixel characteristics of the input image, the generated background model being either the updated background model when the pixel state is stationary and the first characteristic distance is equal to or less than a characteristic distance threshold, or the preceding background model when the pixel state is not stationary or when the first characteristic distance is more than the characteristic distance threshold.