Target detecting system and method

• US 7,688,999 B2
• Filed: 11/28/2005
• Issued: 03/30/2010
• Est. Priority Date: 12/08/2004
• Status: Expired due to Fees

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Hye-Jin, Kim, Kye-Kyung, Cho, Soo-Hyun, Lee, Jae-Yeon
• Primary Examiner(s): Ahmed; Samir A.
• Assistant Examiner(s): PATEL, JAYESH A
• Application Number:
  US11/287,936
• Publication Number:
  US 20060140481A1
• Time in Patent Office: 1,583 Days
• Field of Search: 382/103, 382/190
• US Class Current: 382/103
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A target detecting system and method for detecting a target from an input image is provided. According to the target detecting system and method, when a target is detected from an input image and there are moving areas in the input image, camera movement parameters are obtained, image frames are transformed, and movement candidate areas are extracted from the image frame and the previous input image frame. In addition, image feature information is extracted from the input image, and based on the movement candidate areas and the image feature information a shape of the target is extracted. Therefore, the target can be exactly and rapidly extracted and tracked.

Claims

1. A target detecting method for detecting a target from an input image from a camera, comprising the steps of:

a shape detector configured for obtaining camera movement parameters from a movement area in an input image from the camera, and correcting an image frame using the obtained movement parameters,

extracting movement candidate areas from the input image frame and a previously input image frame,

extracting image feature information from the input image, and

extracting a target shape based on the movement candidate areas and the image feature information;

a tracker is configured for generating blob information from the extracted shape, and

wherein the blob information is an image or an image portion formed by binding adjacent pixels having the same characteristics in the image;

extracting blob information for the extracted shape; and

checking whether the target is correct by comparing the blob information with that of the shape extracted at the previous frame regardless of background or the camera movement,

wherein the extracting image feature information from the input image further comprises skin color areas are extracted from the input image, skin color being extracted as follows;

$\mathrm{If}<br><br>\left[C_t^b<br><br>\left(x,y\right)><br><br>\mathrm{\alpha <br><br>}\times <br><br>C_t^r<br><br>\left(x,y\right)<br><br>\&<br><br><br><br><br><br>C_t^b<br><br>\left(x,y\right)><br><br>\mathrm{\beta <br><br>}\times <br><br>C_t^r<br><br>\left(x,y\right)<br><br>\&<br><br><br><br><br><br>C_t^g<br><br>\left(x,y\right)><br><br>\mathrm{\gamma <br><br>}\times <br><br>C_t^r<br><br>\left(x,y\right)<br><br>\&<br><br><br><br><br><br>C_t^g<br><br>\left(x,y\right)><br><br>\mathrm{\delta <br><br>}\times <br><br>C_t^r<br><br>\left(x,y\right)\right]<br><br>\&<br><br><br><br>[<br><br>\mathrm{\varepsilon <br><br>}<<br><br><br><br>H[<br><br>x+w\times <br><br>y]<<br><br>\mathrm{\eta <br><br>}]<br><br>\&<br><br><br><br>[<br><br>\mathrm{\omega <br><br>}<<br><br><br><br>S[<br><br>x+w\times <br><br>y]<<br><br>\mathrm{\zeta <br><br>}]<br><br><br><br>\mathrm{Then}<br><br><br><br>f_t<br><br>\left(x+w\times <br><br>y\right)<br><br><br><br>\mathrm{is}<br><br><br><br><br><br>\mathrm{skincolor}$

wherein C_t^r, C_t^g, and C_t^b are respectively r, g, and b values of the t-th frame images, H and S are respectively H(ue) and S(aturation) values obtained from C_t^r, C_t^g, and C_t^b, α

, β

, γ

, δ

, ε

, η

, ω

, ζ

are constants obtained by experiments, and w is a width of the image.

2. The target detecting method of claim 1, wherein the obtaining camera movement parameters from the movement area in the input image from the camera, and correcting the image frame using the obtained movement parameters further comprises the input image frame is compensated using the camera movement parameters when the movement area is greater than a predetermined threshold value (T₂).

3. The target detecting method of claim 1, further comprising prior to the obtaining camera movement parameters from the movement area in the input image from the camera, and correcting the image frame using the obtained movement parameters further comprises the additional step of determining if there is a motion pixel, which is a pixel with a brightness difference of pixels (x,y) between consecutive image frames greater than a predetermined threshold value (T₁), and then, at the obtaining camera movement parameters from the movement area in the input image from the camera, and correcting the image frame using the obtained movement parameters further comprises the image frame is transformed on there being a motion pixel.

4. The target detecting method of claim 3, wherein the motion pixel is extracted as follows,

$\mathrm{If}\; <li></li><br><br><br><br>f_t<br><br>\left(x,y\right)-f_{t-1}<br><br>\left(x,y\right)<br><br>><br><br>T_1<br><br>\&<br><br><br><br><br><br><br><br>f_{t+1}<br><br>\left(x,y\right)-f_t<br><br>\left(x,y\right)<br><br>><br><br>T_1<br><br><br><br>\mathrm{then}<br><br><br><br>m_{t+1}<br><br>\left(x,y\right)<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{moving}<br><br><br><br>\mathrm{object}<br><br><br><br>\mathrm{pixel}$

wherein f_1-1, f_t, and f_t+1 are (t−

1)-th, t-th, and (t+1)-th frames respectively, T₁ is a threshold value for a brightness difference between frames, f(x,y) is a brightness value of a pixel (x,y) at a frame, and m_t+1 (x,y) is a motion pixel (x,y) of (t+1)-th frame.

5. The target detecting method of claim 4, wherein the threshold value (T₁) satisfies 10≦

T₁≦

20.

6. The target detecting method of claim 3, wherein the motion pixel is extracted from the entire pixels of the input image.

7. The target detecting method of claim 3, wherein the obtaining camera movement parameters from the movement area in the input image from the camera, and correcting the image frame using the obtained movement parameters further comprises whether the movement area is greater than a predetermined threshold value (T₂) is determined by checking whether a difference between a maximum value x and a minimum value x among connected components of the motion pixels is greater than a predetermined threshold value (T₂) in comparison with the entire image width.

8. The target detecting method of claim 1, wherein the camera movement parameter is defined to correct the camera right/left movement and includes x/y-direction movement parameters.

9. The target detecting method of claim 8, wherein the x-direction movement parameter is extracted for a part of the image from the edge compared values between the pixels of a predetermined row i of the t-th image frame and the pixels of a predetermined row j of the (t−

1)-th image frame.

10. The target detecting method of claim 9, wherein the x-direction movement parameter (pan) is extracted as follows:

$h_t<br><br>\left(x\right)=\underset{y=0}{\overset{h}{\sum <br><br>}}<br><br>\underset{k=-p}{\overset{p}{\sum <br><br>}}<br><br>\underset{l=o}{\overset{\mathrm{dh}}{\sum <br><br>}}<br><br>\underset{j=p}{\overset{w-p}{\sum <br><br>}}<br><br>\left(E_t<br><br>\left(y+l,j\right)-E_{t-1}<br><br>\left(y+l,j+k\right)\right)$ $\mathrm{pan}=\arg <br><br><br><br>\min <br><br><br><br>h_t<br><br>\left(x\right)$

wherein y is a height of the image in the range of 0 to h, k is a possible range which the camera will move right/left, j is a range of x values, l is a y area range for which edge values are detected, E_t(y+l, j) and E<sub>t−

1</sub>(y+l, j+k) are an edge value of the pixels (y+l, j) and (y+l, j+k) on t-th and (t−

1)-th edge image frames, respectively, and h_t(x) is a defined value to obtain an x-direction movement parameter.

11. The target detecting method of claim 8, wherein the y-direction movement parameter is extracted for a part of the image from the edge compared values between pixels of a predetermined row i of the t-th image frame and pixels of a predetermined row j of the (t−

1)-th image frame.

12. The target detecting method of claim 11, wherein the y-direction movement parameter (tilt) is extracted as follows:

${\mathrm{\upsilon <br><br>}}_t<br><br>\left(y\right)=\underset{x=o}{\overset{w}{\sum <br><br>}}<br><br>\underset{k=-t}{\overset{t}{\sum <br><br>}}<br><br>\underset{l=o}{\overset{\mathrm{dv}}{\sum <br><br>}}<br><br>\underset{i=t}{\overset{h-t}{\sum <br><br>}}<br><br>\left(E_t<br><br>\left(i,x+l\right),E_{t-1}<br><br>\left(i+k,x+l\right)\right)$ $\mathrm{tilt}=\arg <br><br><br><br>\min <br><br><br><br>v_t<br><br>\left(y\right)$

wherein x is a width of the image in the range of 0 to w, k is a possible range which the camera will move up/down, j is a range of y values, l is an x area range for which edge values are detected, E_t(i, x+l) and E<sub>t−

1</sub>(i+k, x+l) are an edge value of the pixels (i, x+l) and (i+k, x+l) on t-th and (t−

1)-th edge image frames, respectively, and v_t(y) a defined value to obtain a y-direction movement parameter.

13. The target detecting method of claim 1, wherein the obtaining camera movement parameters from the movement area in the input image from the camera, and correcting the image frame using the obtained movement parameters further comprises correcting an image frame using the obtained camera movement parameters, x′

and y′

values of the t-th frame are transformed to be x′

=x+pan and y′

=y+tilt by using x-direction movement parameter (pan), y-direction movement parameter (tilt), and x and y of the (t−

1)-th frame.

14. The target detecting method of claim 1, wherein the extracting movement candidate areas from the input image frame and the previously input image frame further comprises the movement candidate areas are extracted by generating an image difference between the t-th frame image and the (t−

1)-th frame image.

15. The target detecting method of claim 1, wherein the image feature information includes at least one among skin color, shape, and corner information.

16. The target detecting method of claim 1, wherein the extracting image feature information from the input image further comprises the shape feature information is extracted by a color clustering method of the input image.

18. The target detecting method of claim 1, wherein the extracting the target shape based on the movement candidate areas and the image feature information further comprises skin color areas are extracted from the movement candidate areas and then the target image is extracted using the shape feature information and the edge feature information.

19. The target detecting method of claim 1, wherein the blob information includes at least one among blob position, blob size, blob shape, and color distribution.

20. The target detecting method of claim 1, wherein the checking whether the target is correct by comparing the blob information with that of the shape extracted at the previous frame further comprises:

removing blobs corresponding to noise by matching the blob information for the extracted target shape information with the blob information of the shape extracted at the previous frame; and

checking whether the target is correct by extracting a color distribution of the extracted blobs.

21. The target detecting method of claim 20, wherein the removing blobs corresponding to noise by matching the blob information for the extracted target shape information with the blob information of the shape extracted at the previous frame further comprises determining whether the blobs correspond to noise based on the size of the blob.

17. A target detecting method for detecting a target from an input image from a camera, comprising the steps of:

a shape detector configured for obtaining camera movement parameters from a movement area in an input image from the camera, and correcting an image frame using the obtained movement parameters,

extracting movement candidate areas from the input image frame and a previously input image frame,

extracting image feature information from the input image, and

extracting a target shape based on the movement candidate areas and the image feature information;

a tracker is configured for generating blob information from the extracted shape, and

wherein the blob information is an image or an image portion formed by binding adjacent pixels having the same characteristics in the image;

extracting blob information for the extracted shape; and

checking whether the target is correct by comparing the blob information with that of the shape extracted at the previous frame regardless of background or the camera movement,

wherein the extracting image feature information from the input image further comprises the edge information is extracted from the input image, the edge information, that is, the edge value (E_t) of the pixel (x,y) being extracted as follows;

$E_t^v=-f_t<br><br>\left(x-1,y-1\right)+f_t<br><br>\left(x+1,y-1\right)-2⨯<br><br>f_t<br><br>\left(x-1,y\right)+2⨯<br><br>f_t<br><br>\left(x+1,y\right)-f_t<br><br>\left(x-1,y+1\right)+f_t<br><br>\left(x+1,y+1\right)$ $E_t^h=-f_t<br><br>\left(x-1,y-1\right)-2⨯<br><br>f_t<br><br>\left(x,y-1\right)-f_t<br><br>\left(x+1,y-1\right)+f_t<br><br>\left(x-1,y+1\right)+2⨯<br><br>f_t<br><br>\left(x,y+1\right)+f_t<br><br>\left(x+1,y+1\right)$ $E_t=\sqrt{{\left(E_t^v\right)}^2+{\left(E_t^h\right)}^2}$

wherein E_t is the t-th edge image frame, E_t^v is the vertical direction edge value of the pixel (x, y), E_t^h is the horizontal direction edge value of the pixel (x, y), and f_t(x,y) is the brightness value of the pixel (x,y).

22. A target detecting system for detecting a target from an input image, comprising:

a camera that obtains a target image to be detected and which transmits the obtained target image to a shape detector;

a shape detector, operatively coupled to said camera, said shape detector extracting a shape from the obtained target image based on movement candidate areas and image feature information extracted through an image difference of temporally-consecutive frames among an image sequence transmitted from the image input unit,

a tracker that extracts blob information for the extracted shape and verifies whether the target is correct by comparing the blob information with the shape extracted at the previous frame regardless of background or the camera movement, and

wherein the blob information is an image or an image portion formed by binding adjacent pixels having the same characteristics in the image,

wherein the shape detector further comprises;

a movement area extractor for extracting the movement candidate area from an image frame and a previous input image frame;

an image feature information extractor for extracting image feature information from the input image; and

a shape extractor for extracting the shape of the target image based on the movement candidate area and the image feature information, and

wherein the image feature information extractor extracts at least one among skin color, shape, and corner information from the input image, wherein;

the skin color is extracted as follows

$\mathrm{If}<br><br>\left[C_t^b<br><br>\left(x,y\right)><br><br>\mathrm{\alpha <br><br>}\times <br><br>C_t^r<br><br>\left(x,y\right)<br><br>\&<br><br><br><br><br><br>C_t^b<br><br>\left(x,y\right)><br><br>\mathrm{\beta <br><br>}\times <br><br>C_t^r<br><br>\left(x,y\right)<br><br>\&<br><br><br><br><br><br>C_t^g<br><br>\left(x,y\right)><br><br>\mathrm{\gamma <br><br>}\times <br><br>C_t^r<br><br>\left(x,y\right)<br><br>\&<br><br><br><br><br><br>C_t^g<br><br>\left(x,y\right)><br><br>\mathrm{\delta <br><br>}\times <br><br>C_t^r<br><br>\left(x,y\right)\right]<br><br>\&<br><br><br><br>[<br><br>\mathrm{\varepsilon <br><br>}<<br><br><br><br>H[<br><br>x+w\times <br><br>y]<<br><br>\mathrm{\eta <br><br>}]<br><br>\&<br><br><br><br>[<br><br>\mathrm{\omega <br><br>}<<br><br><br><br>S[<br><br>x+w\times <br><br>y]<<br><br>\mathrm{\zeta <br><br>}]<br><br><br><br>\mathrm{Then}<br><br><br><br>f_t<br><br>\left(x+w\times <br><br>y\right)<br><br><br><br>\mathrm{is}<br><br><br><br><br><br>\mathrm{skincolor}$

wherein C_t^r, C_t^g, C_t^b are respectively r, g, and b values of the t-th frame images, H and S are H(ue) and S(aturation) values obtained from C_t^r, C_t^g, and C_t^b, α

, β

, γ

, δ

, ε

, η

, ω

, ζ

are constants obtained by experiment, and w is a width of the image;

the edge value (E_t) of the pixel (x,y) is obtained as follows,

$E_t^v=-f_t<br><br>\left(x-1,y-1\right)+f_t<br><br>\left(x+1,y-1\right)-2⨯<br><br>f_t<br><br>\left(x-1,y\right)+2⨯<br><br>f_t<br><br>\left(x+1,y\right)-f_t<br><br>\left(x-1,y+1\right)+f_t<br><br>\left(x+1,y+1\right)$ $E_t^h=-f_t<br><br>\left(x-1,y-1\right)-2⨯<br><br>f_t<br><br>\left(x,y-1\right)-f_t<br><br>\left(x+1,y-1\right)+f_t<br><br>\left(x-1,y+1\right)+2⨯<br><br>f_t<br><br>\left(x,y+1\right)+f_t<br><br>\left(x+1,y+1\right)$ $E_t=\sqrt{{\left(E_t^v\right)}^2+{\left(E_t^h\right)}^2}$

wherein is the E_t is the t-th edge image frame, E_t^v is the vertical direction edge value of the pixel (x, y), E_t^h is the horizontal direction edge value of the pixel (x, y), and f_t(x,y) is the brightness value of the pixel (x,y);

and the shape feature information is extracted through color clustering of the input image.

23. The system for detecting a target of claim 22, wherein the movement area extractor extracts the movement candidate area by obtaining camera movement parameters when the movement area of the input image is greater than a predetermined threshold value (T₂) and using the obtained camera movement parameters to correct the image frame.

24. The system for detecting a target of claim 22, wherein the movement area extractor checks whether the movement area of the input image is greater than a predetermined threshold value (T₂) on there being a motion pixel, which is a pixel with a brightness difference between consecutive image frames greater than a predetermined threshold value (T₁).

25. The system for detecting a target of claim 24, wherein the motion pixel is extracted as follows:

$\mathrm{If}<br><br><br><br><br><br>f_t<br><br>\left(x,y\right)-f_{t-1}<br><br>\left(x,y\right)<br><br>><br><br>T_1<br><br>\&<br><br><br><br><br><br><br><br>f_{t+1}<br><br>\left(x,y\right)-f_t<br><br>\left(x,y\right)<br><br>><br><br>T_1<br><br><br><br>\mathrm{then}<br><br><br><br>m_{t+1}<br><br>\left(x,y\right)<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{moving}<br><br><br><br>\mathrm{object}<br><br><br><br>\mathrm{pixel}$

wherein f<sub>t−

1</sub>, t_t, and f_t+1 are respectively (t−

1)-th, t-th, (t+1)-th frames, T₁ is a threshold value for brightness difference between frames, f(x,y) is a brightness value of a pixel (x,y) at a frame, and m_t+1(x,y) is a motion pixel (x,y) of the (t+1)-th frame.

26. The system for detecting a target of claim 25, wherein the threshold value (T₁) satisfies 10≦

T₁≦

20.

27. The system for detecting a target of claim 24, wherein the motion pixel is extracted from the entire pixels of the input image.

28. The system for detecting a target of claim 24, wherein the movement area extractor checks, on there being motion pixels, whether a difference between a maximum value x and a minimum value x among connected components of the motion pixels is greater than a predetermined threshold value (T₂) in comparison with the entire image width when the movement area is greater than a predetermined threshold value (T₂).

29. The target detecting system of claim 23, wherein the camera movement parameters are defined to correct the right/left movement of the camera and include an x-direction movement parameter and a y-direction movement parameter, the x-direction movement parameter being extracted from the edge compared values between pixels of a predetermined row i of the t-th image frame and pixels of a predetermined row j of the (t−

1)-th image frame for a part of the image, and the y-direction movement parameter being extracted from the edge compared values between pixels of a predetermined row i of the t-th image frame and pixels of a predetermined row j of the (t−

1)-th image frame for a part of the image.

30. The target detecting system of claim 29, wherein the x-direction movement parameter (pan) is extracted as follows:

$h_t<br><br>\left(x\right)=\underset{y=0}{\overset{h}{\sum <br><br>}}<br><br>\underset{k=-}{\overset{p}{\sum <br><br>}}<br><br>\underset{\mathrm{pl}=o}{\overset{\mathrm{dh}}{\sum <br><br>}}<br><br>\underset{j=p}{\overset{w-p}{\sum <br><br>}}<br><br>\left(E_t<br><br>\left(y+l,j\right)-E_{t-1}<br><br>\left(y+l,j+k\right)\right)$ $\mathrm{pan}=\arg <br><br><br><br>\min <br><br><br><br>h_t<br><br>\left(x\right)$

wherein y is a height of the image in the range of 0 to h, k is a possible range which the camera will move right/left, j is a range of x values, l is a y area range checking edge values, E_t(y+l, j) and E<sub>t−

1</sub>(y+l, j+k) are an edge values of the pixels (y+l ,j) and (y+l, j+k) on t-th and (t−

1)-th edge image frames, respectively, and h_t(x) is a defined value to obtain an x-direction movement parameter.

31. The target detecting system of claim 29, wherein the y-direction movement parameter (tilt) is extracted as follows:

${\mathrm{\upsilon <br><br>}}_t<br><br>\left(y\right)=\underset{x=o}{\overset{w}{\sum <br><br>}}<br><br>\underset{k=-}{\overset{t}{\sum <br><br>}}<br><br>\underset{u=o}{\overset{\mathrm{dv}}{\sum <br><br>}}<br><br>\underset{i=t}{\overset{h-t}{\sum <br><br>}}<br><br>\left(E_t<br><br>\left(i,x+l\right),E_{t-1}<br><br>\left(i+k,x+l\right)\right)$ $\mathrm{tilt}=\arg <br><br><br><br>\min <br><br><br><br>v_t<br><br>\left(y\right)$

wherein x is a width of the image in the range of 0 to w, k is a possible range which the camera will move up/down, j is a range of y values, l is an x area range checking edge values, E_t(i, x+l) and E<sub>t−

1</sub>(i+k, x+l) are an edge values of the pixels (i, x+l) and (i+k, x+l) on t-th and (t−

1)-th edge image frames, respectively, and v_t(y) a defined value to obtain a y-direction movement parameter.

32. The target detecting system of claim 31, wherein the movement area extractor corrects the t-th frame image by x′

and y′

values of the t-th frame obtained as x′

=x+pan and y′

=y+tilt using the x-direction movement parameter (pan), the y-direction movement parameter (tilt), and x and y of the (t−

1)-th frame, and extracts the movement candidate areas by generating an image difference between the t-th frame image and (t−

1)-th frame image.

33. The target detecting system of claim 22, wherein the image characteristic information includes at least one among skin color, shape, and corner information.

34. The target detecting system of claim 22, wherein the shape extractor extracts the skin color areas from the movement candidate areas and then extracts the target area using the edge feature.

35. The target detecting system of claim 22, wherein the tracker includes a blob information extractor for extracting the blob information for the extracted shape of the target;

and

a human shape tracker for removing blobs corresponding to noise by matching the blob information for the extracted target shape information with the blob information of the shape extracted at the previous frame and checking whether the target is correct by extracting a color distribution of the extracted blobs.

36. The target detecting system of claim 22, wherein the blob information includes at least one among blob position, blob size, blob shape, and color distribution.