Target detecting system and method

• US 20060140481A1
• Filed: 11/28/2005
• Published: 06/29/2006
• Est. Priority Date: 12/08/2004
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Hye-Jin, Kim, Kye-Kyung, Lee, Jae-Yeon, Cho, Soo-Hyun
• Granted Patent: US 7,688,999 B2
• Time in Patent Office: Days
• US Class Current: 382/190
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A target detecting system and method for detecting a target from an input image is provided. According to the target detecting system and method, when a target is detected from an input image and there are moving areas in the input image, camera movement parameters are obtained, image frames are transformed, and movement candidate areas are extracted from the image frame and the previous input image frame. In addition, image feature information is extracted from the input image, and based on the movement candidate areas and the image feature information a shape of the target is extracted. Therefore, the target can be exactly and rapidly extracted and tracked.

Claims

1. A target detecting method for detecting a target from an input image from a camera, comprising the steps of:

a) obtaining camera movement parameters from a movement area in an input image from the camera, and correcting an image frame using the obtained movement parameters;

b) extracting movement candidate areas from the input image frame and a previously input image frame;

c) extracting image feature information from the input image; and

d) extracting a target shape based on the movement candidate areas and the image feature information.

2. The target detecting method of claim 1, wherein at step a), the input image frame is compensated using the camera movement parameters when the movement area is greater than a predetermined threshold value (T₂).

3. The target detecting method of claim 1, further comprising prior to step a) the additional step of determining if there is a motion pixel, which is a pixel with a brightness difference of pixels (x,y) between consecutive image frames greater than a predetermined threshold value (T₁), and then, at step a) the image frame is transformed on there being a motion pixel.

4. The target detecting method of claim 3, wherein the motion pixel is extracted as follows,

If |f_t(x,y)−

f<sub>t−

1</sub>(x,y)|>

T₁ &

|f_t+1(x,y)−

f_t(x,y)|>

T₁
then m_t+1(x,y) is moving object pixel

wherein f<sub>t−

1</sub>, f_t, and f_t+1 are (t−

1)-th, t-th, and (t+1)-th frames respectively, T₁ is a threshold value for a brightness difference between frames, f(x,y) is a brightness value of a pixel (x,y) at a frame, and m_t+1(x,y) is a motion pixel (x,y) of (t+1)-th frame.

5. The target detecting method of claim 4, wherein the threshold value (T₁) satisfies 10≦

T₁≦

20.

6. The target detecting method of claim 3, wherein the motion pixel is extracted from the entire pixels of the input image.

7. The target detecting method of claim 3, wherein at a), whether the movement area is greater than a predetermined threshold value (T₂) is determined by checking whether a difference between a maximum value x and a minimum value x among connected components of the motion pixels is greater than a predetermined threshold value (T₂) in comparison with the entire image width.

8. The target detecting method of claim 7, wherein at a), whether the movement area is greater than a predetermined threshold value (T₂) is checked as follows,

$\begin{array}{c}\max \\ x\end{array}$

[ M t c ⁡

( x , y ) ] - min x ⁡

[ M t c ⁡

( x , y ) ] >

T 2 ⁢

 

⁢

c = 1 , 2 , 3 , ⋯

⁢

 

⁢

n

wherein M_t^c is a connected component of the t-th frame motion pixels, T₂ is a threshold value for the movement area of the input image,

$\underset{x}{\max }<br><br>\left[M_t^c<br><br>\left(x,y\right)\right]$

is the maximum value x of the connected component of the t-th frame motion pixels, and

$\underset{x}{\min }<br><br>\left[M_t^c<br><br>\left(x,y\right)\right]$

is the minimum value x of the connected component of the t-th frame motion pixels.

9. The target detecting method of claim 8, wherein the threshold value (T₂) is given as 0.8*w, and where herein w is given as the traverse width of the image.

10. The target detecting method of claim 1, wherein the camera movement parameter is defined to correct the camera right/left movement and includes x/y-direction movement parameters.

11. The target detecting method of claim 10, wherein the x-direction movement parameter is extracted for a part of the image from the edge compared values between the pixels of a predetermined row i of the t-th image frame and the pixels of a predetermined row j of the (t−

1)-th image frame.

12. The target detecting method of claim 11, wherein the x-direction movement parameter (pan) is extracted as follows:

$h_t<br><br>\left(x\right)=\underset{y=0}{\overset{h}{\sum <br><br>}}<br><br>\underset{k=-p}{\overset{p}{\sum <br><br>}}<br><br>\underset{l=o}{\overset{\mathrm{dh}}{\sum <br><br>}}<br><br>\underset{j=p}{\overset{w-p}{\sum <br><br>}}<br><br>\left(E_t<br><br>\left(y+l,j\right)-E_{t-1}<br><br>\left(y+l,j+k\right)\right)$ $\mathrm{pan}=\arg <br><br>\text{\hspace{1em}<br><br>}<br><br>\min <br><br>\text{\hspace{1em}<br><br>}<br><br>h_t<br><br>\left(x\right)$

wherein y is a height of the image in the range of 0 to h, k is a possible range which the camera will move right/left, j is a range of x values, l is a y area range for which edge values are detected, E_t is the t-th edge image frame, and E(x,y) is an edge value of the pixel (x, y).

13. The target detecting method of claim 10, wherein the y-direction movement parameter is extracted for a part of the image from the edge compared values between pixels of a predetermined row i of the t-th image frame and pixels of a predetermined row j of the (t−

1)-th image frame.

14. The target detecting method of claim 13, wherein the y-direction movement parameter (tilt) is extracted as follows:

${\mathrm{\upsilon <br><br>}}_t<br><br>\left(y\right)=\underset{x=o}{\overset{w}{\sum <br><br>}}<br><br>\underset{k=-t}{\overset{t}{\sum <br><br>}}<br><br>\underset{l=o}{\overset{\mathrm{dv}}{\sum <br><br>}}<br><br>\underset{i=t}{\overset{h-t}{\sum <br><br>}}<br><br>\left(E_t<br><br>\left(i,x+l\right),E_{t-1}<br><br>\left(i+k,x+l\right)\right)$ $\mathrm{tilt}=\arg <br><br>\text{\hspace{1em}<br><br>}<br><br>\min <br><br>\text{\hspace{1em}<br><br>}<br><br>v_t<br><br>\left(y\right)$

wherein x is a width of the image in the range of 0 to w, k is a possible range which the camera will move up/down, j is a range of y values, l is an x area range for which edge values are detected, E_t is the t-th edge image frame, and E(x,y) is an edge value of the pixel (x, y).

15. The target detecting method of claim 1, wherein at a) correcting an image frame using the obtained camera movement parameters, x′

and y′

values of the t-th frame are tranformed to be x′

=x+pan and y′

=y+tilt by using x-direction movement parameter (pan), y-direction movement parameter (tilt), and x and y of the (t−

1)-th frame.

16. The target detecting method of claim 1, wherein at b), the movement candidate areas are extracted by generating an image difference between the t-th frame image and the (t−

1)-th frame image.

17. The target detecting method of claim 1, wherein the image feature information includes at least one among skin color, shape, and corner information.

18. The target detecting method of claim 1, wherein at c), skin color areas are extracted from the input image, skin color being extracted as follows:

If

[C_t^b(x,y)>

α

×

C_t^r(x,y)&

C_t^b(x,y)>

β

×

C_t^r(x,y)&

C_t^g(x,y)>

γ

×

C_t^r(x,y) &

C_t^g(x,y)>

δ

×

C_t^r(x,y)]&

[ε

<

H[x+w×

y]<

η

]&

[ω

<

S[x+w×

y]<

ζ

]

Then

f_t(x+w×

y) is skincolor

wherein C_t^r, C_t^g, and C_t^b are respectively r, g, and b values of the t-th frame images, H and S are respectively H(ue) and S(aturation) values obtained from C_t^r, C_t^g, and C_t^b, α

, β

, γ

, δ

, ε

, η

, ω

, ζ

are constants obtained by experiments, and w is a width of the image.

19. The target detecting method of claim 1, wherein at c), the shape feature information is extracted by a color clustering method of the input image.

20. The target detecting method of claim 1, wherein at c), the edge information is extracted from the input image, the edge information, that is, the edge value (E_t) of the pixel (x,y) being extracted as follows:

$E_t^v=-f_t<br><br>\left(x-1,y-1\right)+f_t<br><br>\left(x+1,y-1\right)-2⨯<br><br>f_t<br><br>\left(x-1,y\right)+2⨯<br><br>f_t<br><br>\left(x+1,y\right)-f_t<br><br>\left(x-1,y+1\right)+f_t<br><br>\left(x+1,y+1\right)$ $E_t^h=-f_t<br><br>\left(x-1,y-1\right)-2⨯<br><br>f_t<br><br>\left(x,y-1\right)-f_t<br><br>\left(x+1,y-1\right)+f_t<br><br>\left(x-1,y+1\right)+2⨯<br><br>f_t<br><br>\left(x,y+1\right)+f_t<br><br>\left(x+1,y+1\right)$ $E_t=\sqrt{{\left(E_t^v\right)}^2+{\left(E_t^h\right)}^2}$

wherein E_t is the t-th edge image frame, E_t(x,y) is the edge value of the pixel (x,y), and f_t(x,y) is the brightness value of the pixel (x,y).

21. The target detecting method of claim 1, wherein at d), skin color areas are extracted from the movement candidate areas and then the target image is extracted using the shape feature information and the edge feature information.

22. The target detecting method of claim 1, further comprising after step d):

e) extracting blob information for the extracted shape; and

f) checking whether the target is correct by comparing the blob information with that of the shape extracted at the previous frame.

23. The target detecting method of claim 22, wherein the blob information includes at least one among blob position, blob size, blob shape, and color distribution.

24. The target detecting method of claim 22, wherein the step f) further includes:

f-1) removing blobs corresponding to noise by matching the blob information for the extracted target shape information with the blob information of the shape extracted at the previous frame; and

f-2) checking whether the target is correct by extracting a color distribution of the extracted blobs.

25. The target detecting method of claim 24, wherein the step f-1) includes determining whether the blobs correspond to noise based on the size of the blob.

26. A target detecting system for detecting a target from an input image, comprising:

a camera that obtains a target image to be detected and which transmits the obtained target image to a shape detector; and

a shape detector, operatively coupled to said camera, said shape detector extracting a shape from the obtained target image based on movement candidate areas and image feature information extracted through an image difference of temporally-consecutive frames among an image sequence transmitted from the image input unit.

27. The system for detecting a target of claim 26, wherein the shape detector includes:

a movement area extractor for extracting the movement candidate area from an image frame and a previous input image frame;

an image feature information extractor for extracting image feature information from the input image; and

a shape extractor for extracting the shape of the target image based on the movement candidate area and the image feature information.

28. The system for detecting a target of claim 27, wherein the movement area extractor extracts the movement candidate area by obtaining camera movement parameters when the movement area of the input image is greater than a predetermined threshold value (T₂) and using the obtained camera movement parameters to correct the image frame.

29. The system for detecting a target of claim 27, wherein the movement area extractor checks whether the movement area of the input image is greater than a predetermined threshold value (T₂) on there being a motion pixel, which is a pixel with a brightness difference between consecutive image frames greater than a predetermined threshold value(T₁).

30. The system for detecting a target of claim 29, wherein the motion pixel is extracted as follows:

If |f_t(x,y)−

f<sub>t−

1</sub>(x,y)|>

T₁&

|f_t+1(x,y)−

f_t(x,y)|>

T₁
then m_t+1(x,y) is moving object pixel

wherein f<sub>t−

1</sub>, f_t, and f_t+1 are respectively (t−

1)-th, t-th, (t+1)-th frames, T₁ is a threshold value for brightness difference between frames, f(x,y) is a brightness value of a pixel (x,y) at a frame, and m_t+1(x,y) is a motion pixel (x,y) of the (t+1)-th frame.

31. The system for detecting a target of claim 30, wherein the threshold value (T1) satisfies 10≦

T₁≦

20.

32. The system for detecting a target of claim 29, wherein the motion pixel is extracted from the entire pixels of the input image.

33. The system for detecting a target of claim 29, wherein the movement area extractor checks, on there being motion pixels, whether a difference between a maximum value x and a minimum value x among connected components of the motion pixels is greater than a predetermined threshold value (T₂) in comparison with the entire image width when the movement area is greater than a predetermined threshold value (T₂).

34. The system for detecting a target of claim 33, wherein the movement area extractor checks whether a difference between a maximum value x and a minimum value x among connected components of the motion pixels is greater than a predetermined threshold value (T₂) in comparison with the entire image width, as follows:

$\begin{array}{c}\max \\ x\end{array}<br><br>\left[M_t^c<br><br>\left(x,y\right)\right]-\begin{array}{c}\min \\ x\end{array}<br><br>\left[M_t^c<br><br>\left(x,y\right)\right]><br><br>T_2<br><br>\text{\hspace{1em}<br><br>}<br><br>c=1,2,3,\mathrm{\cdots <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>n$

wherein M_t^c is a connected component of the t-th frame motion pixels, T₂ is a threshold value for the movement area of the input image,

$\begin{array}{c}\max \\ x\end{array}<br><br>\left[M_t^c<br><br>\left(x,y\right)\right]$

is the maximum value x of the connected component of the t-th frame motion pixels,

$\underset{x}{\min }<br><br>\left[M_t^c<br><br>\left(x,y\right)\right]$

is the minimum value x of the connected component of the t-th frame motion pixels.

35. The target detecting system of claim 34, wherein the threshold value (T₂) is given as 0.8*w, and where herein w is given as the traverse width of the image.

36. The target detecting system of claim 28, wherein the camera movement parameters are defined to correct the right/left movement of the camera and include an x-direction movement parameter and a y-direction movement parameter, the x-direction movement parameter being extracted from the edge compared values between pixels of a predetermined row i of the t-th image frame and pixels of a predetermined row j of the (t−

1)-th image frame for a part of the image, and the y-direction movement parameter being extracted from the edge compared values between pixels of a predetermined row i of the t-th image frame and pixels of a predetermined row j of the (t−

1)-th image frame for a part of the image.

37. The target detecting system of claim 36, wherein the x-direction movement parameter (pan) is extracted as follows:

$h_t<br><br>\left(x\right)=\underset{y=0}{\overset{h}{\sum <br><br>}}<br><br>\underset{k=-}{\overset{p}{\sum <br><br>}}<br><br>\underset{\mathrm{pl}=o}{\overset{\mathrm{dh}}{\sum <br><br>}}<br><br>\underset{j=p}{\overset{w-p}{\sum <br><br>}}<br><br>\left(E_t<br><br>\left(y+l,j\right)-E_{t-1}<br><br>\left(y+l,j+k\right)\right)$ $\mathrm{pan}=\arg <br><br>\text{\hspace{1em}<br><br>}<br><br>\min <br><br>\text{\hspace{1em}<br><br>}<br><br>h_t<br><br>\left(x\right)$

wherein y is a height of the image in the range of 0 to h, k is a possible range which the camera will move right/left, j is a range of x values, l is a y area range checking edge values, E_t is the t-th edge image frame, and E(x,y) is an edge value of pixel (x, y).

38. The target detecting system of claim 36, wherein

the y-direction movement parameter (tilt) is extracted as follows:

${\mathrm{\upsilon <br><br>}}_t<br><br>\left(y\right)=\underset{x=o}{\overset{w}{\sum <br><br>}}<br><br>\underset{k=-}{\overset{t}{\sum <br><br>}}<br><br>\underset{u=o}{\overset{\mathrm{dv}}{\sum <br><br>}}<br><br>\underset{i=t}{\overset{h-t}{\sum <br><br>}}<br><br>\left(E_t<br><br>\left(i,x+l\right),E_{t-1}<br><br>\left(i+k,x+l\right)\right)$$\mathrm{tilt}=\arg <br><br>\text{\hspace{1em}<br><br>}<br><br>\min <br><br>\text{\hspace{1em}<br><br>}<br><br>v_t<br><br>\left(y\right)$

wherein x is a width of the image in the range of 0 to w, k is a possible range which the camera will move up/down, j is a range of y values, l is an x area range checking edge values, E_t is the t-th edge image frame, and E(x,y) is an edge value of pixel (x, y).

39. The target detecting system of claim 38, wherein the movement area extractor corrects the t-th frame image by x′

and y′

values of the t-th frame obtained as x′

=x+pan and y′

=y+tilt using the x-direction movement parameter (pan), the y-direction movement parameter (tilt), and x and y of the (t−

1)-th frame, and extracts the movement candidate areas by generating an image difference between the t-th frame image and (t−

1)-th frame image.

40. The target detecting system of claim 26, wherein the image characteristic information includes at least one among skin color, shape, and corner information.

41. The target detecting system of claim 27, wherein the image feature information extractor extracts at least one among skin color, shape, and corner information from the input image, wherein:

the skin color is extracted as follows

If

[C_t^b(x,y)>

α

×

C_t^r(x,y)&

C_t^b(x,y)>

β

×

C_t^r(x,y)&

C_t^g(x,y)>

γ

×

C_t^r(x,y)&

C_t^g(x,y)>

δ

×

C_t^r(x,y)]&

[ε

<

H[x+w×

y]<

η

]&

[ω

<

S[x+w×

y]<

ζ

]

Then

f_t(x+w×

y) is skincolor

wherein C_t^r, C_t^g, and C_t^b are respectively r, g, and b values of the t-th frame images, H and S are respectively H(ue) and S(aturation) values obtained from C_t^r, C_t^g, and C_t^b, α

, β

, γ

, δ

, ε

, η

, ω

, ζ

are constants obtained by experiment, and w is a width of the image;

the edge value (E_t) of the pixel (x,y) is obtained as follows,

E_t^v=−

f_t(x−

1,y−

1)+f_t(x+1,y−

1)−

2×

f_t(x−

1,y)+2×

f_t(x+1,y)−

f_t(x−

1,y+1)+f_t(x+1,y+1)
E_t^h=−

f_t(x−

1,y−

1)−

2×

f_t(x,y−

1)−

f_t(x+1,y−

1)+f_t(x−

1,y+1)+2×

f_t(x,y+1)+f_t(x+1,y+1)
E_t=√

{overscore ((E_t^v)²+(E_t^h)²)}

wherein E_t is the t-th edge image frame, E_t(x,y) is the edge value of the pixel (x,y), and f_t(x,y) is the brightness value of the pixel (x,y);

and the shape feature information is extracted through color clustering of the input image.

42. The target detecting system of claim 27, wherein the shape extractor extracts the skin color areas from the movement candidate areas and then extracts the target area using the edge feature.

43. The target detecting system of claim 26, further comprising a tracker that extracts blob information for the extracted shape and verifies whether the target is correct by comparing the blob information with the shape extracted at the previous frame.

44. The target detecting system of claim 43, wherein the tracker includes a blob information extractor for extracting the blob information for the extracted shape of the target;

and

a human shape tracker for removing blobs corresponding to noise by matching the blob information for the extracted target shape information with the blob information of the shape extracted at the previous frame and checking whether the target is correct by extracting a color distribution of the extracted blobs.

45. The target detecting system of claim 43, wherein the blob information includes at least one among blob position, blob size, blob shape, and color distribution.