Illumination-invariant object tracking method and image editing system using the same

• US 7,171,023 B2
• Filed: 11/05/2002
• Issued: 01/30/2007
• Est. Priority Date: 11/05/2001
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Jong-ha, Cho, In-myung, Kim, Tae-kyun
• Primary Examiner(s): Johns; Andrew W.
• Application Number:
  US10/287,592
• Publication Number:
  US 20030099376A1
• Time in Patent Office: 1,547 Days
• Field of Search: 382/103, 382/162, 382/164, 382/168, 382/171, 382/173
• US Class Current: 382/103
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

An illumination-invariant object tracking method and an image editing system using the same are provided. The illumination-invariant object tracking method involves: designating an initial position of an object of interest to be tracked in an image; constructing a probability map for object tracking in consideration of a color ratio of adjacent pixels in the image; and performing Kalman filtering based on the initial position of the object and the probability map. The image editing system based on the illumination-invariant object tracking method includes: a data input unit which receives an image to be edited; an object tracking unit which tracks a target object in consideration of a color ratio of adjacent pixels in the image; an image editing unit which edits and outputs an edited image in a predetermined region of the image, in response to information on the position, size, and boundary of the tracked target object.

Claims

1. An illumination-invariant object tracking method comprising:

(a) designating an initial position of an object of interest to be tracked in an image;

(b) modeling color information on the image in consideration of a color ratio of adjacent pixels in the image;

(c) constructing a probability map based on the result of the color modeling; and

(d) tracking a current position of the target object in response to the initial position and the probability map.

2. The illumination-invariant object tracking method of claim 1,wherein (a) designating the initial position of the object of interest comprises:

(a-1) determining whether to manually or automatically designate the initial position of the object;

(a-2) if it is determined in (a-1) to manually designate the initial position of the object, a user directly inputting the initial position of the object; and

(a-3) if it is determined in (a-1) to automatically designate the initial position of the object, reading a color histogram of the object from a data storing unit where color histrograms for a plurality of objects have been stored and automatically designating the initial position of the object by back-projecting the read color histogram.

3. The illumination-invariant object tracking method of claim 2, wherein denoting a disk having a radius r as Dr, (a-3) automatically designating the initial position of the object by back-projecting the read color histogram comprises:

(a-3-1) outputting a smaller value between the quotient of dividing the color histogram of a previous model for each histogram bin by the color histogram of a current image for each histogram bin and 1;

(a-3-2) performing the operation of (a-3-1) on all pixels in the image;

(a-3-3) convolving the result of the operation of (a-3-2) with Dr; and

(a-3-4) tracking a coordinate having the largest convolution result and designating the coordinate as the initial position of the object;

D^r(x,y)=1 if √

{square root over (x²+y²)}0 otherwise

wherein

where (x, y) denotes a coordinate in the image.

4. The illumination-invariant image tracking method of claim 1, wherein (b) constructing the probability map comprises:

(b-1) modeling color information on the object in consideration of the color ratio of adjacent pixels; and

(b-2) modeling color information on the entire image in consideration of the color ratio of adjacent pixels.

5. The illumination-invariant image tracking method of claim 4,wherein (b-1) modeling the color information on the object comprises:

(b-1-1) analyzing color components of the object;

(b-1-2) if the object is determined to be polychromatic as the result of the analysis (b-1-1), constructing a 3-dimensional histogram of the object with an axis of the brightness ratio for each of R, G, and B channels;

(b-1-3) if the object is determined to be monochromatic as the result of the analysis (b-1-1), defining a color probability for the object by 2-dimensional Gaussian modeling for the hue and saturation of the object; and

(b-1-4) constructing a color histogram of the object with an axis of the color probability defined in (b-1-3).

6. The illumination-invariant image tracking method of claim 5, wherein in (b-1-1), a value of

$\frac{1}{S}$

∑

i , j ⁢

{ ( R ⁢

( i , j ) - u R ) 2 + ( G ⁢

( i , j ) - u G ) 2 + ( B ⁢

( i , j ) - u B ) 2 } ,

where S indicates the number of (i,j) pairs, U_R indicates an average value of red (R) image data of the (i,j) pairs, U_G indicates an average value of green (G) image data of the (i,j) pairs, and U_B indicates an average value of blue (B) image data of the (i,j) pairs, is calculated, the object is determined to be monochromic if the result of the calculation is smaller than a predetermined threshold, and the object is determined to be polychromatic if the result of the calculation is greater than or equal to the predetermined threshold.

7. The illumination-invariant image tracking method of claim 5, wherein in (b-1-3), the color probability is defined using the following equation:

$P_{\mathrm{skin}}<br><br>\left(x,y\right)=\frac{\underset{i=x-n/2}{\overset{i=x+n/2}{\sum <br><br>}}<br><br><br><br>\underset{j=x-n/2}{\overset{j=x+n/2}{\sum <br><br>}}<br><br><br><br>g<br><br>\left(\mathrm{Hue}<br><br>\left(i,j\right),\mathrm{Sat}<br><br>\left(i,k\right);<br><br>\stackrel{\to <br><br>}{u},\mathrm{\Sigma <br><br>}\right)}{n^2}$

where

$\stackrel{\_}{u}=\left[\begin{array}{c}175\\ -\frac{0.6}{150}<br><br>I+1.0\end{array}\right],\mathrm{\Sigma <br><br>}=\left[\begin{array}{cc}{30}^2& 0\\ 0& {\mathrm{\sigma <br><br>}}_{\mathrm{sat}}^2\end{array}\right],$

g(i,j;

{right arrow over (u)},Σ

)denotes a 2-dimensional Gaussian function, n denotes a size of the object, and I denotes an average brightness intensity of the image.

8. The illumination-invariant object tracking method of claim 4, wherein (b-2) modelling the color information on the entire image comprises:

(b-2-1) measuring color variations of the entire image;

(b-2-2) if the object is determined to be illuminated by white light as the result of (b-2-1), constructing a color distribution histogram of the entire image irrespective of the result of the object color information modelling in (b-1); and

(b-2-3) if the object is determined to be illuminated by colored light as the result of (b-2-1), re-performing (b-1) on an object tracked in a previous image frame and constructing a color distribution histogram of the entire image.

9. The illumination-invariant object tracking method of claim 8, wherein in (b-2-1), a value of

$\frac{1}{S}$

∑

i , j ⁢

{ ( R n R n + G n + B n - R m R m + G m + B m ) 2 ⁢

( G n R n + G n + B n - G m R m + G m + B m ) 2 } ,

where S indicates the number of (i,j) pairs, n indicates a current frame image, m indicates a previous frame image, and R, G, and B indicate red, green, and blue data of the image, respectively, is calculated, the object is determined to be illuminated by the white light if the result of the calculation is smaller than a predetermined threshold, and the object is determined to be illuminated by the colored light if the result of the calculation is greater than or equal to the predetermined threshold.

10. The illumination-invariant object tracking method of claim 4, wherein the probability map is expressed as

$P(\mathrm{object}$

⁢

color ) = h ratio ⁡

( color ) = h object ⁡

( color ) h total ⁡

( color )

where h_object (color) indicates the result of the color information modelling of the object and h_total (color) indicates the result of the color information modelling of the entire image.

11. The illumination-invariant image tracking method of claim 1, wherein the step (d) comprises:

(d-1) storing a previous position of the object;

(d-2) performing Kalman filtering based on the probability map and the initial position of the object for tracking the current position and size of the object of interest in a series of pictures; and

(d-3) updating a covariance matrix of a Kalman filter in response to the previous position and the current position of the object.

12. A computer readable physical media having embodied thereon a computer program for the method of claim 1.

13. An object tracking system comprising:

an initial position designating portion which designates an initial position of a target object to be tracked in an image;

a color modelling portion which performs a color modelling on the target object and the entire image in consideration of a color ratio of adjacent pixels in the image;

a probability map constructing portion which constructs a probability map for object tracking based on the result of the color modelling; and

an object tracking portion which tracks a current position of the target object in response to the initial position and the probability map.

14. The object tracking system of claim 13, wherein the initial position designating portion comprises:

a first initial position designator which receives a predetermined position designated by a user as the initial position of the target object;

an object database which stores a list of objects and color modelling data for the objects; and

a second initial position designator which reads the color modelling data for the target object designated by the user from the object database and automatically designates the initial position of the target object by back-projecting the read color modelling data.

15. The object tracking system of claim 13, wherein the color modelling portion comprises:

a first color modeler which models color information on the target object;

a second color modeler which models color information on the entire image; and

a third color modeler which adaptively models the color information on the target object based on illumination variations of the image.

16. The object tracking system of claim 15, wherein the first color modeler analyzes color components of the target object, constructs a 3-dimensional histogram of the target object with an axis of the brightness ratio for each of R, 0,and B channels if the target object is determined to be polychromatic as the result of the analysis, and defines a color probability for the target object by 2-dimensional Gaussian modeling for the hue and saturation of the target object and constructs a color histogram of the target object with an axis of the color probability if the target object is determined to be monochromatic as the result of the analysis.

17. The object tracking system of claim 16, wherein the first color modeler calculates a value of

$\frac{1}{S}$

∑

i , j ⁢

{ ( R ⁢

( i , j ) - u R ) 2 + ( G ⁢

( i , j ) - u G ) 2 + ( B ⁢

( i , j ) - u B ) 2 } ,

where S indicates the number of (i,j) pairs, U_R indicates an average value of red (R) image data of the (i,j) pairs, U_G indicates an average value of green (G) image data of the (i,j) pairs, and U_B indicates an average value of blue (B) image data of the (i,j) pairs, determines the target object to be monochromic if the result of the calculation is smaller than a predetermined threshold, and determines the target object to be polychromatic if the result of the calculation is greater than or equal to the predetermined threshold.

18. The object tracking system of claim 16, wherein the first color modeler defines the color probability of the target object using the following equation:

$P_{\mathrm{skin}}<br><br>\left(x,y\right)=\frac{\underset{i=x-n/2}{\overset{i=x+n/2}{\sum <br><br>}}<br><br><br><br>\underset{j=x-n/2}{\overset{j=x+n/2}{\sum <br><br>}}<br><br><br><br>g<br><br>\left(\mathrm{Hue}<br><br>\left(i,j\right),\mathrm{Sat}<br><br>\left(i,k\right);<br><br>\stackrel{\to <br><br>}{u},\mathrm{\Sigma <br><br>}\right)}{n^2}$

where

$\stackrel{\_}{u}=\left[\begin{array}{c}175\\ -\frac{0.6}{150}<br><br>I+1.0\end{array}\right],\mathrm{\Sigma <br><br>}=\left[\begin{array}{cc}{30}^2& 0\\ 0& {\mathrm{\sigma <br><br>}}_{\mathrm{sat}}^2\end{array}\right],$

g(i,j;

{right arrow over (u)},Σ

)denotes a 2-dimensional Gaussian function, n denotes a size of the object, and I denotes an average brightness intensity of the image.

19. The object tracking system of claim 15, wherein the third color modeler measures color variations of the entire image and re-performs the color modelling on an object tracked in a previous image and constructs a color distribution histogram of the entire image if the target object is determined to be illuminated by colored light as the result of the color variation measurement.

20. The object tracking system of claim 19, wherein the third color modeler calculates a value of

$\frac{1}{S}$

∑

i , j ⁢

{ ( R n R n + G n + B n - R m R m + G m + B m ) 2 ⁢

( G n R n + G n + B n - G m R m + G m + B m ) 2 } ,

where S indicates the number of (i,j) pairs, n indicates a current frame image, m indicates a previous frame image, and R, G, and B indicate red, green, and blue data of the image, respectively, determines the target object to be illuminated by white light if the result of the calculation is smaller than a predetermined threshold, and determines the target object to be illuminated by the colored light if the result of the calculation is greater than or equal to the predetermined threshold.

21. The object tracking system of claim 13, wherein the probability map constructing portion calculates the probability map as the following equation

$P(\mathrm{object}$

⁢

color ) = h ratio ⁡

( color ) = h object ⁡

( color ) h total ⁡

( color )

where h_object (color) indicates the result of the color information modelling of the target object and h_total (color) indicates the result of the color information modelling of the entire image.

22. The object tracking system of claim 13, wherein the object tracking portion comprises:

a memory for storing a previous position of the object;

a Kalman filter for tracking the current position and size of the object of interest in a series of pictures, based on the probability map and the Initial position of the object; and

a velocity predictor for updating a covariance matrix of the Kalman filter in response to the previous position and the current position of the object.