Image processing device and method, recording medium, and program for tracking a desired point in a moving image

• US 7,899,208 B2
• Filed: 01/06/2005
• Issued: 03/01/2011
• Est. Priority Date: 01/06/2004
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Ohtsuki, Tomoyuki, Yamaguchi, Nobuyuki, Mukai, Hitoshi, Kondo, Tetsujiro, Watanabe, Tsutomu
• Primary Examiner(s): Perungavoor; Sath V
• Assistant Examiner(s): HEIDEMANN, JASON E
• Application Number:
  US10/585,255
• Publication Number:
  US 20090175496A1
• Time in Patent Office: 2,245 Days
• Field of Search: 348169-172, 348152-155, 348/143, 348/403.1, 348/404.1, 348/407.1, 348/409.1, 348/410.1, 348/411.1, 348/412.1, 348/413.1, 348/414.1, 348/415.1, 348/416.1, 348/417.1, 348/418.1, 348/420.1, 348/421.1, 348/422.1, 382/100, 382/103, 382/107, 382/164, 382/165, 382/168, 382170-172, 382/173, 382/181, 382/199, 382/203, 382/215, 382/254, 382/266
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

The present invention relates to image processing apparatus and method, a recording medium, and a program for providing reliable tracking of a tracking point. When a right eye 502 of a human face 504 serving as a tracking point is tracked in a frame n−1 and when a tracking point 501 appears as in a frame n, the tracking point 501 continues to be tracked. When, as in a frame n+1, the right eye 502 serving as the tracking point 501 disappears due to the rotation of the face 504 of an object to be tracked, the tracking point is transferred to a left eye 503, which is a different point of the face 504 serving as the object including the right eye 502. The present invention is applicable to a security camera system.

Claims

1. An image processing apparatus comprising:

position estimating means for estimating the position of a second point representing a tracking point in an image of a temporally next unit of processing, the second point corresponding to a first point representing the tracking point in an image of a temporally previous unit of processing;

generating means for generating estimated points serving as candidates of the first point when the position of the second point is inestimable;

determining means for determining the second point in the next unit of processing on the basis of the estimation result of the position estimating means when the position of the second point in the next unit of processing is estimable; and

selecting means for selecting the first point from among the estimated points when the position of the second point is inestimable, wherein

the determining means includes;

evaluation value computing means for computing an evaluation value representing a correlation between pixels of interest representing at least one pixel including the first point in the temporally previous unit of processing and the corresponding pixels representing at least one pixel in the temporally next unit of processing and defined on the basis of a motion vector of the pixels of interest;

variable value computing means for computing a variable value representing the variation of a pixel value with respect to the pixels of interest; and

accuracy computing means for computing the accuracy of the motion vector, and

the variable value computing means computes the variable value representing the sum of values obtained by dividing the sum of absolute differences between the pixels of interest and the adjacent pixels that are adjacent to the pixels of interest in a block including the pixels of interest by the number of the adjacent pixels.

2. The image processing apparatus according to claim 1, wherein the unit of processing is a frame.

3. The image processing apparatus according to claim 1, wherein the position estimating means further computes the accuracy of the estimation of the position and wherein, if the computed accuracy is greater than a reference value, the position estimating means determines that the position of the second point is estimable.

4. The image processing apparatus according to claim 1, wherein, if the position of the second point in the next unit of processing is inestimable, the position estimating means estimates the position of the second point on the basis of the first point selected by the selecting means.

5. The image processing apparatus according to claim 1, wherein, if the position of the second point is estimable, the position estimating means considers the position of the second point to be a new first point and estimates the position of the tracking point in the image of the next unit of processing.

6. The image processing apparatus according to claim 1, wherein the generating means includes region estimating means for estimating a set of at least one point, the set belonging to an object including the first point, to be a target region in the previous unit of processing or in a more previous unit of processing than the previous unit of processing and estimated point generating means for generating the estimated points on the basis of the target region.

7. The image processing apparatus according to claim 6, wherein the region estimating means finds a position that overlaps at least the target region serving as an object to be estimated by prediction, determines a region estimation range at the predicted point including the tracking point in the unit of processing for estimating the target region, sets sample points in the determined region estimation range, and estimates a region consisting of a set of the sample points having the same motion and having the largest dimensions among the sample points to be the target region.

8. The image processing apparatus according to claim 7, wherein the shape of the region estimation range is fixed.

9. The image processing apparatus according to claim 7, wherein the shape of the region estimation range is variable.

10. The image processing apparatus according to claim 7, wherein the region estimating means estimates the target region in a more previous unit of processing than the previous unit of processing and wherein the generating means generates a point in the estimated target region in the more previous unit of processing than the previous unit of processing as the estimated point.

11. The image processing apparatus according to claim 7, wherein the region estimating means estimates the target region in the previous unit of processing and wherein the generating means generates a point forming the target region as the estimated point.

12. The image processing apparatus according to claim 6, wherein the region estimating means estimates points that are adjacent to the first point and that have pixel values similar to the pixel value of the first point and points that are adjacent to the points adjacent to the first point to be the target region.

13. The image processing apparatus according to claim 6, wherein the region estimating means extracts sample points in a region having a predetermined size and including the first point in a more previous unit of processing than the previous unit of processing and wherein the region estimating means estimates a region including the points in the previous unit of processing obtained by shifting a region of the sample points having the same motion and having the largest dimensions by an amount of the same motion to be the target region.

14. The image processing apparatus according to claim 6, further comprising:

template generating means for generating a template; and

correlation computing means for computing a correlation between a block representing a predetermined region in the next unit of processing and a block representing a predetermined region of the template in a unit of processing more previous than the unit of processing of the block by one or more units of processing when the second point is not determined on the basis of the estimated points;

wherein the tracking point is detected by using at least the determining means when the correlation is determined to be high on the basis of the correlation computed by the correlation computing means.

15. The image processing apparatus according to claim 14, wherein the template generating means determines a predetermined region around the tracking point to be the template.

16. The image processing apparatus according to claim 14, wherein the template generating means generates the template on the basis of the target region.

17. The image processing apparatus according to claim 14, wherein, when the correlation is determined to be high on the basis of the correlation computed by the correlation computing means, the second point is determined on the basis of a relationship between the block representing the predetermined region of the template in a unit of processing more previous than a block representing the predetermined region in the next unit of processing by one or more units of processing and the tracking point and on the basis of the position of the block having the correlation determined to be high.

18. The image processing apparatus according to claim 14, wherein the template generating means determines a region formed from a sample point in the target region and a predetermined area around the sample point to be the template.

19. The image processing apparatus according to claim 14, wherein the correlation computing means determines the correlation by computing an error between the block in the next unit of processing and a block of the template in a unit of processing more previous than the unit of processing of the block by one or more units of processing.

20. The image processing apparatus according to claim 1, further comprising:

detecting means for detecting a scene change;

wherein the position estimating means and the selecting means terminate the processes thereof on the basis of a predetermined condition and change the condition on the basis of the presence of the scene change when the position estimating means and the selecting means are unable to select the second point from among the estimated points.

21. The image processing apparatus according to claim 1, wherein the number of the pixels of interest is equal to the number of the corresponding pixels.

22. The image processing apparatus according to claim 1, wherein the variable value indicates the variation of a pixel value in the spatial direction.

23. The image processing apparatus according to claim 1, wherein the variable value indicates one of a degree of dispersion and a dynamic range.

24. The image processing apparatus according to claim 1, wherein the unit of processing is one of a frame and a field.

25. The image processing apparatus according to claim 1, wherein the accuracy computing means computes the accuracy of the motion vector on the basis of a value normalized from the evaluation value with respect to the variable value.

26. The image processing apparatus according to claim 1, wherein the accuracy computing means determines a value normalized from the evaluation value with respect to the variable value to be the accuracy of the motion vector when the variable value is greater than a predetermined threshold value and wherein the accuracy computing means determines a fixed value indicating that the accuracy of the motion vector is low when the variable value is less than the predetermined threshold value.

27. The image processing apparatus according to claim 1, wherein the evaluation value computing means computes the evaluation value representing the sum of absolute differences between pixels in a block including the pixels of interest and pixels in a block including the corresponding pixels.

28. The image processing apparatus according to claim 1, wherein the accuracy computing means includes:

comparing means for comparing the variable value with a first reference value;

difference computing means for computing the difference between a second reference value and the value normalized from the evaluation value with respect to the variable value; and

outputting means for computing the accuracy of the motion vector on the basis of the comparison result of the comparing means and the difference computed by the difference computing means and outputting the accuracy of the motion vector.

29. The image processing apparatus according to claim 1, further comprising:

motion vector detecting means for detecting the motion vector from an input image and delivering the motion vector to the evaluation value computing means;

motion compensating means for motion-compensating the input image on the basis of the motion vector detected by the motion vector detecting means;

selecting means for selecting one of the image that is motion-compensated by the motion compensating means and the image that is not motion-compensated on the basis of the accuracy of the motion vector; and

encoding means for encoding the image selected by the selecting means.

30. The image processing apparatus according to claim 1, further comprising:

frequency distribution computing means for computing a frequency distribution weighted with the accuracy of the motion vector; and

maximum value detecting means for detecting a maximum value of the frequency distribution computed by the frequency distribution computing means and detecting a background motion on the basis of the detected maximum value.

31. The image processing apparatus according to claim 1, further comprising:

average value computing means for computing the average of the accuracy of the motion vectors in the unit of processing; and

determining means for comparing the average computed by the average value computing means with a reference value and determining the presence of a scene change on the basis of the comparison result.

32. The image processing apparatus according to claim 31, wherein the average value computing means computes one average for one unit of processing.

33. The image processing apparatus according to claim 1, further comprising:

first-point detecting means for detecting the first point of a moving object in an image;

correction area setting means for setting a correction area having a predetermined size around the object in the image on the basis of the estimation result;

correcting means for correcting the image in the correction area in the image; and

display control means for controlling the display of the image including the image in the correction area corrected by the correcting means.

34. The image processing apparatus according to claim 33, wherein the correcting means corrects blurring of the image.

35. The image processing apparatus according to claim 34, wherein the correcting means includes:

delivering means for delivering a control signal for identifying an image in the correction area and a parameter indicating the level of blurring of the image;

feature detecting means for detecting the feature of the image in the correction area identified on the basis of the control signal and outputting a feature code representing the detected feature;

storage means for storing the parameter representing the level of blurring of the image and a coefficient corresponding to the feature code output from the feature detecting means;

readout means for reading out the parameter and the coefficient corresponding to the feature code output from the feature detecting means from the storage means;

inner-product computing means for computing the inner product of the values of pixels in the input image on the basis of the coefficient read out by the readout means; and

selectively-outputting means for selecting one of the computation result from the inner-product computing means and the value of the pixel of the input image and outputting the selected one;

wherein the image in the correction area is corrected so that blurring of the image is removed.

36. The image processing apparatus according to claim 35, wherein the first-point detecting means includes:

first extracting means for extracting a plurality of pixels around the pixel to be subjected to the inner product computation in a predetermined first area from the input image;

second extracting means for extracting a plurality of pixels in each of a plurality of second areas contiguous to the first area in a plurality of vertical and horizontal directions;

block difference computing means for computing a plurality of block differences by computing the sum of absolute differences between the values of the pixels extracted by the first extracting means and the values of the corresponding pixels extracted by the second extracting means; and

difference determining means for determining whether the block difference is greater than a predetermined threshold value.

37. The image processing apparatus according to claim 35, wherein the parameter is a parameter of the Gaussian function in a model expression representing a relationship between a pixel of a blurred image and a pixel of an unblurred image.

38. The image processing apparatus according to claim 37, wherein the coefficient stored by the storage means is a coefficient obtained by computing the inverse matrix of the model expression.

39. The image processing apparatus according to claim 35, wherein the selectively-outputting means includes:

first extracting means for extracting a plurality of pixels subjected to the inner product computation by the inner-product computing means;

dispersion computing means for computing the degree of dispersion representing the level of dispersion of the plurality of pixels extracted by the first extracting means; and

dispersion determining means for determining whether the degree of dispersion computed by the dispersion computing means is greater than a predetermined threshold value.

40. The image processing apparatus according to claim 39, wherein the selectively-outputting means further includes pixel selecting means for selecting one of the computation result of the inner-product computing means and the value of the pixel of the input image as an output value of the pixel on the basis of the determination result of the dispersion determining means.

41. An image processing method comprising:

an estimating step for estimating the position of a second point representing a tracking point in an image of a temporally next unit of processing, the second point corresponding to a first point representing the tracking point in an image of a temporally previous unit of processing;

a generating step for generating estimated points serving as candidates of the first point when the position of the second point is inestimable;

a determining step for determining the second point in the next unit of processing on the basis of the estimation result of the position estimating step when the position of the second point in the next unit of processing is estimable; and

a selecting step for selecting the first point from among the estimated points when the position of the second point is inestimable, wherein

the determining step includes

an evaluation value computing step for computing an evaluation value representing a correlation between pixels of interest representing at least one pixel including the first point in the temporally previous unit of processing and the corresponding pixels representing at least one pixel in the temporally next unit of processing and defined on the basis of a motion vector of the pixel of interest;

a variable value computing step for computing a variable value representing the variation of a pixel value with respect to the pixel of interest; and

an accuracy computing step for computing the accuracy of the motion vector, and

the variable value computing step includes computing the variable value representing the sum of values obtained by dividing the sum of absolute differences between the pixels of interest and the adjacent pixels that are adjacent to the pixels of interest in a block including the pixels of interest by the number of the adjacent pixels.

42. The image processing method according to claim 41, further comprising:

first-point detecting step for detecting the first point of a moving object in an image;

a correction area setting step for setting a correction area having a predetermined size around the object in the image on the basis of the estimation result;

a correcting step for correcting an image in the correction area in the image; and

a display control step for controlling the display of the image including the image in the correction area corrected by the correcting step.

43. A non-transitory recording medium storing a computer-readable program, the computer-readable program comprising:

an estimating step for estimating the position of a second point representing a tracking point in an image of a temporally next unit of processing, the second point corresponding to a first point representing the tracking point in an image of a temporally previous unit of processing;

a generating step for generating estimated points serving as candidates of the first point when the position of the second point is inestimable;

a determining step for determining the second point in the next unit of processing on the basis of the estimation result of the position estimating step when the position of the second point in the next unit of processing is estimable; and

a selecting step for selecting the first point from among the estimated points when the position of the second point is inestimable, wherein

the determining step includes

an evaluation value computing step for computing an evaluation value representing a correlation between pixels of interest representing at least one pixel including the first point in the temporally previous unit of processing and the corresponding pixels representing at least one pixel in the temporally next unit of processing and defined on the basis of a motion vector of the pixel of interest;

a variable value computing step for computing a variable value representing the variation of a pixel value with respect to the pixel of interest; and

an accuracy computing step for computing the accuracy of the motion vector, and

the variable value computing step includes computing the variable value representing the sum of values obtained by dividing the sum of absolute differences between the pixels of interest and the adjacent pixels that are adjacent to the pixels of interest in a block including the pixels of interest by the number of the adjacent pixels.