Image processing apparatus and method, and image pickup apparatus

• US 20040105493A1
• Filed: 10/20/2003
• Published: 06/03/2004
• Est. Priority Date: 06/27/2001
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Sawao, Takashi, Ishibashi, Junichi, Kondo, Tetsujiro, Nagano, Takahiro, Wada, Seiji, Miyake, Toru, Fujiwara, Naoki
• Granted Patent: US 7,477,289 B2
• Time in Patent Office: Days
• US Class Current: 375/240.08
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

An object of the present invention is to make it possible to detect the mixture state of an image. An area specifying unit 103 specifies, in correspondence with image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed and a non-mixed area formed of a foreground area consisting of only the foreground object components and a background area consisting of only the background object components. A mixture-ratio calculator 104 estimates, based on the ratio of the width of the mixed area in a predetermined direction with respect to the moving direction of the foreground object, the width of the mixed area being specified by the area information, to the distance from the position of a designated pixel to the edge of the mixed area in the predetermined direction, the mixture ratio indicating the ratio of the foreground object components and the background object components of the designated pixel. The present invention is applicable to an image processing apparatus.

Claims

1. An image processing apparatus for processing image data which is formed of a predetermined number of pixel data obtained by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, said image processing apparatus comprising:

area specifying means for specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed and a non-mixed area formed of a foreground area consisting of only the foreground object components and a background area consisting of only the background object components; and

mixture-ratio estimating means for estimating, based on a ratio of a width of the mixed area in a predetermined direction with respect to a moving direction of the foreground object component, the width of the mixed area being specified by the area information, to a distance from a position of a designated pixel to an edge of the mixed area in the predetermined direction, a mixture ratio indicating the ratio between the foreground object components and the background object components of the designated pixel.

2. An image processing apparatus according to claim 1, further comprising motion-vector detection means for detecting a motion vector of the foreground object,

wherein said mixture-ratio estimating means detects the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area in the moving direction of the motion vector, and estimates the mixture ratio based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

3. An image processing apparatus according to claim 1, further comprising moving-direction detection means for detecting the moving direction of the foreground object,

wherein said mixture-ratio estimating means detects the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area in the moving direction, and estimates the mixture ratio based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

4. An image processing apparatus according to claim 3, wherein said moving-direction detection means comprises motion-vector generating means for generating a motion vector based on the area information and the position of the mixed area in a designated frame and the position of the mixed area in a frame in the vicinity of the designated frame, thereby detecting the moving direction based on the generated motion vector.

5. An image processing method for an image processing apparatus for processing image data which is formed of a predetermined number of pixel data obtained by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, said image processing method comprising:

an area specifying step of specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed and a non-mixed area formed of a foreground area consisting of only the foreground object components and a background area consisting of only the background object components; and

a mixture-ratio estimating step of estimating, based on a ratio of a width of the mixed area in a predetermined direction with respect to a moving direction of the foreground object component, the width of the mixed area being specified by the area information, to a distance from a position of a designated pixel to an edge of the mixed area in the predetermined direction, a mixture ratio indicating the ratio between the foreground object components and the background object components of the designated pixel.

6. An image processing method according to claim 5, further comprising a motion-vector detection step of detecting a motion vector of the foreground object,

wherein, in said mixture-ratio estimating step, the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area are detected in the moving direction of the motion vector, and the mixture ratio is estimated based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

7. An image processing method according to claim 5, further comprising a moving-direction detection step of detecting the moving direction of the foreground object,

wherein, in said mixture-ratio estimating step, the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area are detected in the moving direction, and the mixture ratio is estimated based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

8. An image processing method according to claim 7, wherein said moving-direction detection step comprises a motion-vector generating step of generating a motion vector based on the area information and the position of the mixed area in a designated frame and the position of the mixed area in a frame in the vicinity of the designated frame, thereby detecting the moving direction based on the generated motion vector.

9. A recording medium in which a computer-readable program for processing image data which is formed of a predetermined number of pixel data obtained by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, is recorded, said program comprising:

an area specifying step of specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed and a non-mixed area formed of a foreground area consisting of only the foreground object components and a background area consisting of only the background object components; and

a mixture-ratio estimating step of estimating, based on a ratio of a width of the mixed area in a predetermined direction with respect to a moving direction of the foreground object component, the width of the mixed area being specified by the area information, to a distance from a position of a designated pixel to an edge of the mixed area in the predetermined direction, a mixture ratio indicating the ratio between the foreground object components and the background object components of the designated pixel.

10. A recording medium according to claim 9, wherein:

said program further comprises a motion-vector detection step of detecting a motion vector of the foreground object; and

in said mixture-ratio estimating step, the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area are detected in the moving direction of the motion vector, and the mixture ratio is estimated based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

11. A recording medium according to claim 9, wherein:

said program further comprises a moving-direction detection step of detecting the moving direction of the foreground object; and

in said mixture-ratio estimating step, the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area are detected in the moving direction, and the mixture ratio is estimated based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

12. A recording medium according to claim 11, wherein said moving-direction detection step comprises a motion-vector generating step of generating a motion vector based on the area information and the position of the mixed area in a designated frame and the position of the mixed area in a frame in the vicinity of the designated frame, thereby detecting the moving direction based on the generated motion vector.

13. A program for allowing a computer that processes image data which is formed of a predetermined number of pixel data obtained by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, to execute:

an area specifying step of specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed and a non-mixed area formed of a foreground area consisting of only the foreground object components and a background area consisting of only the background object components; and

a mixture-ratio estimating step of estimating, based on a ratio of a width of the mixed area in a predetermined direction with respect to a moving direction of the foreground object component, the width of the mixed area being specified by the area information, to a distance from a position of a designated pixel to an edge of the mixed area in the predetermined direction, a mixture ratio indicating the ratio between the foreground object components and the background object components of the designated pixel.

14. A program according to claim 13, further comprising a motion-vector detection step of detecting a motion vector of the foreground object,

wherein, in said mixture-ratio estimating step, the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area are detected in the moving direction of the motion vector, and the mixture ratio is estimated based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

15. A program according to claim 13, further comprising a moving-direction detection step of detecting the moving direction of the foreground object,

wherein, in said mixture-ratio estimating step, the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area are detected in the moving direction, and the mixture ratio is estimated based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

16. A program according to claim 15, wherein said moving-direction detection step comprises a motion-vector generating step of generating a motion vector based on the area information and the position of the mixed area in a designated frame and the position of the mixed area in a frame in the vicinity of the designated frame, thereby detecting the moving direction based on the generated motion vector.

17. An image-capturing apparatus comprising:

image-capturing means for outputting a subject image captured by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, as image data formed of a predetermined number of pixel data;

area specifying means for specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed and a non-mixed area formed of a foreground area consisting of only the foreground object components and a background area consisting of only the background object components; and

mixture-ratio estimating means for estimating, based on a ratio of a width of the mixed area in a predetermined direction with respect to a moving direction of the foreground object component, the width of the mixed area being specified by the area information, to a distance from a position of a designated pixel to an edge of the mixed area in the predetermined direction, a mixture ratio indicating the ratio between the foreground object components and the background object components of the designated pixel.

18. An image-capturing apparatus according to claim 17, further comprising motion-vector detection means for detecting a motion vector of the foreground object,

wherein said mixture-ratio estimating means detects the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area in the moving direction of the motion vector, and estimates the mixture ratio based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

19. An image-capturing apparatus according to claim 17, further comprising moving-direction detection means for detecting the moving direction of the foreground object,

wherein said mixture-ratio estimating means detects the width of the mixed area and the distance from the position of the designated pixel to the edge of the mixed area in the moving direction, and estimates the mixture ratio based on the ratio of the width of the mixed area to the distance from the position of the designated pixel to the edge of the mixed area.

20. An image-capturing apparatus according to claim 19, wherein said moving-direction detection means comprises motion-vector generating means for generating a motion vector based on the area information and the position of the mixed area in a designated frame and the position of the mixed area in a frame in the vicinity of the designated frame, thereby detecting the moving direction based on the generated motion vector.

21. An image processing apparatus for processing image data which is formed of a predetermined number of pixel data obtained by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, said image processing apparatus comprising:

area specifying means for outputting area information used for specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed; and

mixture-ratio estimating means for estimating a mixture ratio indicating the ratio of the foreground object components and the background object components in the mixed area based on the area information and a moving direction of the foreground object by assuming that the mixture ratio changes substantially linearly with respect to a position of the pixel.

22. An image processing apparatus according to claim 21, wherein said mixture-ratio estimating means estimates, based on a position on a segment whose one end is positioned at a foreground area consisting of only the foreground object components and the other end is positioned at a background area consisting of only the background object components, the segment traversing the mixed area, the mixture ratio of the pixel corresponding to the position on the segment.

23. An image processing apparatus according to claim 22, wherein said mixture-ratio estimating means estimates the mixture ratio of the pixel corresponding to the position on the segment based on the position on the segment parallel to the moving direction of the foreground object.

24. An image processing apparatus according to claim 22, wherein said mixture-ratio estimating means estimates the mixture ratio of the pixel corresponding to the position on the segment based on the position on the segment obliquely intersecting with the moving direction of the foreground object.

25. An image processing method for processing image data which is formed of a predetermined number of pixel data obtained by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, said image processing method comprising:

an area specifying step of outputting area information used for specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed; and

a mixture-ratio estimating step of estimating a mixture ratio indicating the ratio of the foreground object components and the background object components in the mixed area based on the area information and a moving direction of the foreground object by assuming that the mixture ratio changes substantially linearly with respect to a position of the pixel.

26. An image processing method according to claim 25, wherein, in said mixture-ratio estimating step, based on a position on a segment whose one end is positioned at a foreground area consisting of only the foreground object components and the other end is positioned at a background area consisting of only the background object components, the segment traversing the mixed area, the mixture ratio of the pixel corresponding to the position on the segment is estimated.

27. An image processing method according to claim 26, wherein, in said mixture-ratio estimating step, the mixture ratio of the pixel corresponding to the position on the segment is estimated based on the position on the segment parallel to the moving direction of the foreground object.

28. An image processing method according to claim 26, wherein, in said mixture-ratio estimating step, the mixture ratio of the pixel corresponding to the position on the segment is estimated based on the position on the segment obliquely intersecting with the moving direction of the foreground object.

29. A recording medium in which a computer-readable program for processing image data which is formed of a predetermined number of pixel data obtained by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, is recorded, said program comprising:

an area specifying step of outputting area information used for specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed; and

a mixture-ratio estimating step of estimating a mixture ratio indicating the ratio of the foreground object components and the background object components in the mixed area based on the area information and a moving direction of the foreground object by assuming that the mixture ratio changes substantially linearly with respect to a position of the pixel.

30. A recording medium according to claim 29, wherein, in said mixture-ratio estimating step, based on a position on a segment whose one end is positioned at a foreground area consisting of only the foreground object components and the other end is positioned at a background area consisting of only the background object components, the segment traversing the mixed area, the mixture ratio of the pixel corresponding to the position on the segment is estimated.

31. A recording medium according to claim 30, wherein, in said mixture-ratio estimating step, the mixture ratio of the pixel corresponding to the position on the segment is estimated based on the position on the segment parallel to the moving direction of the foreground object.

32. A recording medium according to claim 30, wherein, in said mixture-ratio estimating step, the mixture ratio of the pixel corresponding to the position on the segment is estimated based on the position on the segment obliquely intersecting with the moving direction of the foreground object.

33. A program for allowing a computer for processing image data which is formed of a predetermined number of pixel data obtained by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, to execute:

an area specifying step of outputting area information used for specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed; and

a mixture-ratio estimating step of estimating a mixture ratio indicating the ratio of the foreground object components and the background object components in the mixed area based on the area information and a moving direction of the foreground object by assuming that the mixture ratio changes substantially linearly with respect to a position of the pixel.

34. A program according to claim 33, wherein, in said mixture-ratio estimating step, based on a position on a segment whose one end is positioned at a foreground area consisting of only the foreground object components and the other end is positioned at a background area consisting of only the background object components, the segment traversing the mixed area, the mixture ratio of the pixel corresponding to the position on the segment is estimated.

35. A program according to claim 34, wherein, in said mixture-ratio estimating step, the mixture ratio of the pixel corresponding to the position on the segment is estimated based on the position on the segment parallel to the moving direction of the foreground object.

36. A program according to claim 34, wherein, in said mixture-ratio estimating step, the mixture ratio of the pixel corresponding to the position on the segment is estimated based on the position on the segment obliquely intersecting with the moving direction of the foreground object.

37. An image-capturing apparatus comprising:

image-capturing means for outputting a subject image captured by an image-capturing device including a predetermined number of pixels, the pixels having a time integrating function, as image data formed of a predetermined number of pixel data;

area specifying means for outputting area information used for specifying, in correspondence with the image data, a mixed area in which foreground object components forming a foreground object and background object components forming a background object are mixed; and

mixture-ratio estimating means for estimating a mixture ratio indicating the ratio of the foreground object components and the background object components in the mixed area based on the area information and a moving direction of the foreground object by assuming that the mixture ratio changes substantially linearly with respect to a position of the pixel.

38. An image-capturing apparatus according to claim 37, wherein said mixture-ratio estimating means estimates, based on a position on a segment whose one end is positioned at a foreground area consisting of only the foreground object components and the other end is positioned at a background area consisting of only the background object components, the segment traversing the mixed area, the mixture ratio of the pixel corresponding to the position on the segment.

39. An image-capturing apparatus according to claim 38, wherein said mixture-ratio estimating means estimates the mixture ratio of the pixel corresponding to the position on the segment based on the position on the segment parallel to the moving direction of the foreground object.

40. An image-capturing apparatus according to claim 38, wherein said mixture-ratio estimating means estimates the mixture ratio of the pixel corresponding to the position on the segment based on the position on the segment obliquely intersecting with the moving direction of the foreground object.