Image processing apparatus and method

• US 20030090751A1
• Filed: 11/14/2002
• Published: 05/15/2003
• Est. Priority Date: 11/15/2001
• Status: Active Grant

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: Maeda, Mitsuru, Wakae, Ryotaro, Itokawa, Osamu
• Granted Patent: US 7,162,101 B2
• Time in Patent Office: Days
• US Class Current: 358/538
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation

Abstract

This invention provides an image processing apparatus and method, which can appropriately extract an object even when the focus of a camera having an automatic focus adjustment function shifts from the background to the object. To this end, frame images which are sensed by an image sensing unit and are sequential in the time axis direction are input from an input unit (1). The input frame images are smoothed by a filter unit (2). A background image generation unit (3) generates an average image of a predetermined number of smoothed frame images as a background image. An image differential unit (5) generates a differential image between the predetermined smoothed frame image and the background image. An object extraction unit extracts an object region where a predetermined object is sensed, on the basis of the differential image.

Claims

1. An image processing apparatus for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, comprising:

image input means for inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

smoothing means for smoothing the input frame images;

background image determination means for determining the smoothed frame image as a background image;

differential image generation means for generating a differential image between a predetermined smoothed frame image and the background image; and

object region extraction means for extracting an object region where a predetermined object is sensed, on the basis of the differential image.

2. The apparatus according to claim 1, wherein said background image determination means generates an average image of a predetermined number of smoothed frame images as the background image.

3. The apparatus according to claim 1, wherein the image sensing means comprises an automatic focus adjustment function,

said background image determination means generates the background image using a predetermined number of smoothed frame images obtained by sensing a scene without the object, and

the predetermined frame used to generate the differential image with the background image in said differential image generation means is a frame image obtained by sensing a scene including the object.

4. The apparatus according to claim 1, wherein said object region extraction means comprises:

comparison means for comparing a pixel value of the differential image with a predetermined threshold value; and

extraction means for extracting, as the object region, a region in the predetermined frame image corresponding to a set object region, on the basis of the comparison result with the threshold value.

5. The apparatus according to claim 4, wherein said object region extraction means further comprises:

contour region setting means for setting a contour region with a predetermined width, which includes boundary pixels with a background region within the set object region;

edge extraction means for extracting an edge contained in the contour region in the predetermined frame image; and

contour correction means for correcting a contour of the object region on the basis of a shape of the extracted edge.

6. The apparatus according to claim 1, further comprising:

sub-sampling means for decimating the number of pixels of each smoothed frame image to a predetermined value; and

enlargement means for enlarging an object region set in the frame image, the number of pixels of which has been decimated, to an original size.

7. The apparatus according to claim 6, wherein said background image determination means comprises sub-sampling means for decimating the number of pixels of each smoothed frame image to a predetermined value.

8. The apparatus according to claim 6, wherein said smoothing means comprises sub-sampling means for decimating the number of pixels of each frame image to a predetermined value.

9. The apparatus according to claim 1, further comprising:

arithmetic encoding means for arithmetically encoding data of the extracted object region;

padding processing means for executing a padding process of the extracted object region;

texture encoding means for entropy-encoding data of the object region that has undergone the padding process;

header input means for inputting predetermined header information; and

shaping means for encoding data of the extracted object region by a predetermined encoding method using the arithmetically encoded data of the object region, the entropy-encoded data of the object region, and the header information.

10. An image processing method for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, comprising:

the image input step of inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

the smoothing step of smoothing the input frame images;

the background image determination step of determining the smoothed frame image as a background image;

the differential image generation step of generating a differential image between a predetermined smoothed frame image and the background image; and

the object region extraction step of extracting an object region where a predetermined object is sensed, on the basis of the differential image.

11. The method according to claim 10, wherein the background image determination step includes the step of generating an average image of a predetermined number of smoothed frame images as the background image.

12. The method according to claim 10, wherein the image input step has an automatic focus adjustment function,

the background image determination step includes the step of generating the background image using a predetermined number of smoothed frame images obtained by sensing a scene without the object, and

the predetermined frame used to generate the differential image with the background image in the differential image generation step is a frame image obtained by sensing a scene including the object.

13. The method according to claim 10, wherein the object region extraction step comprises:

the comparison step of comparing a pixel value of the differential image with a predetermined threshold value; and

the extraction step of extracting, as the object region, a region in the predetermined frame image corresponding to a set object region, on the basis of the comparison result with the threshold value.

14. The method according to claim 13, wherein the object region extraction step further comprises:

the contour region setting step of setting a contour region with a predetermined width, which includes boundary pixels with a background region within the set object region;

the edge extraction step of extracting an edge contained in the contour region in the predetermined frame image; and

the contour correction step of correcting a contour of the object region on the basis of a shape of the extracted edge.

15. The method according to claim 10, further comprising:

the sub-sampling step of decimating the number of pixels of each smoothed frame image to a predetermined value; and

the enlargement step of enlarging an object region set in the frame image, the number of pixels of which has been decimated, to an original size.

16. The method according to claim 15, wherein the background image determination step comprises the sub-sampling step of decimating the number of pixels of each smoothed frame image to a predetermined value.

17. The method according to claim 15, wherein the smoothing step comprises the sub-sampling step of decimating the number of pixels of each frame image to a predetermined value.

18. The method according to claim 10, further comprising:

the arithmetic encoding step of arithmetically encoding data of the extracted object region;

the padding processing step of executing a padding process of the extracted object region;

the texture encoding step of entropy-encoding data of the object region that has undergone the padding process;

the header input step of inputting predetermined header information; and

the shaping step of encoding data of the extracted object region by a predetermined encoding method using the arithmetically encoded data of the object region, the entropy-encoded data of the object region, and the header information.

19. A computer program for controlling an image processing apparatus for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, comprising:

a program code of the image input step of inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

a program code of the smoothing step of smoothing the input frame images;

a program code of the background image determination step of determining the smoothed frame image as a background image;

a program code of the differential image generation step of generating a differential image between a predetermined smoothed frame image and the background image; and

a program code of the object region extraction step of extracting an object region where a predetermined object is sensed, on the basis of the differential image.

20. A recording medium for storing a computer program of claim 19.

21. An image processing method of extracting an object from image data sensed by image sensing means having an adjustment function of an image sensing parameter, comprising:

the storage step of storing, in storage means, background image data sensed by the image sensing means using a plurality of image sensing parameters;

the selection step of selecting one of the background image data stored in the storage means; and

the extraction step of extracting an object other than a background in image data sensed by the image sensing means on the basis of the background image selected in the selection step.

22. The method according to claim 21, wherein the storage step includes the step of storing the image sensing parameters and background image data in correspondence with each other in the storage means.

23. The method according to claim 22, wherein the selection step includes the step of receiving parameter information associated with an image sensing parameter of the image sensing means, and selects the background image data on the basis of the received parameter information.

24. The method according to claim 21, wherein the selection step includes the calculation step of calculating a matching level between the image data obtained by the image sensing means, and the plurality of background image data stored in the storage means, and includes the step of selecting background image data which has a maximum matching level calculated in the calculation step.

25. The method according to claim 24, wherein the calculation step includes the step of calculating differences for a plurality of local regions, which are set in advance in a field angle.

26. The method according to claim 21, wherein the extraction step includes the step of generating binary information used to identify for each pixel if a pixel of interest is a background pixel or a pixel of an object other than the background pixel.

27. The method according to claim 26, further comprising the encoding step of encoding information data of the object obtained in the extraction step.

28. The method according to claim 27, wherein the encoding step includes the step of encoding by MPEG-4.

29. The method according to claim 21, wherein the image sensing parameter is a parameter associated with focus control or a parameter associated with aperture control.

30. A computer program for an image process for extracting an object from image data sensed by image sensing means having an adjustment function of an image sensing parameter, comprising:

a program code of the storage step of storing, in storage means, background image data sensed by the image sensing means using a plurality of image sensing parameters;

a program code of the selection step of selecting one of the background image data stored in the storage means; and

a program code of the extraction step of extracting an object other than a background in image data sensed by the image sensing means on the basis of the background image selected in the selection step.

31. A computer readable storage medium storing a computer program of claim 30.

32. An image processing method of extracting an object from image data sensed by image sensing means having an adjustment function of an image sensing parameter, comprising:

the storage step of storing, in storage means, background image data sensed by the image sensing means;

the detection step of detecting a size of a difference between image data sensed by the image sensing means, and the background image data stored in the storage means;

the correction step of correcting the background image data in accordance with the size of the difference detected in the detection step; and

the extraction step of extracting an object other than a background in image data sensed by the image sensing means on the basis of the background image corrected in the correction step.

33. The method according to claim 32, wherein the storage step includes the step of storing background image data obtained by averaging a plurality of background image data sensed by the image sensing means.

34. The method according to claim 32, wherein the extraction step includes:

the step of detecting a fluctuation width of each pixel with respect to a time axis from a plurality of background image data sensed by the image sensing means, and generating the fluctuation width as threshold value information, and

the extraction step includes the step of extracting the object other than the background from the image data sensed by the image sensing means on the basis of the threshold value information and the corrected background image data.

35. The method according to claim 32, wherein the detection step includes the step of calculating differences for a plurality of predetermined local regions, which are set in advance in a field angle.

36. The method according to claim 32, wherein the extraction step includes the step of generating binary information used to identify for each pixel if a pixel of interest is a background pixel or a pixel of an object other than the background pixel.

37. The method according to claim 36, further comprising the encoding step of encoding information data of the object obtained in the extraction step.

38. The method according to claim 37, wherein the encoding step includes the step of encoding by MPEG-4.

39. The method according to claim 32, wherein the image sensing parameter is a parameter associated with focus control or a parameter associated with aperture control.

40. A computer program for an image process for extracting an object from image data sensed by image sensing means having an adjustment function of an image sensing parameter, comprising:

a program code of the storage step of storing, in storage means, background image data sensed by the image sensing means;

a program code of the detection step of detecting a size of a difference between image data sensed by the image sensing means, and the background image data stored in the storage means;

a program code of the correction step of correcting the background image data in accordance with the size of the difference detected in the detection step; and

a program code of the extraction step of extracting an object other than a background in image data sensed by the image sensing means on the basis of the background image corrected in the correction step.

41. A computer readable storage medium storing a computer program of claim 40.