Image processing apparatus and method

• US 7,162,101 B2
• Filed: 11/14/2002
• Issued: 01/09/2007
• Est. Priority Date: 11/15/2001
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Maeda, Mitsuru, Itokawa, Osamu, Wakae, Ryotaro
• Primary Examiner(s): Mehta; Bhavesh M.
• Assistant Examiner(s): Seth; Manav
• Application Number:
  US10/293,432
• Publication Number:
  US 20030090751A1
• Time in Patent Office: 1,517 Days
• Field of Search: 382/103, 382/131, 382164-165, 382168-174, 382/180, 382/199, 382/203, 382210-211, 382254-255, 382260-262, 382264-266, 382268-270, 382282-283, 348/169, 348/231.2, 348/208.13, 348/208.14, 348/231.3, 345/472
• US Class Current: 382/282
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation

Abstract

An image processing apparatus and method, which can appropriately extract an object even when the focus of a camera having an automatic focus adjustment function shifts from the background to the object. To this end, frame images which are sensed by an image sensing unit and are sequential in the time axis direction are input from an input unit. The input frame images are smoothed by a filter unit. A background image generation unit generates an average image of a predetermined number of smoothed frame images as a background image. An image differential unit generates a differential image between the predetermined smoothed frame image and the background image. An object extraction unit extracts an object region where a predetermined object is sensed, on the basis of the differential image.

Claims

1. An image processing apparatus for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, comprising:

image input means for inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

smoothing means for smoothing the input frame images;

background image determination means for determining the smoothed frame image as a background image;

differential image generation means for generating a differential image between a predetermined smoothed frame image and the background image; and

object region extraction means for extracting an object region where a predetermined object is sensed, on the basis of the differential image,

wherein said object region extraction means comprises;

comparison means for comparing a pixel value of the differential image with a predetermined threshold value;

extraction means for extracting, as the object region, a region in the predetermined frame image corresponding to a set object region, on the basis of the comparison result with the threshold value;

contour region setting means for setting a contour region with a predetermined width, which includes boundary pixels with a background region within the set object region;

edge extraction means for extracting an edge contained in the contour region in the predetermined frame image; and

contour correction means for correcting a contour of the object region on the basis of a shape of the extracted edge.

2. An image processing apparatus for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, comprising:

image input means for inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

smoothing means for smoothing the input frame images;

background image determination means for determining the smoothed frame image as a background image;

differential image generation means for generating a differential image between a predetermined smoothed frame image and the background image;

object region extraction means for extracting an object region where a predetermined object is sensed, on the basis of the differential image;

sub-sampling means for decimating the number of pixels of each smoothed frame image to a predetermined value; and

enlargement means for enlarging an object region set in the frame image, the number of pixels of which has been decimated, to an original size.

3. The apparatus according to claim 2, wherein said background image determination means comprises sub-sampling means for decimating the number of pixels of each smoothed frame image to a predetermined value.

4. The apparatus according to claim 2, wherein said smoothing means comprises sub-sampling means for decimating the number of pixels of each frame image to a predetermined value.

5. An image processing apparatus for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, comprising:

image input means for inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

smoothing means for smoothing the input frame images;

background image determination means for determining the smoothed frame image as a background image;

differential image generation means for generating a differential image between a predetermined smoothed frame image and the background image;

object region extraction means for extracting an object region where a predetermined object is sensed, on the basis of the differential image;

arithmetic encoding means for arithmetically encoding data of the extracted object region;

padding processing means for executing a padding process of the extracted object region;

texture encoding means for entropy-encoding data of the object region that has undergone the padding process;

header input means for inputting predetermined header information; and

shaping means for encoding data of the extracted object region by a predetermined encoding method using the arithmetically encoded data of the object region, the entropy-encoded data of the object region, and the header information.

6. An image processing method for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, comprising:

an image input step of inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

a smoothing step of smoothing the input frame images;

a background image determination step of determining the smoothed frame image as a background image;

a differential image generation step of generating a differential image between a predetermined smoothed frame image and the background image; and

an object region extraction step of extracting an object region where a predetermined object is sensed, on the basis of the differential image,

wherein said object region extraction step comprises;

a comparison step of comparing a pixel value of the differential image with a predetermined threshold value;

an extraction step of extracting, as the object region, a region in the predetermined frame image corresponding to a set object region, on the basis of the comparison result with the threshold value;

a contour region setting step of setting a contour region with a predetermined width, which includes boundary pixels with a background region within the set object region;

an edge extraction step of extracting an edge contained in the contour region in the predetermined frame image; and

a contour correction step of correcting a contour of the object region on the basis of a shape of the extracted edge.

7. An image processing method for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, comprising:

an image input step of inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

a smoothing step of smoothing the input frame images;

a background image determination step of determining the smoothed frame image as a background image;

a differential image generation step of generating a differential image between a predetermined smoothed frame image and the background image;

an object region extraction step of extracting an object region where a predetermined object is sensed, on the basis of the differential image;

a sub-sampling step of decimating the number of pixels of each smoothed frame image to a predetermined value; and

an enlargement step of enlarging an object region set in the frame image, the number of pixels of which has been decimated, to an original size.

8. The method according to claim 7, wherein said background image determination step comprises the sub-sampling step of decimating the number of pixels of each smoothed frame image to a predetermined value.

9. The method according to claim 7, wherein said smoothing step comprises the sub-sampling step of decimating the number of pixels of each frame image to a predetermined value.

10. An image processing method for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, comprising:

an image input step of inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

a smoothing step of smoothing the input frame images;

a background image determination step of determining the smoothed frame image as a background image;

a differential image generation step of generating a differential image between a predetermined smoothed frame image and the background image;

an object region extraction step of extracting an object region where a predetermined object is sensed, on the basis of the differential image;

an arithmetic encoding step of arithmetically encoding data of the extracted object region;

a padding processing step of executing a padding process of the extracted object region;

a texture encoding step of entropy-encoding data of the object region that has undergone the padding process;

a header input step of inputting predetermined header information; and

shaping step of encoding data of the extracted object region by a predetermined encoding method using the arithmetically encoded data of the object region, the entropy-encoded data of the object region, and the header information.

11. A computer program stored on a computer-readable medium which, when executed, performs a method for controlling an image processing apparatus for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, the program comprising:

program code of an image input step of inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

program code of a smoothing step of smoothing the input frame images;

program code of a background image determination step of determining the smoothed frame image as a background image;

program code of a differential image generation step of generating a differential image between a predetermined smoothed frame image and the background image; and

program code of an object region extraction step of extracting an object region where a predetermined object is sensed, on the basis of the differential images,

wherein the program code of the object region extraction step comprises;

program code of a comparison step of comparing a pixel value of the differential image with a predetermined threshold value;

program code of an extraction step of extracting, as the object region, a region in the predetermined frame image corresponding to a set object region, on the basis of the comparison result with the threshold value;

a contour region setting step of setting a contour region with a predetermined width, which includes boundary pixels with a background region within the set object region;

an edge extraction step of extracting an edge contained in the contour region in the predetermined frame image; and

a contour correction step of correcting a contour of the object region on the basis of a shape of the extracted edge.

12. A recording medium for storing a computer program of claim 11.

13. An image processing method of extracting an object from image data sensed by image sensing means having an adjustment function of an image sensing parameter, comprising:

a storage step of storing, in storage means, background image data sensed by the image sensing means using a plurality of image sensing parameters;

a selection step of selecting one of the background image data stored in the storage means; and

an extraction step of extracting an object other than a background in image data sensed by the image sensing means on the basis of the background image selected in said selection step.

14. The method according to claim 13, wherein said storage step includes the step of storing the image sensing parameters and background image data in correspondence with each other in the storage means.

15. The method according to claim 14, wherein said selection step includes the step of receiving parameter information associated with an image sensing parameter of the image sensing means, and selecting the background image data on the basis of the received parameter information.

16. The method according to claim 13, wherein the selection step includes the calculation step of calculating a matching level between the image data obtained by the image sensing means, and the plurality of background image data stored in the storage means, and includes the step of selecting background image data which has a maximum matching level calculated in said calculation step.

17. The method according to claim 16, wherein said calculation step includes the step of calculating differences for a plurality of local regions, which are set in advance in a field angle.

18. The method according to claim 13, wherein said extraction step includes the step of generating binary information used to identify for each pixel if a pixel of interest is a background pixel or a pixel of an object other than the background pixel.

19. The method according to claim 18, further comprising the encoding step of encoding information data of the object obtained in said extraction step.

20. The method according to claim 19, wherein said encoding step includes the step of encoding by MPEG-4.

21. The method according to claim 13, wherein the image sensing parameter is a parameter associated with focus control or a parameter associated with aperture control.

22. A computer program stored on a computer-readable medium which, when executed, performs an image process for extracting an object from image data sensed by image sensing means having an adjustment function of an image sensing parameter, the program comprising:

program code of a storage step of storing, in storage means, background image data sensed by the image sensing means using a plurality of image sensing parameters;

program code of a selection step of selecting one of the background image data stored in the storage means; and

program code of an extraction step of extracting an object other than a background in image data sensed by the image sensing means on the basis of the background image selected in the selection step.

23. A computer readable storage medium storing a computer program of claim 22.

24. An image processing method of extracting an object from image data sensed by image sensing means having an adjustment function of an image sensing parameter, comprising:

a storage step of storing, in storage means, background image data sensed by the image sensing means;

a detection step of detecting a size of a difference between image data sensed by the image sensing means, and the background image data stored in the storage means;

a correction step of correcting the background image data in accordance with the size of the difference detected in said detection step; and

an extraction step of extracting an object other than a background in image data sensed by the image sensing means on the basis of the background image corrected in said correction step.

25. The method according to claim 24, wherein said storage step includes the step of storing background image data obtained by averaging a plurality of background image data sensed by the image sensing means.

26. The method according to claim 24, wherein said extraction step includes:

the step of detecting a fluctuation width of each pixel with respect to a time axis from a plurality of background image data sensed by the image sensing means, and generating the fluctuation width as threshold value information, and

the extraction step includes the step of extracting the object other than the background from the image data sensed by the image sensing means on the basis of the threshold value information and the corrected background image data.

27. The method according to claim 24, wherein said detection step includes the step of calculating differences for a plurality of predetermined local regions, which are set in advance in a field angle.

28. The method according to claim 24, wherein said extraction step includes the step of generating binary information used to identify for each pixel if a pixel of interest is a background pixel or a pixel of an object other than the background pixel.

29. The method according to claim 28, further comprising the encoding step of encoding information data of the object obtained in said extraction step.

30. The method according to claim 29, wherein said encoding step includes the step of encoding by MPEG-4.

31. The method according to claim 24, wherein the image sensing parameter is a parameter associated with focus control or a parameter associated with aperture control.

32. A computer program stored on a computer-readable medium which, when executed, performs an image process for extracting an object from image data sensed by image sensing means having an adjustment function of an image sensing parameter, the program comprising:

program code of a storage step of storing, in storage means, background image data sensed by the image sensing means;

program code of a detection step of detecting a size of a difference between image data sensed by the image sensing means, and the background image data stored in the storage means;

program code of a correction step of correcting the background image data in accordance with the size of the difference detected in the detection step; and

program code of an extraction step of extracting an object other than a background in image data sensed by the image sensing means on the basis of the background image corrected in the correction step.

33. A computer readable storage medium storing a computer program of claim 32.

34. A computer program stored on a computer-readable medium which, when executed, performs an image processing method for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, the program comprising:

program code of an image input step of inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

program code of a smoothing step of smoothing the input frame images;

program code of a background image determination step of determining the smoothed frame image as a background image;

program code of a differential image generation step of generating a differential image between a predetermined smoothed frame image and the background image; and

program code of an object region extraction step of extracting an object region where a predetermined object is sensed, on the basis of the differential image;

program code of a sub-sampling step of decimating the number of pixels of each smoothed frame image to a predetermined value; and

program code of an enlargement step of enlarging an object region set in the frame image, the number of pixels of which has been decimated, to an original size.

36. A recording medium for storing a computer program of claim 34.

35. A computer program stored on a computer-readable medium which, when executed, performs an image processing method for making predetermined image sensing means sense a scene in a fixed image sensing direction, and extracting an object from an image obtained by the image sensing means, the program comprising:

program code of an image input step of inputting frame images which are sensed by the image sensing means and are sequential in a time axis direction in turn;

program code of a smoothing step of smoothing the input frame images;

program code of a background image determination step of determining the smoothed frame image as a background image;

program code of a differential image generation step of generating a differential image between a predetermined smoothed frame image and the background image;

program code of an object region extraction step of extracting an object region where a predetermined object is sensed, on the basis of the differential image, the arithmetic encoding step of arithmetically encoding data of the extracted object region;

program code of a padding processing step of executing a padding process of the extracted object region;

program code of a texture encoding step of entropy-encoding data of the object region that has undergone the padding process;

program code of a header input step of inputting predetermined header information; and

program code of a shaping step of encoding data of the extracted object region by a predetermined encoding method using the arithmetically encoded data of the object region, the entropy-encoded data of the object region, and the header information.

37. A recording medium for storing a computer program of claim 35.