Object extraction apparatus

• US 6,335,985 B1
• Filed: 12/30/1998
• Issued: 01/01/2002
• Est. Priority Date: 01/07/1998
• Status: Expired due to Fees

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Watanabe, Toshiaki, Ida, Takashi, Sambonsugi, Yoko
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/222,876
• Time in Patent Office: 1,098 Days
• Field of Search: 382/190, 382/191, 382/192, 382/189, 382/193, 382/194, 382/291, 382/220, 382/258, 386/109, 386/111, 386/112
• US Class Current: 382/190
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes

Abstract

Rectangles R(i−1), R(i), and R(i+1) are set to surround three temporally continuous frames f(i−1), f(i), and f(i+1). Difference images fd(i−1, i) and fd(i, i+1) are obtained on the basis of the inter-frame differences between the current frame f(i) and the first reference frame f(i−1) and between the current frame f(i) and the second reference frame (i+1). Background regions are respectively determined for polygons Rd(i−1, i)=R(i−1) or R(i) and Rd(i, i+1)=R(i) or R(i+1), and the remaining regions are selected as object region candidates. By obtaining the intersection between these object region candidates, an object region O(i) on the current frame f(i) can be extracted.

Claims

1. An object extraction apparatus for a moving picture, comprising:

a background region determination section which determines a first background region common to a current frame and a first reference frame, and a second background region common to the current frame and a second reference frame, the current frame containing a target object to be extracted from a moving picture signal, the first reference frame being temporally different from the current frame on the basis of a difference between the current frame and the first reference frame, the second reference frame being temporally different from the current frame on the basis of a difference between the current frame and the second reference frame, and the first background region and the second background region indicating a background in the moving picture; and

an extraction section which extracts a region, in a picture on the current frame, which belongs to neither the first background region nor the second background region as an object region.

2. The apparatus according to claim 1, which comprises a still object determination section which determines pixels of the current frame as the object region when pixels of one of the first and second reference frames belongs to the object region, and which determines the pixels of the current frame as the background region when the pixels of one of the first and second reference frames belongs to the background region, using a predetermined shape of the object of the one of the first and second reference frames in a case that the difference between the pixels of the current frame and the pixels of the one of the first and second reference frames is small.

3. The apparatus according to claim 2, wherein the still object determination section uses the predetermined shape of the object, when the shape of the object of the one of the first and second reference frames has already been extracted, and a shape of the object of one of the first and second reference frames which is created from the frame, from which the shape of the object has been extracted, by a block matching method, when the object region is not extracted.

4. The apparatus according to claim 1, further comprising a background correction section which corrects motion of a background on each of the first and second reference frames or the current frame such that the motion of the background between each of the first and second reference frames and the current frame becomes relatively zero.

5. The apparatus according to claim 1, wherein the background region determination section includes a determination section which determines the common background region using a predetermined threshold value.

6. The apparatus according to claim 5, wherein the background region determination section includes a setting on which sets the threshold value to a larger value than the predetermined threshold value when the difference of the current frame is larger than a predetermined value, and to a smaller value than it when the difference is smaller.

7. The apparatus according to claim 5, wherein the background region determination section includes a dividing section which divides the current frame into a plurality of regions, which measures a difference between each of the regions and each of corresponding regions of one of the first and second reference frames, and which sets the threshold value to a larger value than a predetermined value when the difference is larger than a predetermined value and to a smaller value when it is smaller.

8. The apparatus according to claim 1, further comprising a prediction section which predicts a position or shape of the object on the current frame from a frame from which the object region has already been extracted, and a selection section which selects the first and second reference frames to be used by said background region determination section on the basis of the position or shape of the object on the current frame which is predicted by said prediction section.

9. The apparatus according to claim 1, wherein said apparatus further comprises an initial figure setting section which sets a figure surrounding the target object on an initial frame of the moving picture signal, and a figure setting section which sets on one of the first and second reference frames a figure surrounding a region on each input frame of the moving picture signal which corresponds to an image inside figure of one of the first and second reference frames that temporally differs from the input frame on the basis of a correlation between the input frame and the image inside figure, and said object region extraction section extracts a region, in the image inside figure, which belongs to neither the first background region nor the second background region as an object region.

10. The apparatus according to claim 8, wherein said initial figure setting section sets a figure surrounding the target object on the basis of an external input.

11. An object extraction apparatus for a moving picture comprising:

an initial figure setting section which sets a figure surrounding a target object on an input frame of a moving picture signal;

a figure setting section which sets, on the input frame, a figure surrounding a region on the input frame of the moving picture signal and corresponding to an image inside figure of a reference frame that temporally differs from the input frame for each input frame on the basis of a correlation between the input frame and the image inside figure;

a background region determination section which determines a first background region common to a current frame as an object extraction target and a first reference frame and a second background region common to the current frame and a second reference frame, the first reference frame being temporally different from the current frame on the basis of a difference between the current frame and the first reference frame, the second reference frame being temporally different from the current frame on the basis of a difference between the current frame and the second reference frame, and the first background region and the second background region indicating a background in the moving picture;

a first object extraction section which extracts a region, in the image inside figure of the current frame, which belongs to neither the first background region nor the second background region, as an object region;

a second object extraction section which extracts an object region from the image inside figure on the current frame as the object extraction target by using a method different from that used by said first object extraction section; and

a switching section which selectively switches the first and second object extraction sections.

12. The apparatus according to claim 11, which further comprises a feature extraction section which extracts a feature value of a picture in at least a partial region of the current frame as the object extraction target from the current frame, and wherein said switching section selectively switches said first and second object extraction sections on the basis of the extracted feature value.

13. The apparatus according to claim 11, wherein said second object extraction section includes a prediction section which uses a frame, from which the object region has already been extracted, as a reference frame, to predict a position or shape of the object on the current frame as the object extraction target from the reference frame.

14. The apparatus according to claim 13, wherein said first and second object extraction sections are selectively switched and used in units of blocks of each frame on the basis of a prediction error amount such that the extraction result obtained by said second object extraction section is used as an object region when the prediction error caused by said second object extraction section falls within a predetermined range, and the extraction result obtained by said first object extraction section is used as an object region when the prediction error exceeds the predetermined range.

15. The apparatus according to claim 11, wherein said second object extraction section performs inter-frame prediction in a sequence different from an input frame sequence such that a frame interval between the reference frame and the current frame as the object extraction target is set to not less than a predetermined number of frames.

16. An object extraction apparatus comprising:

a picture input section which inputs moving picture data and shape data representing an object region on a predetermined frame of a plurality of frames corresponding to the moving picture data;

a segmentation section which segments a current frame into a plurality of blocks;

a search section which searches a reference frame for a similar block, for each of the blocks, which is similar in figure represented by picture data to a current block and is larger in area than the current block, the reference frame being temporally different from the current frame;

a paste section which pastes shape data obtained by extracting and reducing shape data of each similar block from the reference frame on each block of the current frame; and

an output section which outputs the pasted shaped data as shape data of the current frame.

17. An object extraction apparatus comprising:

a picture input section which inputs picture data representing a picture including at least one object and a background surrounding the object and shape data representing the object;

a setting section which sets blocks on a contour portion of the object, and searches for a similar block, for each of the blocks, which is similar in graphic figure represented by the picture data to each block and is larger than the block, from the same picture to obtain a plurality of similar blocks;

a replacement section which replaces the shape data of each of the blocks with reduced shape data obtained by reducing the shape data of each of the similar blocks;

a repeat section which repeats the replacement by a predetermined number of times; and

an output section which outputs shape data obtained by repeating the replacement as corrected shape data.

18. The apparatus according to claim 17, further comprising a second repeat section which repeats the searching for the similar block and the predetermined number of replacements of the shape data by a plurality of times while decreasing a block size every repetition.

19. A method for extracting a moving object from an input moving picture, comprising the steps of:

determining a first background region common to a current frame containing a target object to be extracted from a moving picture signal and a first reference frame that temporally differs from the current frame on the basis of a difference between the current frame and the first reference frame;

determining a second background region common to the current frame and a second reference frame that temporally differs from the current frame on the basis of a difference between the current frame and the second reference frame; and

extracting a region, in a picture on the current frame, which belongs to neither the first background region nor the second background region as an object region, the first background region and the second background region indicating a background in the input moving picture.

20. The method according to claim 19, which comprises a step of determining pixels of the current frame as the object region when pixels of one of the first and second reference frames belongs to the object region, and determining the pixels of the current frame as the background region when the pixels of one of the first and second reference frames belongs to the background region, using a predetermined shape of the object of one of the first and second reference frames in a case of that the difference between the pixels of the current frame and the pixels the one of the first and second reference frames is small.

21. The method according to claim 20, wherein the step of determining the pixels of the current frame uses the predetermined shape of the object, when the shape of the object of one of the first and second reference frames has already been extracted, and a shape of the block of the one of the first and second reference frames which is created from the frame, from which the shape of the object has been extracted, by a block matching method, when the object region is not extracted.

22. The method according to claim 19, further comprising a step of correcting motion of a background on one of the first and second reference frames or the current frame such that the motion of the background between each of the first and second reference frames and the current frame becomes relatively zero.

23. The method according to claim 19, wherein the background region determining step includes determining the common background region using a predetermined threshold value.

24. The method according to claim 23, wherein the background region determining step includes setting the threshold value to a larger value than the predetermined threshold value when the difference of the current frame is larger than a predetermined value, and to a smaller value than it when the difference is smaller.

25. The method according to claim 23, wherein the background region determining step includes dividing the current frame into a plurality of regions, measuring a difference between each of the regions and each of corresponding regions of one of the first and second reference frames, and setting the threshold value to a larger value than a predetermined value when the difference is larger than a predetermined value and to a smaller value when it is smaller.

26. The method according to claim 19, further comprising the step of predicting a position or shape of the object on the current frame from a frame from which the object region has already been extracted, and selecting the first and second reference frames to be used by said background region determining step on the basis of the position or shape of the object on the current frame which is predicted by said predicting step.

27. The method according to claim 19, which further comprises a step of setting a figure surrounding the target object on an initial frame of the moving picture signal, and a step of setting on one of the first and second reference frames a figure surrounding a region on each input frame of the moving picture signal which corresponds to an image inside figure of one of the first and second reference frames that temporally differs from the input frame on the basis of a correlation between the input frame and the image inside figure, and said object region extracting step extracts a region, in the image inside figure, which belongs to neither the first background region nor the second background region as an object region.

28. The method according to claim 26, wherein said setting step sets a figure surrounding the target object on the basis of an external input.

29. A method for extracting an object from an input picture comprising the steps of:

setting a figure surrounding a target object on an initial frame of a moving picture signal;

setting on an input frame a figure surrounding a region on the input frame of the moving picture signal and corresponding to an image inside figure of a reference frame that temporally differs from the input frame on the basis of a correlation between the input frame and the image inside figure;

determining a first background region common to a current frame as an object extraction target and a first reference frame that temporally differs from the current frame on the basis of a difference between the current frame and the first reference frame, and determining a second background region common to the current frame and a second reference frame that temporally differs from the current frame on the basis of a difference between the current frame and the second reference frame, the first background region and the second background region indicating a background in the input picture

extracting a region, in the image inside figure of the current frame, which belongs to neither the first background region nor the second background region, as an object region;

extracting an object region from the image inside figure on the current frame as the object extraction target by using a method different from that used by said extracting steps; and

selectively switching the extracting steps.

30. The method according to claim 29, further comprising the step of extracting a feature value of a picture in at least a partial region of the current frame as the object extraction target from the current frame, and wherein said switching step selectively switches said extracting steps on the basis of the extracted feature value.

31. The method according to claim 29, wherein said extracting an object region step includes predicting a position or shape of the object on the current frame as the object, using a frame, from which an object region has already been extracted, as a reference frame, to predict a position or shape of the object on the current frame as an object extraction target from the reference frame.

32. The method according to claim 31, wherein said extracting steps are selectively switched and used in units of blocks of each frame on the basis of a prediction error amount such that the extraction result obtained by said extracting an object region step is used as an object region when the prediction error caused by said extracting an object region step falls within a predetermined range, and the extraction result obtained by said extracting a region step is used as an object region when the prediction error exceeds the predetermined range.

33. The method according to claim 29, wherein said extracting an object region step performs inter-frame prediction in a sequence different from an input frame sequence such that a frame interval between a reference frame and the current frame as the object extraction target is set to not less than a predetermined number of frames.

34. A method of extracting an object from an input picture comprising the steps of:

inputting moving picture data and shape data representing an object region on a predetermined frame of a plurality of frames constituting the moving picture data;

segmenting a currently processed frame into a plurality of blocks;

searching for a similar block, for each of the blocks, which is similar in figure represented by picture data to the currently processed block and is larger in area than the currently processed block, from the reference frame;

pasting shape data obtained by extracting and reducing shape data of each similar block from the reference frame on each block of the currently processed frame; and

outputting the pasted shaped data as shape data of the currently processed frame.

35. A method of extracting an object from an input picture comprising:

inputting picture data and shape data representing an object region on the picture;

setting blocks on a contour portion of the shape data;

searching for a similar block, for each of the blocks, which is similar in graphic figure represented by the picture data to each block and is larger than the block, from the same picture;

replacing the shape data of each of the blocks with shape data obtained by reducing the shape data of each of the similar blocks;

repeating the replacement by a predetermined number of times; and

outputting shape data obtained by repeating the replacement as corrected shape data.

36. The method according to claim 35, further comprising a step of secondly repeating the searching for the similar block and the predetermined number of replacements of the shape data by a plurality of times while decreasing a block size every repetition.