Object extraction

• US 20030194131A1
• Filed: 04/11/2002
• Published: 10/16/2003
• Est. Priority Date: 04/11/2002
• Status: Active Grant

• Current Assignee: ArcSoft, Inc.
• Original Assignee: ArcSoft, Inc.
• Inventors: Li, Na, Zhao, Bin
• Granted Patent: US 7,254,268 B2
• Time in Patent Office: Days
• US Class Current: 382/190
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20216 Image averaging
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 10/40 Extraction of image or video features

Abstract

A video sequence is processed to extract one or more objects. The extracted objects are used to create a new video sequence. In particular, a user is able to select a background and/or an audio recording. Alternatively, the object extraction system automatically selects a background and/or audio recording. The extracted objects are integrated into the selected background and/or the audio recording is added to form a new video sequence.

Claims

1. A method for extracting a foreground object from a background, comprising:

(a) when a value component of a pixel in a non-static frame exceeds a mean value of the value component for the pixel across multiple frames by a first value, classifying the pixel as foreground;

(b) when the pixel is in shadow, classifying the pixel as background;

(c) when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value, classifying the pixel as background; and

(d) when the pixel is not classified by acts (a), (b), or (c), classifying the pixel as foreground.

2. The method of claim 1, wherein the first value comprises a first threshold.

3. The method of claim 2, wherein the first threshold is an empirical value of 75.

4. The method of claim 1, wherein the second value comprises a standard deviation of a component multiplied by a second threshold.

5. The method of claim 1 wherein the second threshold is dependent on illumination conditions.

6. The method of claim 1, further comprising:

adjusting one or more pixels of the non-static frame by applying a filter core to the non-static frame.

7. The method of claim 6, wherein the non-static frame is a first frame, and further comprising:

mapping each pixel in the first frame to a binary value;

generating a second frame comprised of the binary values of each pixel; and

applying the filter core to the second frame.

8. The method of claim 1, wherein the non-static frame is part of a sequence of frames and wherein one of the other frames in the sequence is a static frame.

9. The method of claim 8, further comprising identifying the static frame in the sequence.

10. The method of claim 1, wherein a component is usable if the standard deviation of the component does not exceed a threshold.

11. The method of claim 10, wherein the threshold comprises 25.

12. The method of claim 10, wherein the component is a hue component.

13. The method of claim 10, wherein the component is a saturation component.

14. The method of claim 1, further comprising:

extracting a foreground object by selecting one or more pixels classified as foreground.

15. The method of claim 14, further comprising:

extracting multiple foreground objects.

16. The method of claim 1, further comprising:

providing a user interface, wherein the user interface enables a user to select a new background.

17. The method of claim 16, further comprising:

extracting one or more foreground objects from the frame; and

integrating the extracted foreground objects with the background.

18. The method of claim 1, further comprising:

providing a user interface, wherein the user interface enables a user to select an audio recording.

19. The method of claim 18 further comprising:

extracting one or more foreground objects from the frame; and

integrating the extracted foreground objects with the audio recording.

20. The method of claim 1, wherein the non-static frame comprises pixels in a RGB color space and further comprising:

converting the RGB color space to a type of HSV color space.

21. A method for processing a sequence of frames, comprising:

extracting one or more objects from a frame by classifying each pixel of the frame as being background or foreground based on a statistical model of a background in the frame, a type model of each pixel, shadow conditions, and illumination conditions;

selecting a new background;

selecting an audio recording; and

integrating the extracted objects with the new background and audio recording.

22. The method of claim 21, further comprising:

applying a filter core to the frame to confirm that each pixel has been accurately classified as background or foreground.

23. A system, comprising:

a computer including a memory;

a sequence of frames stored in the memory; and

a computer program executable by the computer, wherein the computer program comprises instructions for;

(a) when a value component of a pixel in a non-static frame in the sequence of frames exceeds a mean value of the value component for the pixel across multiple frames in the sequence of frames by a first value, classifying the pixel as foreground;

(b) when the pixel is in shadow, classifying the pixel as background;

(c) when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value, classifying the pixel as background; and

(d) when the pixel is not classified by acts (a), (b), or (c), classifying the pixel as foreground.

24. The system of claim 23, wherein the first value comprises a first threshold.

25. The system of claim 24, wherein the first threshold is an empirical value of 75.

26. The system of claim 23, wherein the second value comprises a standard deviation of a component multiplied by a second threshold.

27. The system of claim 23 wherein the second threshold is dependent on illumination conditions.

28. The system of claim 23, wherein the computer program further comprises instructions for:

adjusting one or more pixels of the non-static frame by applying a filter core to the frame.

29. The system of claim 28, wherein the non-static frame is a first frame, and wherein the computer program further comprises instructions for:

mapping each pixel in the first frame to a binary value;

generating a second frame comprised of the binary values of each pixel; and

applying the filter core to the second frame.

30. The system of claim 23, wherein one of the frames in the sequence of frames comprises a static frame.

31. The system of claim 30, wherein the computer program further comprises instructions for:

identifying the static frame in the sequence of frames.

32. The system of claim 23, wherein a component is usable if the standard deviation of the component does not exceed a threshold.

33. The system of claim 32, wherein the threshold comprises 25.

34. The system of claim 32, wherein the component is a hue component.

35. The system of claim 32, wherein the component is a saturation component.

36. The system of claim 23, wherein the computer program further comprises instructions for:

extracting a foreground object by selecting one or more pixels classified as foreground.

37. The system of claim 36, wherein the computer program further comprises instructions for:

extracting multiple foreground objects.

38. The system of claim 23, wherein the computer program further comprises instructions for:

providing a user interface, wherein the user interface enables a user to select a new background.

39. The system of claim 38, wherein the computer program further comprises instructions for:

extracting one or more foreground objects from the frame; and

integrating the extracted foreground objects with the background.

40. The system of claim 23, wherein the computer program further comprises instructions for:

providing a user interface, wherein the user interface enables a user to select an audio recording.

41. The system of claim 40, wherein the computer program further comprises instructions for:

extracting one or more foreground objects from the frame; and

integrating the extracted foreground objects with the audio recording.

42. The system of claim 23, wherein the frame comprises pixels in a RGB color space and wherein the computer program further comprises instructions for:

converting the RGB color space to a type of HSV color space.

43. A system, comprising:

means for extracting one or more objects from a frame by classifying each pixel of the frame as being background or foreground based on a statistical model of a background in the frame, a type model of each pixel, shadow conditions, and illumination conditions;

means for selecting a new background;

means for selecting an audio recording; and

means for integrating the extracted objects with the new background and audio recording.

44. The system of claim 43, further comprising:

means for applying a filter core to the frame to confirm that each pixel has been accurately classified as background or foreground.

45. A computer readable storage medium encoded with a computer program executable by a computer, wherein the computer program comprises instructions for:

(a) when a value component of a pixel in a non-static frame exceeds a mean value of the value component for the pixel across multiple frames by a first value, classifying the pixel as foreground;

(b) when the pixel is in shadow, classifying the pixel as background;

(c) when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value, classifying the pixel as background; and

(d) when the pixel is not classified by acts (a), (b), or (c), classifying the pixel as foreground.

46. The computer readable storage medium of claim 45, wherein the first value comprises a first threshold.

47. The computer readable storage medium of claim 46, wherein the first threshold is an empirical value of 75.

48. The computer readable storage medium of claim 45, wherein the second value comprises a standard deviation of a component multiplied by a second threshold.

49. The computer readable storage medium of claim 45 wherein the second threshold is dependent on illumination conditions.

50. The computer readable storage medium of claim 45, wherein the computer program further comprises instructions for:

adjusting one or more pixels of the non-static frame by applying a filter core to the frame.

51. The computer readable storage medium of claim 50, wherein the non-static frame is a first frame, and wherein the computer program further comprises instructions for:

mapping each pixel in the first frame to a binary value;

generating a second frame comprised of the binary values of each pixel; and

applying the filter core to the second frame.

52. The computer readable storage medium of claim 45, wherein the non-static frame is part of a sequence of frames and wherein one of the frames is a static frame.

53. The computer readable storage medium of claim 52, wherein the computer program further comprises instructions for:

identifying the static frame in the sequence.

54. The computer readable storage medium of claim 45, wherein a component is usable if the standard deviation of the component does not exceed a threshold.

55. The computer readable storage medium of claim 54, wherein the threshold comprises 25.

56. The computer readable storage medium of claim 54, wherein the component is a hue component.

57. The computer readable storage medium of claim 54, wherein the component is a saturation component.

58. The computer readable storage medium of claim 45, wherein the computer program further comprises instructions for:

extracting a foreground object by selecting one or more pixels classified as foreground.

59. The computer readable storage medium of claim 58, wherein the computer program further comprises instructions for:

extracting multiple foreground objects.

60. The computer readable storage medium of claim 45, wherein the computer program further comprises instructions for:

providing a user interface, wherein the user interface enables a user to select a new background.

61. The computer readable storage medium of claim 60, wherein the computer program further comprises instructions for:

extracting one or more foreground objects from the frame; and

integrating the extracted foreground objects with the background.

62. The computer readable storage medium of claim 45, wherein the computer program further comprises instructions for:

providing a user interface, wherein the user interface enables a user to select an audio recording.

63. The computer readable storage medium of claim 62, wherein the computer program further comprises instructions for:

extracting one or more foreground objects from the frame; and

integrating the extracted foreground objects with the audio recording.

64. The computer readable storage medium of claim 45, wherein the frame comprises pixels in a RGB color space and wherein the computer program further comprises instructions for:

converting the RGB color space to a type of HSV color space.

65. A computer readable storage medium encoded with software instructions, wherein the computer program further comprises instructions for:

extracting one or more objects from a frame by classifying each pixel of the frame as being background or foreground based on a statistical model of a background in the frame, a type model of each pixel, shadow conditions, and illumination conditions;

selecting a new background;

selecting an audio recording; and

integrating the extracted objects with the new background and audio recording.

66. The computer readable storage medium of claim 65, wherein the computer program further comprises instructions for:

applying a filter core to the frame to confirm that each pixel has been accurately classified as background or foreground.