Object extraction

• US 7,254,268 B2
• Filed: 04/11/2002
• Issued: 08/07/2007
• Est. Priority Date: 04/11/2002
• Status: Active Grant

• Current Assignee: ArcSoft, Inc.
• Original Assignee: ArcSoft, Inc.
• Inventors: Zhao, Bin, Li, Na
• Primary Examiner(s): Mancuso; Joseph
• Assistant Examiner(s): Akhavannik; Hadi
• Application Number:
  US10/123,019
• Publication Number:
  US 20030194131A1
• Time in Patent Office: 1,944 Days
• Field of Search: 382/173, 382190-194, 348586-587, 345/604
• US Class Current: 382/190
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20216 Image averaging
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 10/40 Extraction of image or video features

Abstract

A video sequence is processed to extract one or more objects. The extracted objects are used to create a new video sequence. In particular, a user is able to select a background and/or an audio recording. Alternatively, the object extraction system automatically selects a background and/or audio recording. The extracted objects are integrated into the selected background and/or the audio recording is added to form a new video sequence.

Claims

1. A method for classifying an object in a non-static frame as foreground or background, comprising:

(a) classifying a pixel in the non-static frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;

(b) classifying the pixel as background when the pixel is in shadow;

(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and

(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c).

2. The method of claim 1, wherein the first value comprises a first threshold.

3. The method of claim 2, wherein the first threshold is an empirical value of 75.

4. The method of claim 1, wherein the second value comprises a standard deviation of a component multiplied by a second threshold.

5. The method of claim 1 wherein the second value is dependent on illumination conditions.

6. The method of claim 1, further comprising:

adjusting one or more pixels of the non-static frame by applying a filter core to the non-static frame.

7. The method of claim 6, wherein the non-static frame is a first frame, and further comprising:

mapping each pixel in the first frame to a binary value;

generating a second frame comprised of the binary values of each pixel; and

applying the filter core to the second frame.

8. The method of claim 1, wherein the non-static frame is part of a sequence of frames and wherein one of the other frames in the sequence is a static frame.

9. The method of claim 8, further comprising identifying the static frame in the sequence.

10. The method of claim 1, wherein a component is usable if the standard deviation of the component does not exceed a threshold.

11. The method of claim 10, wherein the threshold comprises 25.

12. The method of claim 10, wherein the component is a hue component.

13. The method of claim 10, wherein the component is a saturation component.

14. The method of claim 1, further comprising:

extracting a foreground object by selecting one or more pixels classified as foreground.

15. The method of claim 14, further comprising:

extracting multiple foreground objects.

16. The method of claim 1, further comprising:

providing a user interface, wherein the user interface enables a user to select a new background.

17. The method of claim 16, further comprising:

extracting one or more foreground objects from the non-static frame; and

integrating the extracted foreground objects with the new background.

18. The method of claim 1, further comprising:

providing a user interface, wherein the user interface enables a user to select an audio recording.

19. The method of claim 18, further comprising:

extracting one or more foreground objects from the non-static frame; and

integrating the extracted foreground objects with the audio recording.

20. The method of claim 1, wherein the non-static frame comprises pixels in a RGB color space and further comprising:

converting the RGB color space to a type of HSV color space.

21. A method for processing a sequence of frames, comprising:

extracting one or more objects from a frame by classifying each pixel of the frame as being background or foreground, said classifying each pixel comprising;

(a) classifying a pixel in the frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;

(b) classifying the pixel as background when the pixel is in shadow;

(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and

(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c);

selecting a new background;

selecting an audio recording; and

integrating the extracted objects with the new background and the audio recording.

22. The method of claim 21, further comprising:

applying a filter core to the frame to confirm that each pixel has been accurately classified as background or foreground.

23. A system, comprising:

a computer including a memory;

a sequence of frames stored in the memory; and

a computer program executable by the computer, the computer program comprising instructions for classifying an object in a non-static frame from the sequence of frame as foreground or background, the instructions comprising;

(a) classifying in a pixel in the non-static frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames in the sequence of frames by a first value;

(b) classifying the pixel as background when the pixel is in shadow;

(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and

(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c).

24. The system of claim 23, wherein the first value comprises a first threshold.

25. The system of claim 24, wherein the first threshold is an empirical value of 75.

26. The system of claim 23, wherein the second value comprises a standard deviation of a component multiplied by a second threshold.

27. The system of claim 23 wherein the second value is dependent on illumination conditions.

28. The system of claim 23, wherein the computer program further comprises instructions for:

adjusting one or more pixels of the non-static frame by applying a filter core to the frame.

29. The system of claim 28, wherein the non-static frame is a first frame, and wherein the computer program further comprises instructions for:

mapping each pixel in the first frame to a binary value;

generating a second frame comprised of the binary values of each pixel; and

applying the filter core to the second frame.

30. The system of claim 23, wherein one of the frames in the sequence of frames comprises a static frame.

31. The system of claim 30, wherein the computer program further comprises instructions for:

identifying the static frame in the sequence of frames.

32. The system of claim 23, wherein a component is usable if the standard deviation of the component does not exceed a threshold.

33. The system of claim 32, wherein the threshold comprises 25.

34. The system of claim 32, wherein the component is a hue component.

35. The system of claim 32, wherein the component is a saturation component.

36. The system of claim 23, wherein the computer program further comprises instructions for:

extracting a foreground object by selecting one or more pixels classified as foreground.

37. The system of claim 36, wherein the computer program further comprises instructions for:

extracting multiple foreground objects.

38. The system of claim 23, wherein the computer program further comprises instructions for:

providing a user interface, wherein the user interface enables a user to select a new background.

39. The system of claim 38, wherein the computer program further comprises instructions for:

extracting one or more foreground objects from the non-static frame; and

integrating the extracted foreground objects with the new background.

40. The system of claim 23, wherein the computer program further comprises instructions for:

providing a user interface, wherein the user interface enables a user to select an audio recording.

41. The system of claim 40, wherein the computer program further comprises instructions for:

extracting one or more foreground objects from the non-static frame; and

integrating the extracted foreground objects with the audio recording.

42. The system of claim 23, wherein the non-static frame comprises pixels in a RGB color space and wherein the computer program further comprises instructions for:

converting the RGB color space to a type of HSV color space.

43. A system, comprising:

means for extracting one or more objects from a frame by classifying each pixel of the frame as being background or foreground, said classifying comprising;

(a) classifying a pixel as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;

(b) classifying the pixel as background when the pixel is in shadow;

(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and

(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c);

means for selecting a new background;

means for selecting an audio recording; and

means for integrating the extracted objects with the new background and audio recording.

44. The system of claim 43, further comprising:

means for applying a filter core to the frame to confirm that each pixel has been accurately classified as background or foreground.

45. A computer readable storage medium encoded with a computer program executable by a computer, the computer program comprising instructions for classifying an object in a non-static frame as foreground or background, wherein the instructions comprise:

(a) classifying a pixel in the non-static frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;

(b) classifying the pixel as background when the pixel is in shadow;

(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and

(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c).

46. The computer readable storage medium of claim 45, wherein the first value comprises a first threshold.

47. The computer readable storage medium of claim 46, wherein the first threshold is an empirical value of 75.

48. The computer readable storage medium of claim 45, wherein the second value comprises a standard deviation of a component multiplied by a second threshold.

49. The computer readable storage medium of claim 45, wherein the second value is dependent on illumination conditions.

50. The computer readable storage medium of claim 45, wherein the computer program further comprises instructions for:

adjusting one or more pixels of the non-static frame by applying a filter core to the frame.

51. The computer readable storage medium of claim 50, wherein the non-static frame is a first frame, and wherein the computer program further comprises instructions for:

mapping each pixel in the first frame to a binary value;

generating a second frame comprised of the binary values of each pixel; and

applying the filter core to the second frame.

52. The computer readable storage medium of claim 45, wherein the non-static frame is part of a sequence of frames and wherein one of the frames is a static frame.

53. The computer readable storage medium of claim 52, wherein the computer program further comprises instructions for:

identifying the static frame in the sequence.

54. The computer readable storage medium of claim 45, wherein a component is usable if the standard deviation of the component does not exceed a threshold.

55. The computer readable storage medium of claim 54, wherein the threshold comprises 25.

56. The computer readable storage medium of claim 54, wherein the component is a hue component.

57. The computer readable storage medium of claim 54, wherein the component is a saturation component.

58. The computer readable storage medium of claim 45, wherein the computer program further comprises instructions for:

extracting a foreground object by selecting one or more pixels classified as foreground.

59. The computer readable storage medium of claim 58, wherein the computer program further comprises instructions for:

extracting multiple foreground objects.

60. The computer readable storage medium of claim 45, wherein the computer program further comprises instructions for:

providing a user interface, wherein the user interface enables a user to select a new background.

61. The computer readable storage medium of claim 60, wherein the computer program further comprises instructions for:

extracting one or more foreground objects from the non-static frame; and

integrating the extracted foreground objects with the new background.

62. The computer readable storage medium of claims 45, wherein the computer program further comprises instructions for:

providing a user interface, wherein the user interface enables a user to select an audio recording.

63. The computer readable storage medium of claim 62, wherein the computer program further comprises instructions for:

extracting one or more foreground objects from the non-static frame; and

integrating the extracted foreground objects with the audio recording.

64. The computer readable storage medium of claim 45, wherein the non-static frame comprises pixels in a RGB color space and wherein the computer program further comprises instructions for:

converting the RGB color space to a type of HSV color space.

65. A computer readable storage medium encoded with software instructions, wherein the the software instructions comprise:

extracting one or more objects from a frame by classifying each pixel of the frame as being background or foreground, said classifying comprising;

(a) classifying a pixel in the frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;

(b) classifying the pixel as background when the pixel is in shadow;

(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and

(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c);

selecting a new background;

selecting an audio recording; and

integrating the extracted objects with the new background and audio recording.

66. The computer readable storage medium of claim 65, wherein the computer program further comprises instructions for:

applying a filter core to the frame to confirm that each pixel has been accurately classified as background or foreground.