Method and apparatus for model-based compositing

• US 6,532,022 B1
• Filed: 10/15/1997
• Issued: 03/11/2003
• Est. Priority Date: 10/15/1997
• Status: Expired due to Term

• Current Assignee: Hanger Solutions, LLC (IP Investments Group LLC)
• Original Assignee: Planet Electric Incorporated
• Inventors: Ahmad, Subutai
• Primary Examiner(s): Nguyen, Thu
• Application Number:
  US08/951,089
• Time in Patent Office: 1,973 Days
• Field of Search: 345/629, 382/282, 382/284, 382/164, 382/167, 382/272, 382/283, 382/291, 358/464, 358/453, 358/450
• US Class Current: 345/629
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

An object is composited digitally from an input image onto a destination image. A background model is created and a frame of an input image containing the object is obtained. An alpha image is created in which each pixel is either a zero, indicating it is not part of the object, or a one, indicating that it is part of the object. The effect of shadows emanating from the object is reduced. A set of templates is then derived to allow holes or gaps in the object created during the compositing process to be filled to a large extent. The object is blended onto a destination image using the alpha image as a blending coefficient.

Claims

1. A computer-implemented digital video object compositing method comprising:

creating a background image model by examining a plurality of frames of an input image that does not contain an object to be composited;

obtaining a frame of an input image that contains the object to be composited;

creating an alpha image according to the input image that contains the object to be composited and the background image model;

determining whether a portion of the input image that contains the object to be composited comprises a shadow, wherein the shadow is determined by assigning a color value to a pixel in the background image, assigning a brightness value to the pixel in the background image, comparing the color and brightness values assigned to a corresponding pixel in the input image that contains the object, and identifying the corresponding pixel as being in a shadow where the corresponding pixel color value is substantially unchanged and the corresponding pixel brightness value is substantially less than the brightness value of the pixel in the background image;

transferring shadow information to the alpha image;

deriving a set of templates comprising the shape of the object being composited, wherein the set of templates are derived from the alpha image;

filling gaps in the alpha image according to the set of templates; and

blending the object in the input image that contains the object onto a destination image using the alpha image, thereby compositing the object onto a destination image with reduced shadows and gaps in the object.

2. A method as recited in claim 1 wherein creating a background image model further includes initializing the input image that does not contain the object.

3. A method as recited in claim 1 wherein creating a background image model further includes retrieving minimum values and maximum values of pixels in the input image that does not contain the object thereby setting a first tolerance value for pixels in the input image that does not contain the object to be the difference between the minimum values and the maximum values of the pixels.

4. A method as recited in claim 1 wherein creating a background image model further includes creating a pixel value chart containing minimum values and maximum values of pixels in the background image model.

5. A method as recited in claim 4 further including updating the minimum and maximum pixel values in the pixel value chart through a process having:

determining a corresponding pixel in the input image that contains the object;

assigning a current color value of the corresponding pixel;

comparing the current color value of the corresponding pixel to the maximum color value of the pixel in the chart;

updating the maximum color value of the pixel in the chart to the current color value of the corresponding pixel, if the current color value of the corresponding pixel is greater than the maximum color value of the pixel in the chart;

comparing the current color value of the corresponding pixel to the minimum color value of the pixel in the chart; and

updating the minimum color value of the pixel in the chart to the current color value of the corresponding pixel, if the current color value of the corresponding pixel is less than the minimum color value of the pixel in the chart.

6. A method as recited in claim 1 wherein creating an alpha image further includes initializing the alpha image.

7. A method as recited in claim 1 wherein creating an alpha image further includes determining whether the absolute value of differences between values corresponding to pixels in the input image that contains the object and values corresponding to pixels in the input image that does not contain the object is greater than predetermined tolerances of pixels in the input image that contains the object.

8. A method as recited in claim 1 wherein transferring shadow information to the alpha image further includes determining similarities between the input image that contains the object and the input image that does not contain the object and adjusting the alpha image accordingly.

9. A method as recited in claim 1 wherein blending the object onto the destination image further includes determining whether portions of the alpha image are set to values greater than zero.

10. A method as recited in claim 1 wherein deriving a set of templates further includes using default templates and altering the default templates based on historical data.

11. A method as recited in claim 1 further comprising transforming the alpha image before compositing the object onto the destination image.

12. A method as recited in claim 1 further including comparing the brightness of the input image that contains the object with the brightness of the input image that does not contain the object.

13. A method as recited in claim 1 further including determining whether the hue of the input image that contains the object is within a predetermined hue tolerance of the input image that does not contain the object.

14. A computer program embodied on a computer-readable medium that composites video objects, comprising:

a code segment that creates a background image model by examining a plurality of frames of an input image that does not contain an object to be composited;

a code segment that obtains a frame of an input image that contains the object to be composited;

a code segment that creates an alpha image according to the input image that contains the object to be composited and the background image model;

a code segment that determines whether a portion of the input image that contains the object to be composited comprises a shadow, wherein the shadow is determined by assigning a color value to a pixel in the background image, assigning a brightness value to the pixel in the background image, comparing the color and brightness values assigned to a corresponding pixel in the input image that contains the object, and identifying the corresponding pixel as being in a shadow where the corresponding pixel color value is substantially unchanged and the corresponding pixel brightness value is substantially less than the brightness value of the pixel in the background image;

a code segment that transfers shadow information to the alpha image;

a code segment that derives a set of templates comprising the shape of the object being composited, wherein the set of templates are derived from the alpha image;

a code segment that fills gaps in the alpha image according to the set of templates; and

a code segment that blends the object in the input image that contains the object onto a destination image using the alpha image, thereby compositing the object onto a destination image with reduced shadows and gaps in the object.

15. A computer-implemented system for digitally compositing an object onto a destination image comprising:

an image modeller for creating a background image model by examining a plurality of frames of an input image that does not contain a object to be composited;

a frame capturer for obtaining a frame of an input image that contains said object to be composited;

an image creator for creating an alpha image according to the input image that contains the object to be composited and the background image model;

a shadow detector for determining whether a portion of the input image that contains the object to be composited comprises a shadow, wherein the shadow is determined by assigning a color value to a pixel in the background image, assigning a brightness value to the pixel in the background image, comparing the color and brightness values assigned to a corresponding pixel in the input image that contains the object, and identifying the corresponding pixel as being in a shadow where the corresponding pixel color value is substantially unchanged and the corresponding pixel brightness value is substantially less than the brightness value of the pixel in the background image;

a template configurator for deriving a set of templates comprising the shape of the object being composited, wherein the set of templates are derived from the alpha image;

an image modifier for transferring shadow information to the alpha image and filling gaps in the alpha image according to the set of templates; and

an image blender for blending the object in the input image that contains the object onto a destination image using the alpha image, thereby compositing the object onto said destination image with reduced shadows and gaps in the object.

16. A system as recited in claim 15 wherein the image modeller further comprises an image initializer for initializing the input image that does not contain the object.

17. A system as recited in claim 15 wherein the image modeller further comprises a pixel manipulator for retrieving minimum values and maximum values of pixels in the input image that does not contain the object thereby setting a first tolerance value for pixels in the input image that does not contain the object to be the difference between the minimum values and the maximum values of the pixels.

18. A system as recited in claim 15 wherein the image modeller further comprises a pixel value configurator for creating a pixel value chart containing minimum values and maximum values of pixels in the background image model.

19. A system as recited in claim 18 further comprising a pixel value updater configured to update the minimum and maximum pixel values in the pixel value chart through a process having:

determining a corresponding pixel in the input image that contains the object;

assigning a current color value of the corresponding pixel;

comparing the current color value of the corresponding pixel to the maximum color value of the pixel in the chart;

updating the maximum color value of the pixel in the chart to the current color value of the corresponding pixel, if the current color value of the corresponding pixel is greater that the maximum color value of the pixel in the chart;

comparing the current color value of the corresponding pixel to the minimum color value of the pixel in the chart; and

updating the minimum color value of the pixel in the chart to the current color value of the corresponding pixel, if the current color value of the corresponding pixel is less than the minimum color value of the pixel in the chart.

20. A system as recited in claim 15 wherein the image creator further comprises an image initializer for initializing the alpha image.

21. A system as recited in claim 15 wherein the image creator further comprises a pixel evaluator for determining whether the absolute value of differences between values corresponding to pixels in the input image that contains the object and values corresponding to pixels in the input image that does not contain the object is greater than predetermined tolerances of pixels in the input image that contains the object.

22. A system as recited in claim 15 wherein the image modifier further comprises an image comparator for determining similarities between the input image that contains the object and the input image that does not contain the object and adjusting the alpha image accordingly.

23. A system as recited in claim 15 wherein the image blender further comprises an alpha image evaluator for determining whether portions of the alpha image are set to values greater than zero.

24. A system as recited in claim 15 wherein the template configurator further comprises a default template creator for creating default templates and altering the default templates based on historical data.

25. A system as recited in claim 15 further comprising an image transformer for transforming the alpha image before compositing the object onto the destination image.

26. A system as recited in claim 15 further comprising an image brightness comparator for comparing the brightness of the input image that contains the object with the brightness of the input image that does not contain the object.

27. A system as recited in claim 15 further comprising an image hue comparator for determining whether the hue of the input image that contains the object is within a predetermined hue tolerance of the input image that does not contain the object.

28. A computer-implemented system for digitally compositing an object onto a destination image, the system comprising:

means for creating a background image model by examning a plurality of frames of an input image that does not contain the object to be composited;

means for inserting the object into said input image that does not contain the object to be composited;

means for obtaining a frame of said input image that contains said object to be composited;

means for creating an alpha image according to the input image that contains the object to be composited and the background image model;

means for determining whether a portion of the input image that contains the object to be composited comprises a shadow, wherein the shadow is determined by assigning a color value to a pixel in the background image, assigning a brightness value to the pixel in the background image, comparing the color and brightness values assigned to a corresponding pixel in the input image that contains the object, and identifying the corresponding pixel as being in a shadow where the corresponding pixel color value is substantially unchanged and the corresponding pixel brightness value is substantially less than the brightness value of the pixel in the background image;

means for transferring shadow information to sad alpha image;

means for deriving a set of templates comprising the shape of the object being composited, wherein the set of templates are derived from the alpha image;

means for filling gaps in said alpha image according to the set of templates; and

means for blending be object in the input image that contains the object onto a destination image using said alpha image, thereby compositing said object onto said destination image with reduced shadows and gaps in said object.

29. A system as recited in claim 28 wherein means for creating a background image model further comprises means for initializing the input image that does not contain the object.

30. A system as recited in claim 28 wherein means for creating a background image model further comprises means for retrieving minimum values and maximum values for pixels in the input image that does not contain the object thereby setting a first tolerance value for pixels in the input image that does not contain the object to be the difference between the minimum values and the maximum values for of the pixels.

31. A system as recited in claim 28 wherein means for creating a background image model further comprises means for creating a pixel value chart containing minimum values and maximum values of pixels in the background image model.

32. A system as recited in claim 31 further comprising means for updating the minimum and maximum pixel values in the pixel value chart through a process having:

determining a corresponding pixel in the input image that contains the object;

assigning a current color value of the corresponding pixel;

comparing the current color value of the corresponding pixel to the maximum color value of the pixel in the chart;

updating the maximum color value of the pixel in the chart tot the current color value of the corresponding pixel, if the current color value of the corresponding pixel is greater than the maximum color value of the pixel in the chart;

comparing the current color value of the corresponding pixel to the maximum color value of the pixel in the chart; and

updating the minimum color value of the pixel in the chart to the current color value of the corresponding pixel, if tie current color value of the corresponding pixel is less than the minimum color value of the pixel in the chart.

33. A system as recited in claim 28 wherein means for creating an alpha image further comprises means for initializing the alpha image.

34. A system as recited in claim 28 wherein means for creating an alpha image further comprises means for determining whether the absolute value of differences between values corresponding to pixels in the input image that contains the object and values corresponding to pixels in the input image that does not contain the object is greater than predetermined tolerances of pixels in the input image that contains the object.

35. A system as recited in claim 28 wherein the means for transferring shadow information to said alpha image further comprises means for determining similarities between the input image that contains the object and the input image that does not contain the object and adjusting the alpha image accordingly.

36. A system as recited in claim 28 wherein means for blending the object onto the destination image further comprises means for determining whether portions of the alpha image are set to values greater than zero.

37. A system as recited in claim 28 wherein means for deriving a set of templates further comprises means for using default templates and altering the default templates based on historical data.

38. A system as recited in claim 28 further comprising means for transforming the alpha image before compositing the object onto the destination image.

39. A system as recited in claim 28 further comprising means for comparing the brightness of the input image that contains the object with the brightness of the input image that does not contain the object.

40. A system as recited in claim 28 further comprising means for determining whether the hue of the input image that contains the object is within a predetermined hue tolerance of the input image that does not contain the object.

41. A computer-implemented digital video object compositing method comprising:

creating a background model by examining a plurality of frames of a background image;

obtaining a frame of an input image that contains an object to be composited;

creating an alpha image such that a value of each pixel in the alpha image is set according to a value corresponding to a pixel from the input image and a value corresponding to a pixel from the background image;

deriving a set of templates such that the set of templates comprises the shape of the object being composited, who the set of templates are derived from the alpha image;

setting the values of pixels in the alpha image according to the set of templates such that portions of the alpha image contained within the templates are adjusted so that the pixels do not represent holes in the object; and

blending the object in the input image onto a destination image using the alpha image as a blending coefficient thereby compositing the object into the destination image with reduced shadows and holes in the object wherein the shadow is determined by assigning a color value to a pixel in the background image, assigning a brightness value to the pixel in the background image, comparing the color and brightness values assigned to a corresponding pixel in the input image that contains the object, and identifying the corresponding pixel as being in a shadow where the corresponding pixel color value is substantially unchanged and the corresponding pixel brightness value is substantially less than the brightness value of the pixel in the background image.

42. An object compositing method of extracting an object from an image and blending the object onto a destination image, the method comprising:

creating a background model by examining a plurality of frames of a background image;

obtaining a frame of an input image that contains an object to be composited;

creating an alpha image such that a value of each pixel in the alpha image is set according to a difference between a corresponding value of a pixel from the input image and a corresponding value of a pixel from the background image;

comparing a brightness value of each pixel in the input image with a brightness value of each corresponding pixel in the background image;

determining whether the hue of the pixel from the input image is within a predetermined hue tolerance of a corresponding pixel from the background image;

calculating a first rank statistic of the pixel from the input image relative to an adjacent pixel from the input image and comparing the first rank statistic with a calculated second rank statistic of the corresponding pixel from the background image to the adjacent pixel from the background image, thereby determining whether the pixel from the input image is part of a shadow; and

blending the object in the input image onto the destination image using the alpha image as a blending coefficient thereby compositing the object into the destination image with