Automatic compositing of 3D objects in a still frame or series of frames

• US 7,477,777 B2
• Filed: 10/28/2005
• Issued: 01/13/2009
• Est. Priority Date: 10/28/2005
• Status: Active Grant

• Current Assignee: IP3 2017, Series 200 of Allied Security Trust I (Allied Security Trust)
• Original Assignee: AEPX Animation Incorporated
• Inventors: Wells, Barton S.
• Primary Examiner(s): KIM, CHONG R
• Application Number:
  US11/262,262
• Publication Number:
  US 20070098290A1
• Time in Patent Office: 1,173 Days
• Field of Search: 345/421, 345/586, 345/632, 345/633
• US Class Current: 382/154
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 7/11 Region-based segmentation

Abstract

Systems, methods, and computer program products that provide broadcast-quality photo-realistic rendering of one or more 3D objects added to a still frame (photo mode), or series of frames (movie mode). The basic steps performed during creation of an image include analysis of a 2D image frame during a set up sequence, adding a 3D object to the frame, and adding visual effects to the frame with little or no user intervention, the visual effects including but not limited to shadowing, reflection, refraction, and transparency.

Claims

1. A method for generating an image, comprising the computer implemented steps of:

analyzing a two dimensional (2D) image for determining several properties thereof;

detecting edges in the 2D image based on the properties of the 2D image;

finding objects in the 2D image based at least in part on the properties of the 2D image;

adding a three dimensional (3D) object to the 2D image;

estimating which objects in the 2D image are positioned in front of the 3D object, or which objects in the 2D image are positioned behind the 3D object;

rendering the following effects;

a reflection of one of the objects in the 2D image on an outer surface of the 3D object a shadow cast by one of the objects in the 2D image on an outer surface of the 3D object, a representation of one of the objects in the 2D image viewable through a transparent or semi-transparent portion of the 3D object, and an effect of refraction on one of the objects in the 2D image viewable through a transparent or semi-transparent portion of the 3D object; and

providing the following features;

slide show and movie output,

batch processing of images,

image labeling,

matching color schemes across multiple images,

color replacement for changing a color of selected portions of an image,

shadow and highlight correction for improving the contrast of over- or underexposed areas of an image

photo touch-up including removal of redeye, dust, scratches, blemishes, wrinkles,

dimensional effects including image wrapping, stretching, curling, and bending,

removing image blurring,

correcting lens distortion,

image noise reduction,

cropping and straightening an image,

painting and drawing tools, and

allowing addition of text and labels.

2. A method as recited in claim 1, wherein the properties are selected from a group consisting of hue, saturation, brightness, red color content, green color content, and blue color content.

3. A method as recited in claim 1, wherein the properties include each of hue, saturation, brightness, red color content, green color content, and blue color content.

4. A method as recited in claim 1, further comprising smoothing the 2D image prior to detecting the edges in the 2D image.

5. A method as recited in claim 1, further comprising refining lines based on the detected edges in the 2D image.

6. A method as recited in claim 1, further comprising detecting shadows in the 2D image based at least in part on the properties of the 2D image.

7. A method as recited in claim 6, further comprising adjusting a boundary of the detected shadows in the 2D image based on a proximity to the detected edges.

8. A method as recited in claim 1, further comprising adjusting the shadow cast by one of the objects in the 2D image on the outer surface of the 3D object to reflect an atmospheric condition in the 2D image.

9. A method as recited in claim 1, further comprising adjusting the outer surface of the 3D object based on an effect of an atmospheric condition in the 2D image.

10. A method as recited in claim 1, wherein the objects in the 2D image are found using a flood fill algorithm.

11. A method as recited in claim 1, wherein the objects in the 2D image are found using a histogram.

12. A method as recited in claim 1, wherein at least some of the objects in the 2D image are found using motion-based detection.

13. A method as recited in claim 1, further comprising storing leveling data, wherein the finding the objects in the 2D image uses the leveling data.

14. A method as recited in claim 1, further comprising performing radiosity processing.

15. A method as recited in claim 1, further comprising performing caustics processing.

16. A method as recited in claim 1, further comprising adding motion blur to the 3D object.

17. A method as recited in claim 1, further comprising performing depth of field focusing.

18. A method as recited in claim 1, wherein the effects are added to the 3D object based at least in part on ray tracing.

19. A method as recited in claim 1, wherein the method is performed for a series of images.

20. A method for generating a motion picture, comprising:

creating a sequence of frames, each frame having a two dimensional (2D) image of a real-life scene;

performing the method of claim 1 for each frame in the sequence for creating a sequence of composite images; and

allowing a user to review the composite images.

21. A method as recited in claim 20, further comprising allowing the user to manually change the composite images.

22. A method for generating a composite image, comprising:

performing the method of claim 1 for creating a composite image, wherein the 3D object represents a physical building construction.

23. A method for generating a medical image, comprising:

performing the method of claim 1 for creating a composite image, wherein the 3D object represents a human anatomical feature.

24. A method for generating a medical image, comprising:

performing the method of claim 1 for creating a composite image, wherein the 3D object represents a medical device.

25. A method as recited in claim 24, wherein the 2D image includes a real-life human anatomical feature.

26. A method as recited in claim 1, wherein the image includes a landscape.

27. A method as recited in claim 1, wherein the image includes an interior of a building structure.

28. A method as recited in claim 1, further comprising allowing a user to apply masks for manipulating the image.

29. A method as recited in claim 28, further comprising allowing the user to mask a 2D object, and attaching a previously unassociated portion of the 2D image with the masked object.

30. A method as recited in claim 1, further comprising:

storing leveling data, wherein the finding the objects in the 2D image uses the leveling data;

performing radiosity processing;

performing caustics processing;

adding motion blur to the 3D object,

wherein at least some of the objects in the 2D image are found using motion-based detection.

31. A method for generating an image, comprising the computer implemented steps of:

adding a three dimensional (3D) object to a two dimensional (2D) image;

rendering a reflection of one of the objects in the 2D image on an outer surface of the 3D object;

rendering a shadow cast by one of the objects in the 2D image on an outer surface of the 3D object;

rendering a representation of one of the objects in the 2D image viewable through a transparent or semi-transparent portion of the 3D object;

rendering an effect of refraction on one of the objects in the 2D image viewable through the transparent or semi-transparent portion of the 3D object; and

storing leveling data, wherein the objects in the 2D image are found using the leveling data;

performing radiosity processing;

performing caustics processing;

adding motion blur to the 3D object, wherein at least some of the objects in the 2D image are found using motion-based detection.

32. A method as recited in claim 31, further comprising wherein the user is allowed to apply the mask after the image is generated and repeating at least some of the steps.

33. A method as recited in claim 31, further comprising allowing the user to mask a partially-found 2D object, and attaching a previously unassociated portion of the 2D image with the masked object.

34. A method for generating a motion picture, comprising:

creating a sequence of frames, each frame having a two dimensional (2D) image of a real-life scene;

performing the method of claim 31 for each frame in the sequence for creating a sequence of composite images; and

allowing a user to review the composite images.

35. A method as recited in claim 34, further comprising allowing the user to manually change the composite images.

36. A method for generating a composite image, comprising:

performing the method of claim 31 for creating a composite image, wherein the 3D object represents a physical building construction.

37. A method for generating a medical image, comprising:

performing the method of claim 31 for creating a composite image, wherein the 3D object represents a human anatomical feature.

38. A method for generating a medical image, comprising:

performing the method of claim 31 for creating a composite image, wherein the 3D object represents a medical device.

39. A method as recited in claim 38, wherein the 2D image includes a real-life human anatomical feature.

40. A method as recited in claim 31, wherein the image includes a landscape.

41. A method as recited in claim 31, wherein the image includes an interior of a building structure.

42. A method as recited in claim 31, further comprising providing the following features:

slide show and movie output,

batch processing of images,

image labeling,

matching color schemes across multiple images,

color replacement for changing a color of selected portions of an image,

shadow and highlight correction for improving the contrast of over- or underexposed areas of an image

photo touch-up including removal of redeye, dust, scratches, blemishes, wrinkles,

dimensional effects including image wrapping, stretching, curling, and bending,

removing image blurring,

correcting lens distortion,

image noise reduction,

cropping and straightening an image,

painting and drawing tools, and

allowing addition of text and labels.

43. A method as recited in claim 31, further comprising allowing a user to apply masks for manipulating the image.

44. A method for generating an image, comprising the computer implemented steps of:

analyzing a two dimensional (2D) image for determining several properties thereof, wherein the properties are selected from a group consisting of hue, saturation, brightness, red color content, green color content, and blue color content;

smoothing the 2D image;

detecting edges in the 2D image based on the properties of the 2D image;

refining lines based on the detected edges in the 2D image;

detecting shadows in the 2D image based at least in part on the properties of the 2D image;

adjusting a boundary of the detected shadows in the 2D image based on a proximity to the detected edges;

finding objects in the 2D image based at least in part on the properties of the 2D image;

adding a three dimensional (3D) object to the 2D image;

estimating which objects in the 2D image are positioned in front of the 3D object, or which objects in the 2D image are positioned behind the 3D object;

rendering the following effects;

a reflection of one of the objects in the 2D image on an outer surface of the 3D object, a shadow cast by one of the objects in the 2D image on an outer surface of the 3D object, a representation of one of the objects in the 2D image viewable through a transparent or semi-transparent portion of the 3D object, and an effect of refraction on one of the objects in the 2D image viewable through a transparent or semi-transparent portion of the 3D object.

45. A method as recited in claim 44, further comprising adjusting the shadow cast by one of the objects in the 2D image on the outer surface of the 3D object to reflect an atmospheric condition in the 2D image.

46. A method as recited in claim 45, further comprising adjusting the outer surface of the 3D object based on an effect of an atmospheric condition in the 2D image.

47. A method as recited in claim 46, further comprising storing leveling data, wherein the finding the objects in the 2D image uses the leveling data.

48. A method as recited in claim 47, further comprising performing radiosity processing, caustics processing, adding motion blur to the 3D object, and performing depth of field focusing.

49. A method as recited in claim 44, wherein the method is performed for a series of images.