Stylized rendering using a multi-flash camera

• US 7,738,725 B2
• Filed: 05/17/2004
• Issued: 06/15/2010
• Est. Priority Date: 03/19/2003
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Feris, Rogerio, Raskar, Ramesh
• Primary Examiner(s): Hung; Yubin
• Application Number:
  US10/847,068
• Publication Number:
  US 20040212725A1
• Time in Patent Office: 2,220 Days
• Field of Search: 382/190, 382/199, 382/224, 382/254, 382/256, 382/284, 382/312, 250/559.36, 345/419, 345/582, 348/207.1, 348/207.2, 348/239, 348/371, 348/578, 348/584, 356/613
• US Class Current: 382/266
• CPC Class Codes: G06T 15/02 Non-photorealistic rendering
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2625 for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect sequence generated by event triggered capturing H04N7/188

Abstract

A method generates a stylized image of a scene including an object. A set of n input images are acquired of the scene with a camera. Each one of the n input images is illuminated by one of a set of n light sources mounted on a body of the camera at different positions from a center of projection of a lens of the camera. Ambient lighting can be used to illuminate one image. Features in the set of n input images are detected. The features include depth edges, intensity edges, and texture edges to determine qualitative depth relationships between the depth edges, the intensity edges and the texture edges. The set of n input images are then combined in an output image to enhance the detected features according to the qualitative relationships.

Claims

1. A method for generating a stylized image of a scene including an object, comprising the steps of:

acquiring a set of n input images I_k, k=1 . . . n, of the scene with a camera, each one of the n input images being illuminated by one of a set of n light sources mounted on a body of the camera at different positions from a center of projection of a lens of the camera;

determining for each of the n input images I_k an intensity gradient G_k(x,y) at each pixel position (x,y), a median of the n determined intensity gradients G(x,y)=median_k(G_k(x, y)) to remove outliers, and a reconstructed image I′

that minimizes |H′

−

G|, wherein H′

is the gradient image of I′

;

removing specular highlights from the n input images;

detecting features in the set of n input images, the features including silhouette edges;

combining the set of n input images in which the detected silhouette edges are enhanced in an output image; and

wherein the acquiring, determining, removing, detecting and combining steps are performed using a computer.

2. The method of claim 1, in which the silhouette edges correspond to a depth discontinuities in the n input images.

3. The method of claim 1, in which the silhouette edges are signed, wherein foreground pixels in the n input images have positive values, and background pixels have negative values, and the texture edges delineate reflectance and material discontinuities in the scene.

4. The method of claim 3, in which a foreground side of the silhouette edges have the positive values, and a background side of the silhouette edges have the negative values, and further comprising:

rendering the foreground side as white; and

rendering the background side as black.

5. The method of claim 4, further comprising:

rendering the foreground side and the background side of the silhouette edges by displacing the contour along a surface normal.

6. The method of claim 5, further comprising:

modifying a width of the silhouette edges to convey an arbitrary light direction in the output image.

7. The method of claim 6, in which the width is proportional to a dot product of an image space normal and the arbitrary light direction.

8. The method of claim 1, in which the silhouette edges correspond to shadow regions in the scene in terms of coordinates of an image space.

9. The method of claim 8, in which the object in the scene and the shadow regions are contiguous in the image space.

10. The method of claim 8, further comprising:

generating, for each of the n input image, a ratio image from pixel intensities in the shadow regions that are substantially close to zero;

performing intensity edge detection on each ratio image along epipolar rays marking negative step edges at the silhouette edges to generate a depth edge map for each ratio image; and

combining the edge maps from all n input images to obtain a final depth edge map.

11. The method of claim 1, in which the set of n light sources are mounted on the body of the camera mounted at different distances from the center of projection of a lens of the camera to provide a plurality of baselines.

12. The method of claim 1, in which a ratio I_k(x,y)/I′

(x,y) is set to 1.0 to remove the specular highlights if the ratio is greater than 1.0.

13. The method of claim 1, further comprising:

connecting the silhouette edge pixels into a contour; and

smoothing the contour.

14. The method of claim 1, further comprising:

rendering the silhouette edges in a selected color.

15. The method of claim 14, in which one input image is illuminated by ambient lighting, and further comprising:

segmenting the one input image illuminated by ambient lighting; and

superimposing the colored silhouette edges on the segmented input image.

16. The method of claim 1, further comprising:

rendering the silhouette edges in white on a black background in the output image;

convolving the silhouette edges with a filter that is gradient of an edge enhancement filter to obtain a convolved image; and

integrating the convolved image to obtain sharp transition at the silhouette edges.

17. The method of claim 16, in which the enhancement filter is a Gaussian function minus an impulse function.

18. The method of claim 1, in which the scene includes a background object and a foreground object, and further comprising:

acquiring a first set of n input images of the scene with only the background object;

acquiring a second set of n input images of the scene with the background object and the foreground object;

applying a mask to portions of the second set of n images corresponding to the foreground object to enhance an appearance of the foreground object in the output image.

19. The method of claim 18, in which pixel intensity values in the mask are set to one.

20. The method of claim 1, in which the texture edges define texture regions in the n images, and further comprising:

masking the texture regions according to a mask image to reduce clutter in the output image.

21. The method of claim 1, in which the camera is a video camera, and the object is moving in the scene, and further comprising:

acquiring a sequence of sets of n input images of the scene with the video camera;

detecting the features in the sequence of sets of n input images; and

combining each of the sequence of sets of n input images in which the detected silhouette edges are enhanced in an output image to obtain a sequence of output images.

22. The method of claim 21, further comprising:

identifying intensity gradients in the sequence of sets of images, the intensity gradients including gradients due to the silhouette edges, gradients due to the texture edges of static parts, and gradients due to texture edges of moving parts; and

removing the texture edges having static parts.

23. The method of claim 22, further comprising:

interpolating displacements of moving depth edges in the sequence of output images.

26. The method of claim 1, in which the light sources emit visible light.

27. The method of claim 1, in which the light sources emit infrared light.

28. The method of claim 1, in which the light sources emit sonar signal.

29. The method of claim 1, in which the light sources emit x-rays.

30. The method of claim 1, in which the light sources emit radar signals.

31. The method of claim 1, in which the light sources simultaneously emit light at a plurality of wavelengths, and further comprising:

decoding the plurality of wavelengths at sensor in the camera using a plurality of filters.

32. The method of claim 31, in which the filters have Bayer mosaic patterns.

33. The method of claim 1, further comprising:

determining a visual hull from the edges.

34. The method of claim 1, further comprising:

segmenting the output image according to the edges.

35. The method of claim 1, further comprising:

resolving depth layers in the output image according to the edges.

36. The method of claim 1, further comprising:

resolving aspect graphs of the output image according to the edges.

37. The method of claim 1, in which the input images are acquired by aerial imaging photography.

38. The method of claim 1, in which the light sources are flash units.

24. The method of 21, further comprising:

assigning colors of pixels in a particular image of the sequence of output images according to a maximum of three successive images.

25. The method of claim 24, in which the three successive images is the particular image, a previous image and a next image in the sequence.