Performance relighting and reflectance transformation with time-multiplexed illumination

• US 20050276441A1
• Filed: 06/10/2005
• Published: 12/15/2005
• Est. Priority Date: 06/12/2004
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Debevec, Paul E.
• Granted Patent: US 7,436,403 B2
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: Y10S 345/96 Iterative display of preconfigured images

Abstract

A lighting apparatus may be configured to illuminate a subject while the subject is undergoing a motion during a time period. An imaging system may be configured to generate image data representative of a sequence of frames of the moving subject. A controller may be configured to drive the lighting apparatus and the imaging system so that the lighting apparatus sequentially illuminates the moving subject with a time-multiplexed series of lighting conditions, and so that each one of the frames shows the subject illuminated with a respective one of the lighting conditions. The controller may be further configured to process the image data to generate re-illumination data representative of novel illumination conditions under which the subject can be re-illuminated, subsequent to the time period.

Claims

1. An apparatus comprising:

a lighting apparatus configured to sequentially illuminate a subject with a time-multiplexed series of lighting conditions while the subject is undergoing a motion during a time period;

an imaging system configured to generate image data representative of a sequence of frames of the moving subject, each one of the frames showing the subject illuminated with a respective one of the lighting conditions; and

a processing system configured to process the image data to generate re-illuminated data representative of the subject re-illuminated under novel illumination conditions, subsequent to the time period.

2. The apparatus of claim 1, wherein the processing system is further configured to generate the re-illuminated data by recombining image data representative of one or more of the frames.

3. The apparatus of claim 1,

wherein the lighting apparatus comprises a plurality of light sources configured to provide one or more illumination bases;

and

further comprising a controller configured to drive the plurality of light sources synchronously with the imaging system and in accordance with a desired illumination basis pattern, so as to generate the time-multiplexed series of lighting conditions.

4. The apparatus of claim 3, wherein each of the illumination bases comprises at least one of:

an individual light source;

a triplet of triangularly adjacent light sources;

a cluster of light sources;

a set of linearly independent light sources;

a Hadamard matrix of light sources; and

a set of gray-level intensities corresponding to an element of the spherical harmonic basis.

5. The apparatus of claim 1, wherein the imaging system comprises a high-speed camera, and wherein at least some of the plurality of light sources comprises LEDs (Light Emitting Diodes).

6. The apparatus of claim 3, further comprising a matting background positionable behind the subject and configured to be illuminated by one or more light sources so as to yield, when imaged by the imaging system, an image of the subject against the illuminated background.

7. The apparatus of claim 6, wherein the controller is further configured to drive the imaging system so as to include within the sequence of frames one or more time-multiplexed matte frames, at desired intervals within a succession of basis frames, so as to allow the subject'"'"'s motion to be composited over a new background subsequent to the time period.

8. The apparatus of claim 3, wherein the processing system is further configured to compensate for the motion of the subject by estimating optical flow between one or more of the sequence of frames.

9. The apparatus of claim 8, wherein the processing system is further configured to use the estimated optical flow to warp the image data so as to align each one of the frames with a target output frame.

10. The apparatus of claim 8, wherein the controller is further configured to drive the lighting apparatus and the imaging system so that one or more similarly illuminated tracking frames are generated, and so that the tracking frames are included within desired intervals within the sequence of frames.

11. The apparatus of claim 10, wherein the processing system is further configured to calculate one or more optical flow fields between one or more tracking frames, to interpolate the optical flow fields across basis frames interspersed amongst the tracking frames, and to warp each one of the basis frames substantially into alignment with a target output frame, using the optical flow fields.

12. The apparatus of claim 11, wherein the apparatus and controller are further configured to illuminate the subject with one or more spatially structured illumination patterns, and wherein the processing system is further configured to apply varying novel illumination conditions to one or more parts of the subject, taking into account the position of said one or more parts within the spatially structured illumination patterns.

13. The apparatus of claim 6, wherein the processing system is further configured to synthesize and apply a motion blur based on an optical flow field.

14. The apparatus of claim 1, wherein the processing system is further configured to process the image data to modify one or more reflectance functions for a surface of the subject, so as to transform a reflectance of the surface of the subject undergoing the motion.

15. The apparatus of claim 14,

wherein each frame comprises a plurality of pixels, each pixel having an associated pixel value that represents the image brightness of a corresponding surface unit area on the surface of the subject;

and

wherein the processing system is further configured to transform the reflectance of the surface of the subject by generating at least one of;

a surface normal map that provides an estimate of a surface normal value for each pixel;

a diffuse albedo map that provides an estimate of a diffuse albedo value for each pixel;

a specular reflectance map that provides an estimate of a specular reflectance value for each pixel; and

an ambient occlusion map that provides an estimate of a geometric self-shadowing for each pixel.

24. The apparatus of claim 1, wherein the processing system is further configured to generate the re-illuminated data by estimating reflectance properties of the subject from image data representative of one or more of the frames, and applying novel lighting to estimated reflectance properties of the subject.

16. An apparatus comprising:

means for sequentially illuminating a subject with a time-multiplexed series of rapidly changing lighting conditions while the subject is undergoing a motion during a time period;

means for generating image data representative of a sequence of frames of the subject undergoing the motion, each one of the frames showing the subject illuminated with a respective one of the lighting conditions; and

processing means for processing the image data to generate re-illuminated data representative of the subject under novel illumination conditions, subsequent to the time period.

17. The apparatus of claim 16, wherein the sequence of frames comprise matte frames of the subject, and further comprising additional processing means for compositing the motion of the subject into a new background using the matte frames.

18. An apparatus comprising:

a lighting apparatus configured to illuminate a subject while the subject is undergoing a motion during a time period;

a matting background positionable behind the subject and configured to be time-varyingly illuminated so as to yield, when imaged, an image of the subject against a time-varyingly illuminated background;

a video camera configured to generate image data representative of a sequence of frames of the subject undergoing the motion, and further configured to generate matte image data representative of time-multiplexed matte frames; and

a processing system configured to process the matte image data so as to composite the motion of the subject into a new background.

19. A method comprising:

sequentially illuminating a subject with a time-multiplexed series of lighting conditions while the subject is undergoing a motion;

generating image data representative of a sequence of frames of the moving subject, wherein each one of the frames shows the subject illuminated with a respective one of the lighting conditions; and

processing the image data to generate re-illumination data representative of novel illumination conditions under which the subject can be re-illuminated, subsequent to the time period.

20. The method of claim 19, wherein the act of generating the image data comprises generating matte image data representative of time-multiplexed matte frames, and wherein the act of processing the image data comprises processing the matte image data so as to composite the motion of the subject into a new background.

21. The method of claim 19, further comprising the act of compensating for the motion of the subject by estimating optical flow between one or more of the sequence of frames, and using the estimated optical flow to warp the image data so as to align each one of the frames with a target output frame.

22. The method of claim 19, further comprising the act of transforming a reflectance of a surface of the subject by processing the image data to modify one of more reflectance functions for the surface of the subject.

23. An apparatus comprising:

a lighting apparatus configured to sequentially illuminate a subject with a time-multiplexed series of lighting conditions while the subject is undergoing a motion during a time period;

a plurality of imaging systems, each of the imaging systems configured to substantially simultaneously generate from a corresponding plurality of viewpoints image data representative of a sequence of frames of the moving subject, each of the frames showing the subject illuminated with a respective one of the lighting conditions; and

a processing system configured to process the image data to generate re-illumination data representative of novel illumination conditions under which the subject can be re-illuminated, from one or more of a plurality of viewpoints, subsequent to the time period.