Performance relighting and reflectance transformation with time-multiplexed illumination

• US 7,436,403 B2
• Filed: 06/10/2005
• Issued: 10/14/2008
• Est. Priority Date: 06/12/2004
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Debevec, Paul E.
• Primary Examiner(s): Chauhan; Ulka
• Assistant Examiner(s): REPKO, JASON MICHAEL
• Application Number:
  US11/150,012
• Publication Number:
  US 20050276441A1
• Time in Patent Office: 1,222 Days
• Field of Search: 345/960
• US Class Current: 345/426
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: Y10S 345/96 Iterative display of preconfigured images

Abstract

A lighting apparatus may be configured to illuminate a subject while the subject is undergoing a motion during a time period. An imaging system may be configured to generate image data representative of a sequence of frames of the moving subject. A controller may be configured to drive the lighting apparatus and the imaging system so that the lighting apparatus sequentially illuminates the moving subject with a time-multiplexed series of lighting conditions, and so that each one of the frames shows the subject illuminated with a respective one of the lighting conditions. The controller may be further configured to process the image data to generate re-illumination data representative of novel illumination conditions under which the subject can be re-illuminated, subsequent to the time period.

Claims

1. An apparatus comprising:

a lighting apparatus including a plurality of light sources, each positioned at different locations relative to each other, each configured to illuminate a subject from a different direction relative to each other, the plurality of light sources configured to provide one or more illumination bases;

wherein the lighting apparatus is configured to sequentially illuminate a subject with a time-multiplexed series of lighting conditions while the subject is undergoing a motion during a time period;

an imaging system configured to generate image data representative of a sequence of frames of the moving subject, each one of the frames showing the subject illuminated with a respective one of the lighting conditions;

a processing system configured to process the image data to generate re-illuminated data representative of the subject, undergoing the motion, re-illuminated under novel illumination conditions, subsequent to the time period; and

a controller configured to drive the plurality of light sources synchronously with the imaging system and in accordance with a desired illumination basis pattern, so as to generate the time-multiplexed series of lighting conditions;

wherein the controller is further configured to drive the imaging system so as to include within the sequence of frames a time-multiplexed matte frame, at desired intervals within a succession of basis frames, so as to allow the subject'"'"'s motion to be composited over a new background subsequent to the time period.

2. The apparatus of claim 1, wherein the processing system is further configured to compensate for the motion of the subject by estimating optical flow between one or more of the sequence of frames.

3. The apparatus of claim 2, wherein the processing system is further configured to use the estimated optical flow to warp the image data so as to align each one of the frames with a target output frame.

4. The apparatus of claim 2, wherein the controller is further configured to drive the lighting apparatus and the imaging system so that one or more similarly illuminated tracking frames are generated, and so that the tracking frames are included at desired intervals within the sequence of frames.

5. The apparatus of claim 4, wherein the processing system is further configured to calculate an optical flow between successive tracking frames, to interpolate local optical flow fields across basis frames interspersed between the tracking frames, and to warp each one of the basis frames into alignment with a nearest tracking frame in the direction of the target output frame using the interpolated local optical flow fields.

6. The apparatus of claim 5, wherein the apparatus and controller are further configured to illuminate the subject with one or more spatially structured illumination patterns, and wherein the processing system is further configured to apply varying novel illumination conditions to one or more parts of the image, taking into account the position of said one or more parts within the spatially structured illumination patterns.

7. The apparatus of claim 2, wherein the processing system is further configured to synthesize and apply a motion blur based on the estimated optical flow field.

8. The apparatus of claim 1, wherein the processing system is further configured to process the image data to modify one or more reflectance functions for a surface of the subject, so as to transform a reflectance of the surface of the subject undergoing the motion.

9. The apparatus of claim 8,

wherein each frame comprises a plurality of pixels, each pixel having an associated pixel value that represents the image brightness of a corresponding surface unit area on the surface of the subject;

and

wherein the processing system is further configured to transform the reflectance of the surface of the subject by generating at least one of;

a surface normal map that provides an estimate of a surface normal value for each pixel;

a diffuse albedo map that provides an estimate of a diffuse albedo value for each pixel;

a specular reflectance map that provides an estimate of a specular reflectance value for each pixel; and

an ambient occlusion map that provides an estimate of a geometric self-shadowing for each pixel.

10. The apparatus of claim 1, wherein the processing system if further configured to generate the re-illuminated data by estimating reflectance properties of the subject from image data representative of one or more of the frames, and applying novel lighting to estimated reflectance properties of the subject.

11. The apparatus of claim 1, wherein the processing system is further configured to generate the re-illumination data by recombining image data representative of one or more of the frames.

12. The apparatus of claim 1, wherein each of the illumination bases comprises at least one of:

an individual light source;

a triplet of triangularly adjacent light sources;

a cluster of light sources;

a Hadamard matrix of light sources; and

a set of gray-level intensities corresponding to an element of the spherical harmonic basis.

13. The apparatus of claim 1, wherein the imaging system comprises a high-speed camera, and wherein at least some of the plurality of light sources comprises LEDs (Light Emitting Diodes).

14. The apparatus of claim 1, further comprising a matting background positionable behind the subject and configured to be illuminated by one or more light sources so as to yield, when imaged by the imaging system, an image of the subject against the illuminated background.

15. An apparatus comprising:

a lighting apparatus including a plurality of light sources, each positioned at different locations relative to each other, each configured to illuminate a subject from a different direction relative to each other, the plurality of light sources configured to provide one or more illumination bases;

wherein the lighting apparatus is configured to illuminate a subject with a time-multiplexed series of lighting conditions while the subject is undergoing a motion during a time period;

a matting background positionable behind the subject and configured to be illuminated so as to yield, when imaged, an image of the subject against the illuminated background;

a video camera configured to generate image data representative of a sequence of frames of the subject undergoing the motion, and further configured to generate matte image data representative of time-multiplexed matte frames;

a controller configured to drive the plurality of light sources synchronously with the imaging system and in accordance with a desired illumination basis pattern, so as to generate the time-multiplexed series of lighting conditions; and

a processing system configured to process the matte image data so as to composite the motion of the subject into a new background.

16. An apparatus comprising:

a lighting apparatus including a plurality of light sources, each positioned at different locations relative to each other, each configured to illuminate a subject from a different direction relative to each other, the plurality of light sources configured to provide one or more illumination bases;

wherein the lighting apparatus is configured to sequentially illuminate a subject with a time-multiplexed series of lighting conditions while the subject is undergoing a motion during a time period;

a plurality of imaging systems, each of the imaging systems configured to substantially simultaneously generate from a corresponding plurality of viewpoints image data representative of a sequence of frames of the moving subject, each of the frames showing the subject illuminated with a respective one of the lighting conditions;

a controller configured to drive the plurality of light sources synchronously with one or more of the imaging systems and in accordance with a desired illumination basis pattern, so as to generate the time-multiplexed series of lighting conditions; and

a processing system configured to process the image data to generate re-illumination data representative of novel illumination conditions under which the subject, undergoing the motion, can be re-illuminated, from one or more of the plurality of viewpoints, subsequent to the time period.