System for modifying lighting in photographs

• US 5,894,309 A
• Filed: 02/27/1997
• Issued: 04/13/1999
• Est. Priority Date: 02/27/1997
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Information Technology Center America, Inc. (Mitsubishi Electric Corporation)
• Inventors: Freeman, William T., Beardsley, Paul, Marks, Joseph
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US08/805,209
• Time in Patent Office: 775 Days
• Field of Search: 345/426, 345/425, 345/427, 345/355, 345/356, 345/357
• US Class Current: 345/426
• CPC Class Codes: G06T 15/50 Lighting effects

Abstract

A system for creating lighting for a photograph generates a large number ofmages based upon structures determined from the photograph and randomly selected light positions, types and directions. One or more photographs are analyzed to determine the three-dimensional structures in the photographs. These structures are represented as gray surfaces, to which lighting is applied. The images are culled to provide a set of images which best spans the lighting space represented by the large number of images. The culling process is iterative; at each iteration, the image which is most dissimilar from a nearest neighbor in the selected set is added to the selected set. The images are organized in a hierarchical structure. A user interface allows the user to review and select images in the hierarchical structure. The images selected by the user are combined to create a final image with composite lighting. The lighting specifications can be saved for future reference by the lighting system or other applications. The lighting of the gray surface images are used to modify the intensity of pixels in the photograph in order to change lighting in the photograph.

Claims

1. A system for adding lighting to a photograph comprising:

structure determining means for determining three-dimensional structures in the photograph and representing the structures in an original image;

light generation means for generating a plurality of images with lighting based upon the original image without lighting and data regarding a plurality of lights;

culling means for selecting a set of said plurality of images with lighting;

selection means for selecting at least one of said plurality of images with lighting from said set;

display means for displaying an image formed from said at least one of said plurality of images with lighting; and

photograph modifying means for modifying the intensity of pixels in the photograph based upon at least one of the displayed images.

2. The system for adding lighting to photograph according to claim 1, wherein said light generation means includes:

means for selecting at least one light-hook surface in said image;

means for selecting at least one light type;

position means for randomly selecting a plurality of light positions at a location on said at least one light-hook surface;

type means for selecting one of said at least one light type for each light position selected by said position means; and

image generation means for generating said plurality of images with lighting, each of said images with lighting being based upon said original image, one of said light positions and a corresponding light type.

3. The system for adding lighting to a photograph according to claim 1, wherein said culling means includes:

means for determining an illumination level for each of said plurality of light images with lighting; and

means for deleting light images with lighting which have an illumination level below a predetermined threshold.

4. The system for adding lighting to a photograph according to claim 1, wherein said culling means selects said set so as to best cover a lighting space representing said plurality of images with lighting.

5. The system for adding lighting to a photograph according to claim 4, wherein said culling means includes:

means for selecting a first image for said set from said plurality of images with lighting and removing said first image from said plurality of images with lighting;

difference means for determining a difference for each of said images with lighting, said difference representing a smallest difference between each of said images with lighting and images in said set; and

selection means for selecting one of said images with lighting having a greatest difference for said set and removing said selected image from said plurality of images.

6. The system for adding lighting to a photograph according to claim 1, wherein said selection means includes:

organizing means for organizing said images in a hierarchical structure;

level display means for displaying images in said hierarchical structure based upon user selection of an image at a higher level in said hierarchical structure; and

selection means for selecting one of said images displayed by the level display means based upon a user input.

7. A method for adding lighting to a photograph comprising the steps of:

generating an original image of three-dimensional structures in the photograph represented as gray surfaces;

generating a plurality of images with lighting based upon the original image without lighting and data regarding a plurality of lights;

selecting a set of said plurality of images with lighting;

selecting at least one of said plurality of images with lighting from said set;

displaying an image formed from said at least one of said plurality of images with lighting; and

modifying an intensity of pixels in the photograph based upon said displayed image.

8. The method for adding lighting to a photograph according to claim 7, wherein said generating step generates images having a first resolution level;

the method further comprising before the second selecting step, a step of generating a second set of images with lighting based upon said set of images, said second set of images having a second resolution level which is higher than said first resolution level; and

wherein said second selecting step selects one of said plurality of images from said second set of images.

9. The method for adding lighting to a photograph according to claim 7, wherein said generating step includes the steps of:

randomly selecting a plurality of light positions at locations on at least one light-hook surface;

selecting at least one light type for each light position; and

generating said plurality of images with lighting based upon said original image, one of said light positions, and a corresponding light type.

10. The method for adding lighting to a photograph according to claim 7, wherein said first selection step includes the steps of:

determining an illumination level for each of said plurality of light images with lighting; and

deleting light images with lighting which have an illumination level below a predetermined threshold.

11. The method for adding lighting to a photograph according to claim 7, wherein said first selection step includes the steps of:

selecting a first image for said set from said plurality of images with lighting and removing said first image from said plurality of images with lighting;

determining a difference for each of said images with lighting, said difference representing a smallest difference between each of said images with lighting and images in said set; and

selecting one of said images with lighting having a greatest difference for said set and removing said selected image from said plurality of images; and

repeating said determining and selecting steps until the set includes a predetermined number of images.

12. The method for adding lighting to a photograph according to claim 7, wherein said second selection step includes the steps of:

organizing said images in a hierarchical structure;

displaying images in said hierarchical structure based upon user selection of an image at a higher level in said hierarchical structure; and

selecting one of said images displayed by the level display means based upon a user input.

13. The method for adding lighting to a photograph according to claim 12, wherein said organizing step includes:

determining a first subset of images from said set of images which are most distinctive from each other;

determining a plurality of second subsets of images from said set of images, each of said second subsets being most similar to one of said images in said first subset; and

determining a plurality of third subsets of images from said set of images, each of said third subsets being most similar to one of said images in said plurality of second subsets.