ILLUMINATION ESTIMATION FROM A SINGLE IMAGE

• US 20180260975A1
• Filed: 03/13/2017
• Published: 09/13/2018
• Est. Priority Date: 03/13/2017
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: SUNKAVALLI, KALYAN K., SHEN, XIAOHUI, YUMER, MEHMET ERSIN, GARDNER, MARC-ANDRE, GAMBARETTO, EMILIANO
• Granted Patent: US 10,607,329 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/00 Image analysis

Abstract

Methods and systems are provided for using a single image of an indoor scene to estimate illumination of an environment that includes the portion captured in the image. A neural network system may be trained to estimate illumination by generating recovery light masks indicating a probability of each pixel within the larger environment being a light source. Additionally, low-frequency RGB images may be generated that indicating low-frequency information for the environment. The neural network system may be trained using training input images that are extracted from known panoramic images. Once trained, the neural network system infers plausible illumination information from a single image to realistically illumination images and objects being manipulated in graphics applications, such as with image compositing, modeling, and reconstruction.

Claims

1. A computer-implemented method for training a neural network system to estimate illumination of images, the method comprising:

receiving training panoramic images;

extracting training patches from the training panoramic images, each training patch being a portion of a training panoramic image;

generating training recovery light masks for the training patches using a neural network system, each training recovery light mask indicating a probability of each pixel of a corresponding training panoramic image being a light source;

based on comparisons of training light recovery masks to reference masks, training the neural network system to synthesize light recovery masks for input images.

2. The computer-implemented method of claim 1, wherein the reference masks are binary light masks generated from the training panoramic images.

3. The computer-implemented method of claim 2, wherein the binary light masks are generated by one or more linear support vector machine classifiers trained to classify each pixel within the training panoramic images as a light source or not a light source.

4. The computer-implemented method of claim 3, wherein the one or more linear support vector machine classifiers comprise a first classifier for small light sources and a second classifier for large light sources.

5. The computer-implemented method of claim 3, wherein, upon generation of the binary light masks by the one or more linear support vector machine classifiers, the binary light masks are refined using a conditional random field to keep only semantically-related pixels in areas detected as being a light source.

6. The computer-implemented method of claim 1, wherein training the neural network system comprises adjusting a neural network within the neural network system using a loss function.

7. The computer-implemented method of claim 1 further comprising generating training RGB images for the training patches using the neural network system, each training RGB image indicating low-frequency information for a corresponding training panoramic image.

8. The computer-implemented method of claim 7 further comprising generating an intermediate representation for each training patch, the training light recovery masks and the training RGB images being generated from the intermediate representations.

9. One or more computer-readable media having a plurality of executable instructions embodied thereon, which, when executed by one or more processors, cause the one or more processors to perform a method for estimating illumination of images, the method comprising:

receiving an input image of an indoor scene, wherein the input image depicts a view of the indoor scene that is less than 360 degrees; and

utilizing a trained neural network system to generate, from the input image, a recovery light mask for a panoramic environment encompassing the view depicted in the input image, wherein the recovery light mask indicates a probability of each pixel within the panoramic environment being a light source.

10. The media of claim 9, wherein the panoramic environment represents a 360-degree view.

11. The media of claim 9, wherein the method further comprises utilizing the trained neural network system to generate, from the input image, a low-frequency RGB image indicating low-frequency information for the panoramic environment.

12. The media of claim 11, the method further comprising encoding the input image into an intermediate representation that is utilized to generate the recovery light mask and the low-frequency RGB image.

13. The media of claim 11, wherein the low-frequency information indicated by the low-frequency RGB image comprises a coarse estimation of color distribution within the panoramic environment.

14. The media of claim 11, wherein the trained neural network system is trained using a training method comprising:

training a first neural network to generate recovery light masks based on comparisons of binary light masks and training recovery light masks generated by the first neural network from training images; and

training a second neural network to generate low-frequency RGB images based on comparisons of training panoramic images and training low-frequency RGB images generated by the second neural network from the training images, each training image being a portion of a training panoramic image.

15. The media of claim 14, wherein the binary light masks are generated using one or more linear support vector machine classifiers, each binary light mask indicating whether each pixel within a training panoramic image is classified as a light source.

16. A computing system comprising:

one or more processors; and

one or more non-transitory computer-readable storage media, coupled with the one or more processors, having instructions stored thereon, which, when executed by the one or more processors, cause the computing system to provide;

means for training a neural network system, wherein the neural network system includes a neural network trained to estimate illumination of images including illumination provided by light sources that are not visible in the images; and

means for generating a recovery light mask indicating a probability that each pixel within a panoramic environment encompassing an input image is a light source.

17. The computing system of claim 16, wherein the neural network is trained based on a comparison of a binary light mask and a training recovery light mask generated by the neural network from a training image.

18. The computing system of claim 17, wherein the binary light mask is generated using a linear support vector machine classifier.

19. The computing system of claim 16, wherein the computing system is further caused to provide:

means for generating a low-frequency RGB image indicating low-frequency information for the panoramic environment encompassing the input image.

20. The computing system of claim 19, wherein the recovery light mask and the low-frequency RGB image are generated by encoding the input image into an intermediate representation and creating the recovery light mask and the low-frequency RGB image from the intermediate representation.