Illumination estimation from a single image

• US 10,607,329 B2
• Filed: 03/13/2017
• Issued: 03/31/2020
• Est. Priority Date: 03/13/2017
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Sunkavalli, Kalyan K., Shen, Xiaohui, Yumer, Mehmet Ersin, Gardner, Marc-Andre, Gambaretto, Emiliano
• Primary Examiner(s): Bezuayehu, Solomon G
• Application Number:
  US15/457,192
• Publication Number:
  US 20180260975A1
• Time in Patent Office: 1,114 Days
• Field of Search: 382156
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/00 Image analysis

Abstract

Methods and systems are provided for using a single image of an indoor scene to estimate illumination of an environment that includes the portion captured in the image. A neural network system may be trained to estimate illumination by generating recovery light masks indicating a probability of each pixel within the larger environment being a light source. Additionally, low-frequency RGB images may be generated that indicating low-frequency information for the environment. The neural network system may be trained using training input images that are extracted from known panoramic images. Once trained, the neural network system infers plausible illumination information from a single image to realistically illumination images and objects being manipulated in graphics applications, such as with image compositing, modeling, and reconstruction.

Claims

1. One or more non-transitory computer-readable media having a plurality of executable instructions embodied thereon, which, when executed by one or more processors, cause the one or more processors to perform a method for estimating illumination of images, the method comprising:

receiving an input image of an indoor scene, wherein the input image depicts a view of the indoor scene that is less than 360 degrees;

encoding the input image to an intermediate representation; and

utilizing a trained neural network system to generate, from the intermediate representation, a recovery light mask and low-frequency RGB image each representing a panoramic environment encompassing the view depicted in the input image, wherein utilizing the trained neural network system includes;

generating the recovery light mask utilizing a first branch of the neural network system, wherein the recovery light mask indicates a probability of each pixel representing an area within the panoramic environment being a light source, wherein at least some pixels in the recovery light mask represent part of the panoramic environment that is beyond the view depicted in the input image, and

generating the low-frequency RGB image indicating low-frequency information utilizing a second branch of the neural network system.

2. The media of claim 1, wherein the panoramic environment represents a 360-degree view.

3. The media of claim 1, wherein the low-frequency information indicated by the low-frequency RGB image comprises a coarse estimation of color distribution within the panoramic environment.

4. The media of claim 1, wherein the method further comprises using a training method comprising:

training a first neural network to generate recovery light masks based on comparisons of binary light masks and training recovery light masks generated by the first neural network from training images, the first neural network forming the first branch of the neural network system; and

training a second neural network to generate low-frequency RGB images based on comparisons of training panoramic images and training low-frequency RGB images generated by the second neural network from the training images, each training image being a portion of a training panoramic image, the second neural network forming the second branch of the neural network system.

5. The media of claim 4, wherein the method further comprises generating the binary light masks using one or more linear support vector machine classifiers, each binary light mask indicating whether each pixel within a training panoramic image is classified as a light source.

6. The media of claim 5, wherein the one or more linear support vector machine classifiers comprise a first classifier for small light sources and a second classifier for large light sources.

7. The media of claim 5, wherein the method further comprises refining the binary light masks using conditional random field.

8. A computer-implemented method for estimating illumination of images, the method comprising:

receiving an input image of an indoor scene, wherein the input image depicts a view of the indoor scene that is less than 360 degrees;

encoding the input image to an intermediate representation; and

utilizing a trained neural network system to generate, from the intermediate representation, a recovery light mask and low-frequency RGB image each representing a panoramic environment encompassing the view depicted in the input image, wherein utilizing the trained neural network system includes;

generating the recovery light mask utilizing a first branch of the neural network system, wherein the recovery light mask indicates a probability of each pixel representing an area within the panoramic environment being a light source, and wherein at least some pixels in the recovery light mask represent part of the panoramic environment that is beyond the view depicted in the input image; and

generating the low-frequency RGB image indicating low-frequency information utilizing a second branch of the neural network system.

9. The computerized method of claim 8, wherein the low-frequency information indicated by the low-frequency RGB image comprises a coarse estimation of color distribution within the panoramic environment.

10. The computerized method of claim 8, wherein the input image depicts a 60-degree view and the panoramic environment represents by the recovery light mask is a 360-degree view.

11. The computerized method of claim 8, wherein pixels within the recovery light mask that have different probabilities representing a light source are assigned different colors.