Adversarial learning of photorealistic post-processing of simulation with privileged information

• US 10,643,320 B2
• Filed: 02/12/2018
• Issued: 05/05/2020
• Est. Priority Date: 11/15/2017
• Status: Active Grant

• Current Assignee: Toyota Research Institute, Inc. (Toyota Motor Corporation)
• Original Assignee: Toyota Research Institute, Inc. (Toyota Motor Corporation)
• Inventors: Lee, Kuan-Hui, Ros, German, Gaidon, Adrien D., Li, Jie
• Primary Examiner(s): Moyer, Andrew M
• Application Number:
  US15/893,864
• Publication Number:
  US 20190147582A1
• Time in Patent Office: 813 Days
• Field of Search: 382156
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/149 involving deformable models, e.g. active contour models
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling

Abstract

Systems and method for generating photorealistic images include training a generative adversarial network (GAN) model by jointly learning a first generator, a first discriminator, and a set of predictors through an iterative process of optimizing a minimax objective. The first discriminator learns to determine a synthetic-to-real image from a real image. The first generator learns to generate the synthetic-to-real image from a synthetic image such that the first discriminator determines the synthetic-to-real image is real. The set of predictors learn to predict at least one of a semantic segmentation labeled data and a privileged information from the synthetic-to-real image based on at least one of a known semantic segmentation labeled data and a known privileged information corresponding to the synthetic image. Once trained, the GAN model may generate one or more photorealistic images using the trained GAN model.

Claims

1. A method for generating photorealistic images comprising:

training a generative adversarial network (GAN) model by jointly learning a first generator, a second generator, a first discriminator, a second discriminator, and a set of predictors through an iterative process of optimizing a minimax objective wherein;

the first discriminator learns to determine a synthetic-to-real image from a real image,

the first generator learns to generate the synthetic-to-real image from a synthetic image such that the first discriminator determines the synthetic-to-real image is real,

the second generator learns to generate a real-to-synthetic image from the real image such that the second discriminator determines the real-to-synthetic image is fake,

the second discriminator learns to determine the real-to-synthetic image from the synthetic image such that differences between the real-to-synthetic image and the synthetic image are minimized, and

the set of predictors learn to predict at least one of a semantic segmentation labeled data and a privileged information from the synthetic-to-real image based on at least one of a known semantic segmentation labeled data and a known privileged information corresponding to the synthetic image; and

generating one or more photorealistic images using the trained GAN model.

2. The method of claim 1, further comprising:

receiving, from a simulator, the synthetic image; and

receiving, from the simulator, the known semantic segmentation labeled data indicating semantic segmentation within the synthetic image and the known privileged information corresponding to the synthetic image.

3. The method of claim 1, wherein optimizing the minimax objective comprises:

updating one or more parameters of the first discriminator during a first iteration of training the first generator, the first discriminator, and the set of predictors; and

updating one or more parameters of the first generator and the set of predictors during a second iteration of training the first generator, the first discriminator, and the set of predictors.

4. The method of claim 1, wherein the minimax objective is characterized by a set of loss functions comprising an adversarial loss (_GAN), a task prediction loss (_task), a privileged information regularization (_pi), and a perceptual regularization (_perc):

min<sub>θ

G,θ

T,θ

P</sub> max<sub>θ

D</sub>α

_GAN+β

_T+γ

_P+δ

_perc

wherein α

, β

, γ

, δ

are weighting parameters and θ

_G is a generator parameter, θ

_D is a discriminator parameter, θ

_T is a task predictor parameter, and θ

_pk is a privileged information predictor parameter.

5. The method of claim 1, wherein the set of predictors comprises at least one of a task network and a privileged information network.

6. The method of claim 5, wherein the task network learns to predict the semantic segmentation labeled data from the synthetic-to-real image.

7. The method of claim 5, wherein the privileged information network learns to predict at least one of depth information, optical flow information, object detection information, and instance segmentation information from the synthetic-to-real image.

8. The method of claim 1, wherein the first generator learns a pixel-level mapping function to generate the synthetic-to-real image from the synthetic image.

9. A computer-implemented system for generating photorealistic images from a synthetic image, the computer-implemented system comprising:

a processor and a non-transitory computer-readable medium storing computer readable instructions that, when executed by the processor, cause the processor to;

train a generative adversarial network (GAN) model comprising a first generator, a second generator, a first discriminator, a second discriminator, and a set of predictors by jointly learning the first generator, the second generator, the first discriminator, the second discriminator, and the set of predictors through an iterative process of optimizing a minimax objective wherein;

the first discriminator learns to determine a synthetic-to-real image from a real image,

the first generator learns to generate the synthetic-to-real image from the synthetic image such that the first discriminator determines the synthetic-to-real image is real,

the second generator learns to generate a real-to-synthetic image from the real image such that the second discriminator determines the real-to-synthetic image is fake,

the second discriminator learns to determine the real-to-synthetic image from the synthetic image such that differences between the real-to-synthetic image and the synthetic image are minimized, and

the set of predictors learn to predict at least one of a semantic segmentation labeled data and a privileged information from the synthetic-to-real image based on at least one of a known semantic segmentation labeled data and a known privileged information corresponding to the synthetic image; and

generate one or more photorealistic images using the trained GAN model.

10. The computer-implemented system of claim 9, further comprising:

a simulator communicatively coupled to the GAN model wherein the simulator generates the synthetic image, the known semantic segmentation labeled data within the synthetic image, and the known privileged information corresponding to the synthetic image, wherein the non-transitory computer-readable medium, when executed by the processor, further causes the processor to;

receive, from the simulator, the synthetic image, the known semantic segmentation labeled data within the synthetic image and the known privileged information corresponding to the synthetic image for training the GAN model.

11. The computer-implemented system of claim 9, wherein optimizing the minimax objective further causes the processor to:

update one or more parameters of the first discriminator during a first iteration of training the first generator, the first discriminator, and the set of predictors; and

update one or more parameters of the first generator and the set of predictors during a second iteration of training the first generator, the first discriminator, and the set of predictors.

12. The computer-implemented system of claim 9, wherein the minimax objective is characterized by a set of loss functions comprising an adversarial loss (_GAN), a task prediction loss (_task), a privileged information regularization (_pi), and a perceptual regularization (_perc):

min<sub>θ

G,θ

T,θ

P</sub> max<sub>θ

D</sub>α

_GAN+β

_T+γ

_P+δ

_perc,

wherein α

, β

, γ

, δ

are weighting parameters and θ

_G is a generator parameter, θ

_D is a discriminator parameter, θ

_T is a task predictor parameter, and θ

_pk is a privileged information predictor parameter.

13. The computer-implemented system of claim 9, wherein the set of predictors comprises at least one of a task network and a privileged information network.

14. The computer-implemented system of claim 13, wherein the task network learns to predict the semantic segmentation labeled data from the synthetic-to-real image.

15. The computer-implemented system of claim 13, wherein the privileged information network learns to predict at least one of depth, optical flow, object detection, and instance segmentation from the synthetic-to-real image.

16. The computer-implemented system of claim 9, wherein the first generator learns a pixel-level mapping function to generate the synthetic-to-real image from the synthetic image.

17. A method for generating photorealistic images comprising:

training a generative adversarial network (GAN) model by jointly learning a first generator, a second generator, a first discriminator, a second discriminator, and a set of predictors through an iterative process comprising the steps of;

generating, with the first generator, a synthetic-to-real image from a synthetic image simulated by a simulator;

determining, with the first discriminator, whether the synthetic-to-real image is real or fake and whether a real image from a dataset of real images is real or fake;

predicting, with the set of predictors, at least one of a labeled data and privileged information from the synthetic-to-real image; and

training the first generator, the second generator, the first discriminator, the second discriminator, and the set of predictors by optimizing a minimax objective wherein;

the first generator learns to generate the synthetic-to-real image from the synthetic image simulated by the simulator such that the first discriminator determines the synthetic-to-real image is real,

the first discriminator learns to determine the synthetic-to-real image from the real image, such that differences between the synthetic-to-real image and the real image are minimized,

the second generator learns to generate a real-to-synthetic image from the real image such that the second discriminator determines the real-to-synthetic image is fake,

the second discriminator learns to determine the real-to-synthetic image from the synthetic image such that differences between the real-to-synthetic image and the synthetic image are minimized, and

the set of predictors learn to predict at least one of the labeled data and the privileged information from the synthetic-to-real image based on at least one of the labeled data and the privileged information from the simulator.

18. The method of claim 17, further comprising:

updating one or more parameters of the first discriminator during a first iteration of training the first generator, the first discriminator, and the set of predictors; and

updating one or more parameters of the first generator and the set of predictors during a second iteration of training the first generator, the first discriminator, and the set of predictors.

19. The method of claim 17, wherein the set of predictors comprises at least one or a task network and a privileged information network wherein the task network learns to predict a semantic segmentation from the synthetic-to-real image and the privileged information network learns to predict at least one of depth, optical flow, object detection, and instance segmentation from the synthetic-to-real image.