Vehicle camera model for simulation using deep neural networks

• US 10,861,189 B2
• Filed: 12/20/2018
• Issued: 12/08/2020
• Est. Priority Date: 12/21/2017
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Potnis, Anuj S.
• Primary Examiner(s): Azarian, Seyed H
• Application Number:
  US16/227,666
• Publication Number:
  US 20190197731A1
• Time in Patent Office: 719 Days
• Field of Search: 382100, 382103-108, 382156, 382162, 382168, 382173, 382181, 382199, 382219, 382224, 382232, 382254, 382274, 382276, 382285-291, 382305, 382312, 706 16, 706 6, 348 38
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A camera simulation system or method or process for simulating performance of a camera for a vehicle includes providing a camera having a lens and imager and providing a learning algorithm. Image data is captured from a raw image input of the camera and the captured image data and raw image input are output from the camera. The output image data and the raw image input are provided to the learning algorithm. The learning algorithm is trained to simulate performance of the lens and/or the imager using the output captured image data and the raw image data input. The performance of the lens and/or the imager is simulated responsive to the learning algorithm receiving raw images.

Claims

1. A method for simulating performance of a vehicular camera, said method comprising:

providing a control comprising a data processor that is operable to execute a learning algorithm, wherein the learning algorithm comprises a generative adversarial network;

providing a vehicular camera comprising a lens and imager;

providing an actual target in a field of view of the vehicular camera;

capturing, via the vehicular camera, image data representative of the actual target as imaged by the vehicular camera;

providing the captured image data to the control;

wherein the learning algorithm comprises (i) a generator that generates an output responsive to capturing image data and (ii) a discriminator that compares the generator output to the captured image data;

providing actual target data to the control, wherein the actual target data represents the actual target provided in the field of view of the vehicular camera;

generating, via the learning algorithm, a learning algorithm output;

processing, at the control, the captured image data and the learning algorithm output, wherein processing the captured image data and the learning algorithm output comprises comparing the captured image data to the learning algorithm output;

responsive to the processing of the captured image data and the learning algorithm output,

training the learning algorithm to simulate performance of the lens and

the imager using the captured image data and the actual target data; and

simulating, based on the training of the learning algorithm, the performance of the lens and the imager.

2. The method of claim 1, wherein the learning algorithm comprises a deep neural network.

3. The method of claim 1, comprising determining if discrimination is correct, and if discrimination is not correct, providing a feedback signal to the generator to adjust the generator output, and if discrimination is correct, outputting image data of the vehicular camera.

4. The method of claim 3, wherein the generator and the discriminator are part of a generative adversarial network.

5. The method of claim 1, wherein the vehicular camera comprises at least one of dynamic tuning parameters or static tuning parameters.

6. The method of claim 5, wherein the control controls tuning of the dynamic tuning parameters.

7. The method of claim 5, wherein simulating the performance of the lens and the imager comprises simulating the at least one of dynamic tuning parameters or static tuning parameters.

8. The method of claim 1, wherein the learning algorithm comprises a parameterized model.

9. The method of claim 1, wherein simulating the performance of the lens and the imager comprises simulating a performance of the lens that is selected from the group consisting of (i) distortion of images by the lens and (ii) a field of view provided by the lens.

10. The method of claim 1, wherein simulating the performance of the lens and the imager comprises simulating a performance of the imager that is selected from the group consisting of (i) noise, (ii) contrast, (iii) white balance, and (iv) gamma correction.

11. A method for simulating performance of a vehicular camera, said method comprising:

providing a control comprising a data processor that is operable to execute a generative adversarial network;

providing a vehicular camera comprising a lens and imager;

providing an actual target in a field of view of the vehicular camera;

capturing, via the vehicular camera, image data representative of the actual target as imaged by the vehicular camera;

providing the captured image data to the control;

providing actual target data to the control, wherein the actual target data represents the actual target provided in the field of view of the vehicular camera;

generating, via the generative adversarial network, a generative adversarial network output;

processing, at the control, the captured image data and the generative adversarial network output, wherein processing the captured image data and the generative adversarial network output comprises comparing the captured image data to the generative adversarial network output;

responsive to the processing of the captured image data and the generative adversarial network output, training the generative adversarial network to simulate performance of the imager using the captured image data and the actual target data;

simulating, based on the training of the generative adversarial network, the performance of the imager; and

wherein simulating the performance of the imager comprises simulating a performance that is selected from the group consisting of (i) noise, (ii) contrast, (iii) white balance, and (iv) gamma correction.

12. The method of claim 11, wherein the generative adversarial network comprises (i) a generator that generates an output responsive to capturing image data and (ii) a discriminator that compares the generator output to the captured image data.

13. The method of claim 12, comprising determining if discrimination is correct, and if discrimination is not correct, providing a feedback signal to the generator to adjust the generator output, and if discrimination is correct, outputting image data of the vehicular camera.

14. The method of claim 11, wherein the vehicular camera comprises at least one of dynamic tuning parameters or static tuning parameters.

15. The method of claim 14, wherein the control controls tuning of the dynamic tuning parameters.

16. The method of claim 15, wherein simulating the performance of the imager comprises simulating the at least one of dynamic tuning parameters or static tuning parameters.

17. The method of claim 11, comprising simulating a performance of the lens that is selected from the group consisting of (i) distortion of images by the lens and (ii) a field of view provided by the lens.

18. A method for simulating performance of a vehicular camera, said method comprising:

providing a control comprising a data processor that is operable to execute a learning algorithm, wherein the learning algorithm comprises a generative adversarial network;

providing a vehicular camera comprising a lens and imager;

providing an actual target in a field of view of the vehicular camera;

capturing, via the vehicular camera, image data representative of the actual target as imaged by the vehicular camera;

providing the captured image data to the control;

wherein the learning algorithm comprises (i) a generator that generates an output responsive to capturing image data and (ii) a discriminator that compares the generator output to the captured image data;

providing actual target data to the control, wherein the actual target data represents the actual target provided in the field of view of the vehicular camera;

generating, via the learning algorithm, a learning algorithm output;

processing, at the control, the captured image data and the learning algorithm output, wherein processing the captured image data and the learning algorithm output comprises comparing the captured image data to the learning algorithm output;

responsive to the processing of the captured image data and the learning algorithm output, training the learning algorithm to simulate performance of the lens and the imager using the captured image data and the actual target data;

simulating, based on the training of the learning algorithm, the performance of the lens and the imager;

wherein simulating the performance of the lens and the imager comprises simulating a performance of the lens that is selected from the group consisting of (i) distortion of images by the lens and (ii) a field of view provided by the lens; and

wherein simulating the performance of the lens and the imager comprises simulating a performance of the imager that is selected from the group consisting of (i) noise, (ii) contrast, (iii) white balance, and (iv) gamma correction.