IMAGE PROCESSING APPARATUS AND METHOD

• US 20200134787A1
• Filed: 12/27/2019
• Published: 04/30/2020
• Est. Priority Date: 06/28/2017
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: BOUZARAA, Fahd, URFALIOGLU, Onay, HALFAOUI, Ibrahim
• Granted Patent: US 11,055,827 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof

Abstract

The invention relates to an image processing apparatus for generating an HDR image associated with a first view on the basis of a plurality of LDR images, including a first LDR image and a second LDR image. The first LDR image is associated with the first view and a first exposure, i.e. a first dynamic range, and the second LDR image is associated with a second view and a second exposure, i.e. a second dynamic range. The image processing apparatus comprises a processor configured to provide a neural network having a plurality of neural subnetworks including a first neural subnetwork. The first neural subnetwork is configured to generate the HDR image on the basis of: (i) the first LDR image, (ii) the second LDR image, and (iii) a modified first LDR image. The modified first LDR image is associated with the first view and the second exposure.

Claims

1. An image processing apparatus for generating a high dynamic range (HDR) image on the basis of a plurality of low dynamic range (LDR) images, the LDR images including a first LDR image and a second LDR image, wherein the first LDR image is associated with a first view and a first exposure, and wherein the second LDR image is associated with a second view and a second exposure, the image processing apparatus comprising:

a processor configured to provide a neural network, wherein the neural network comprises a plurality of neural subnetworks including a first neural subnetwork,

wherein the first neural subnetwork is configured to generate the HDR image based on (i) the first LDR image, (ii) the second LDR image, and (iii) a modified first LDR image,

wherein the modified first LDR image is associated with the first view and the second exposure.

2. The image processing apparatus of claim 1, wherein the plurality of neural subnetworks comprises a second neural subnetwork which is configured to generate the modified first LDR image based on (i) the first LDR image and (ii) the second LDR image by mapping the first LDR image to the second exposure.

3. The image processing apparatus of claim 2, wherein the plurality of LDR images further comprises a third LDR image,

wherein the third LDR image is associated with a third view and a third exposure,

wherein the second neural subnetwork comprises a first portion and a second portion,

wherein the first portion of the second neural subnetwork is configured to generate the modified first LDR image based on (i) the first LDR image and (ii) the second LDR image by mapping the first LDR image to the second exposure,

wherein the second portion of the second neural subnetwork is configured to generate a further modified first LDR image, and

wherein the further modified first LDR image is associated with the first view and the second exposure, based on (i) the first LDR image and (ii) the third LDR image by mapping the first LDR image to the third exposure.

4. The image processing apparatus of claim 1, wherein the plurality of neural subnetworks comprises a third neural subnetwork configured to provide an improved version of the HDR image by removing ghosting artefacts from the HDR image based on (i) the first LDR image, (ii) the second LDR image, (iii) the modified first LDR image, (iv) the HDR image, and (v) a de-ghosting guiding HDR image.

5. The image processing apparatus of claim 4, wherein the processor includes an exposure fusion engine configured to generate the de-ghosting guiding HDR image based on an exposure fusion scheme based on (i) the first LDR image and (ii) the second LDR image.

6. The image processing apparatus of claim 5, wherein the exposure fusion engine is configured to generate the de-ghosting guiding HDR image based on the exposure fusion scheme by performing a weighted blending of (i) the first LDR image and (ii) the second LDR image using a weight map based on one or more quality measures.

7. The image processing apparatus of claim 1, wherein each of the plurality of neural subnetworks comprises one or more convolutional layers and one or more de-convolutional layers.

8. The image processing apparatus of claim 1, wherein the neural network is configured to be trained with a plurality of training sets, wherein each training set comprises an HDR image and a plurality of LDR images and wherein at least some training sets comprise more than two LDR images.

9. The image processing apparatus of claim 1, wherein the first neural subnetwork comprises a weighting layer configured to generate a weighting map based on one or more quality measures for reducing effects of low quality regions of the first LDR image and the second LDR image in generating the HDR image.

10. The image processing apparatus of claim 1, wherein the processor is further configured to select the first LDR image from the plurality of LDR images as a reference image based on a quality measure for reference image selection.

11. The image processing apparatus of claim 1, wherein the image processing apparatus further comprises a camera configured to capture the first LDR image and the second LDR image.

12. The image processing apparatus of claim 1, wherein the image processing apparatus further comprises a display configured to display the HDR image.

13. The image processing apparatus of claim 1, wherein the image processing apparatus is a smartphone.

14. An image processing method for generating a high dynamic range (HDR) image from a plurality of low dynamic range (LDR) images, including a first LDR image and a second LDR image, wherein the first LDR image is associated with a first view and a first exposure, and wherein the second LDR image is associated with a second view and a second exposure, the image processing method comprising:

providing a neural network, the neural network comprising a plurality of neural subnetworks including a first neural subnetwork, and

generating, by the first neural subnetwork, the HDR image based on (i) the first LDR image, (ii) the second LDR image, and (iii) a modified first LDR image,

wherein the modified first LDR image is associated with the first view and the second exposure.

15. A non-transitory computer-readable medium comprising program code which, when executed by a processor, causes the processor to perform the method of claim 14.