Method and apparatus of neural network based image signal processor

• US 10,460,231 B2
• Filed: 03/18/2016
• Issued: 10/29/2019
• Est. Priority Date: 12/29/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Zhang, Qiang, Ji, Zhengping, Wang, Yibing Michelle, Ovsiannikov, Ilia
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US15/075,076
• Publication Number:
  US 20170185871A1
• Time in Patent Office: 1,320 Days
• Field of Search: None
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

An image signal processing (ISP) system is provided. The system includes a neural network trained by inputting a set of raw data images and a correlating set of desired quality output images; the neural network including an input for receiving input image data and providing processed output; wherein the processed output includes input image data that has been adjusted for at least one image quality attribute. A method and an imaging device are disclosed.

Claims

1. An image signal processing (ISP) system, comprising:

a neural network that receives an input image and outputs a processed image, the input image including input image data, the neural network comprising a convolutional neural network having multiple parallel paths in which each path scales the input image data to separate the input image data into separate frequency bands of spatial resolution, a first path comprising a first convolutional layer having an output at a first frequency band of spatial resolution, the output of the first convolutional layer being pooled at a second frequency band of spatial resolution that is less than the first frequency band of spatial resolution, a second path being coupled to the pooled output of the first convolutional layer and comprising a second convolutional layer having an output at the second frequency band of spatial resolution, the output of the second convolutional layer being pooled at a third frequency band of spatial resolution that is less than the second frequency band of spatial resolution, a third path being coupled to the pooled output of the second convolutional layer and comprising a third convolutional layer having an output at the third frequency band of spatial resolution, an output of the third convolutional layer being upsampled to the second frequency band of spatial resolution and concatenated with the output of the second convolutional layer to form a first intermediate output, the first intermediate output being upsampled to the first frequency band of spatial resolution and concatenated with the output of the first convolutional layer to form a second intermediate output, the second intermediate output being input to a fourth convolutional layer to output the processed image, and the neural network adjusting the input image data for at least one image-quality attribute to generate the processed image; and

an image-processing chain that receives the processed image output from the neural network, the image-processing chain providing at least one of color correction, gamma correction, edge enhancement and contrast enhancement of the processed image to output a final image,

wherein the at least one adjusted image-quality attribute comprises an image size, an aspect ratio, a brightness, an intensity, a bit depth, a white value, a dynamic range, a gray level, a contouring, a smoothing, a speckle, a color space values, an interleaving, a gamma correction, a contrast enhancement, a sharpness and a demosaicing.

2. The image signal processing (ISP) system as in claim 1, wherein the neural network has been trained by a back-propagation technique.

3. The image signal processing (ISP) system as in claim 1, wherein the processed image includes at least one detected feature comprising at least one of a line, a curve, an intensity and a color within the input image.

4. The image signal processing (ISP) as in claim 1, wherein the convolutional neural network comprises three parallel paths.

5. The image signal processing (ISP) system as in claim 1, wherein the input image includes noise,

wherein the neural network includes a plurality of layers,

wherein the neural network includes a plurality of filters each having dimensions that are different from each other,

wherein a height and a width of each respective filter is different from other filters,

wherein the neural network includes a pooling layer, and

wherein a filter is shifted across a predetermined layer of the neural network by a predetermined stride s.

6. A method for providing an image signal processing (ISP) system, the method comprising:

inputting an input image into a neural network, the input image comprising input image data, and the neural network comprising a convolutional neural network having multiple parallel paths in which each path scales the input image data to separate the input image data into multiple separate frequency bands of spatial resolution, a first path comprising a first convolutional layer having an output at a first frequency band of spatial resolution, the output of the first convolutional layer being pooled at a second frequency band of spatial resolution that is less than the first frequency band of spatial resolution, a second path being coupled to the pooled output of the first convolutional layer and comprising a second convolutional layer having an output at the second frequency band of spatial resolution, the output of the second convolutional layer being pooled at a third frequency band of spatial resolution that is less than the second frequency band of spatial resolution, a third path being coupled to the pooled output of the second convolutional layer and comprising a third convolutional layer having an output at the third frequency band of spatial resolution, an output of the third convolutional layer being upsampled to the second frequency band of spatial resolution and concatenated with the output of the second convolutional layer to form a first intermediate output, the first intermediate output being upsampled to the first frequency band of spatial resolution and concatenated with the output of the first convolutional layer to form a second intermediate output, the second intermediate output being input to a fourth convolutional layer to output the processed image;

adjusting the input image by the neural network to output a processed image having input image data adjusted for at least one image-quality attribute; and

,

generating by an image-processing chain a final image from the processed image, the image-processing chain processing at least one of a color correction, a gamma correction, an edge enhancement and a contrast enhancement of the processed image,

wherein the at least one adjusted image-quality attribute comprises an image size, an aspect ratio, a brightness, an intensity, a bit depth, a white value, a dynamic range, a gray level, a contouring, a smoothing, a speckle, a color space values, an interleaving, a gamma correction, a contrast enhancement, a sharpness and a demosaicing.

7. The method as in claim 6, further comprising training the neural network by back-propagation.

8. The method as in claim 6, further comprising generating the input image using an imaging device.

9. The method as in claim 6, wherein the processed image includes at least one detected feature comprising at least one of a line, a curve, an intensity and a color within the input image.

10. The method as in claim 6, wherein the input image data includes noise,

wherein the neural network includes a plurality of layers,

wherein the neural network includes three parallel paths,

wherein the neural network includes a plurality of filters each having dimensions that are different from each other,

wherein a height and a width of each respective filter is different from other filters,

wherein the neural network includes a pooling layer, and

wherein a filter is shifted across a predetermined layer of the neural network by a predetermined stride s.

11. An imaging device, comprising:

an imaging sensor that outputs raw image data; and

an image signal processing (ISP) system comprising a convolutional neural network and an image-processing chain, the convolutional neural network comprising multiple parallel paths in which each path scales the raw image data to separate the raw image data into separate frequency bands of spatial resolution, a first path comprising a first convolutional layer having an output at a first frequency band of spatial resolution, the output of the first convolutional layer being pooled at a second frequency band of spatial resolution that is less than the first frequency band of spatial resolution, a second path being coupled to the pooled output of the first convolutional layer and comprising a second convolutional layer having an output at the second frequency band of spatial resolution, the output of the second convolutional layer being pooled at a third frequency band of spatial resolution that is less than the second frequency band of spatial resolution, a third path being coupled to the pooled output of the second convolutional layer and comprising a third convolutional layer having an output at the third frequency band of spatial resolution, an output of the third convolutional layer being upsampled to the second frequency band of spatial resolution and concatenated with the output of the second convolutional layer to form a first intermediate output, the first intermediate output being upsampled to the first frequency band of spatial resolution and concatenated with the output of the first convolutional layer to form a second intermediate output, the second intermediate output being input to a fourth convolutional layer to output the processed image, the convolutional neural network to receive the raw image data from the imaging sensor and to adjust the raw image data to output a processed image in which the raw image data has been adjusted for at least one image-quality attribute, and the image-processing chain to output a final image, the image-processing chain comprising at least one of a color correction, a gamma correction, an edge enhancement and a contrast enhancement of the processed image,

wherein the at least one adjusted image-quality attribute comprises an image size, an aspect ratio, a brightness, an intensity, a bit depth, a white value, a dynamic range, a gray level, a contouring, a smoothing, a speckle, a color space values, an interleaving, a gamma correction, a contrast enhancement, a sharpness and a demosaicing.

12. The imaging device as in claim 11, wherein the processed image includes at least one detected feature comprising at least one of a line, a curve, an intensity and a color within the raw image data.

13. The imaging device as in claim 11, wherein the raw image data includes noise,

wherein the neural network includes a plurality of layers,

wherein the neural network includes three parallel paths,

wherein the neural network includes a plurality of filters each having dimensions that are different from each other,

wherein a height and a width of each respective filter is different from other filters,

wherein the neural network includes a pooling layer, and

wherein a filter is shifted across a predetermined layer of the neural network by a predetermined stride s.