SUPER-RESOLUTION IN PROCESSING IMAGES SUCH AS FROM MULTI-LAYER SENSORS

• US 20150116545A1
• Filed: 02/26/2014
• Published: 04/30/2015
• Est. Priority Date: 10/30/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Ilia Ovsiannikov
• Inventors: OVSIANNIKOV, Ilia, SHI, Lilong
• Granted Patent: US 9,392,166 B2
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/815 for controlling the resolution by using a single image
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/702 SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Pixels of an array capture values for an input image. The captured values correspond to pixel center points of the pixels that captured the values. Additional values about the input image may be further computed from the captured values. These additional values may correspond to additional center points, which can be different from any of the pixel center points. An output image may be constructed from the captured values plus the additional values. These values may be stored together and/or displayed together as the output image. Embodiments are applicable to images such as from multi-layer sensors. Since the output image can be created from a higher total number of values than those captured by the available number of pixels, such image processing can be called super-resolution, and can be applied to image processing within imaging devices, for still pictures, video and motion pictures, and so on.

Claims

1. An imaging device, comprising:

an array having a set of pixels configured to capture respective values for an input image, the captured values corresponding to pixel center points of the respective pixels that captured the values;

a processor configured to compute, from the captured values, additional values about the input image, at least some of the additional values corresponding to additional center points different from any of the pixel center points; and

one or more memories configured to store the captured values and the additional values.

2. The device of claim 1, in which

the additional values are further merged with the captured values when stored.

3. The device of claim 1, further comprising:

a display configured to display an output image constructed from both the captured values and from the additional values.

4. The device of claim 3, in which

the additional values are demosaicked jointly with the captured values, and

the output image is constructed from the joint demosaicking.

5. The device of claim 1, in which

substantially identical tiles are defined from the pixel center points, and

one or more of the additional values that corresponds to one of the additional center points within a certain ones of the tiles is computed from one of the captured values within the certain tile and from one of the captured values outside the certain tile.

6. The device of claim 1, in which

a sum of a number of the captured values plus a number of the additional values is larger than a number of the pixels in the array.

7. The device of claim 1, in which

the pixels of the array are arranged in at least a lower layer and in an upper layer that is vertically stacked over the lower layer, and

the pixel center points are defined as those for the pixels in the upper layer plus those for projections of the pixels in the lower layer onto a plane of the upper layer.

8. The device of claim 1, further comprising:

a display configured to display an output image constructed from both the captured values and from the additional values.

9. The device of claim 8, in which

the additional values are demosaicked jointly with the captured values, and

the output image is constructed from the joint demosaicking.

10. The device of claim 1, in which

the additional values are computed using a filter function.

11. The device of claim 10, in which

the filter function has been trained by a processor performing actions resulting in operations comprising:

receiving input test values for a reference image;

selecting a tentative value for a parameter of the filter function;

computing, from the input test values, additional test values about the reference image using the tentative value; and

adjusting the parameter value.

12. The device of claim 11, in which

the input test values have been determined by exposing a test pixel array to the reference image.

13. The device of claim 11, in which

the input test values have been determined by calculating an expected response of the pixels due to exposure the reference image.

14. The device of claim 11, in which

the parameter of the filter function is one of a size of the filter, and a weight coefficient of the filter.

15. The device of claim 11, in which

a target version of the reference image is compared with an output test image constructed from the input test values and from the additional test values, and

the parameter value is adjusted based on the comparison.

16. The device of claim 15, in which

the additional test values are demosaicked jointly with the input test values, and

the output test image is constructed from the joint demosaicking.

17. The device of claim 15, in which

an additional image operation has been performed on the reference image to derive the target version of the reference image.

18. The device of claim 1, in which

the pixel center points are distributed along a certain direction according to a pixel pitch, and

at least some of the additional center points are distributed along the certain direction according to a construction pitch different than the pixel pitch.

19. An imaging device, comprising:

an array having a set of pixels configured to capture respective values for an input image, the captured values corresponding to pixel center points of the respective pixels that captured the values;

a processor configured to compute, from the captured values, additional values about the input image, at least some of the additional values corresponding to additional center points different from the pixel center points; and

a display configured to display an output image constructed from the captured values and from the additional values.

20-32. -32. (canceled)

33. A computer comprising:

a processor and a non-transitory storage medium coupled with the processor, the storage medium storing one or more programs which, when executed by the processor, result in;

receiving captured values for an input image, the captured values corresponding to pixel center points of respective pixels that captured the values;

computing, from the captured values, additional values about the input image, at least some of the additional values corresponding to additional center points different from the pixel center points; and

causing the captured values and the additional values to be stored in one or more memories.

34-44. -44. (canceled)

45. A method, comprising:

receiving captured values for an input image, the captured values corresponding to pixel center points of respective pixels that captured the values;

computing, from the captured values, additional values about the input image, at least some of the additional values corresponding to additional center points different from the pixel center points; and

causing the captured values and the additional values to be stored in one or more memories.

46-56. -56. (canceled)

57. A method for an imaging device having an array of pixels at respective pixel center points of the array, a processor and a display, the method comprising:

capturing values for an input image in a set of the pixels, the captured values corresponding to pixel center points of the respective pixels that captured the values;

computing, from the captured values, additional values about the input image, at least some of the additional values corresponding to additional center points different from the pixel center points; and

displaying an output image constructed from the captured values and from the additional values.

58-68. -68. (canceled)

69. A method for an imaging device having an array of pixels at respective pixel center points of the array, a processor and one or more memories, the method comprising:

capturing values for elements of an input image in a set of the pixels, the captured values corresponding to pixel center points of the respective pixels that captured the values;

computing, from the captured values, additional values about the input image by the processor, at least some of the additional values corresponding to additional center points different from the pixel center points; and

storing the captured values and the additional values in the one or more memories.

70-80. -80. (canceled)