Image-processing device which generates a high-resolution image by combining images, imaging device, image-processing method, and storage-medium

• US 10,389,952 B2
• Filed: 11/21/2017
• Issued: 08/20/2019
• Est. Priority Date: 06/24/2015
• Status: Active Grant

• Current Assignee: Olympus Corporation
• Original Assignee: Olympus Corporation
• Inventors: Furukawa, Eiji, Kajimura, Kosuke
• Primary Examiner(s): Haliyur, Padma
• Application Number:
  US15/819,968
• Publication Number:
  US 20180077360A1
• Time in Patent Office: 637 Days
• CPC Class Codes: G06T 2207/20216 Image averaging
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 5/265 Mixing

Abstract

An image-processing device includes a high-resolution combining portion that generates a combined image by combining a standard image and at least one reference image, which are acquired by capturing images of a subject in time series by using an imaging element in which multiple types of color filters are arrayed in individual pixels, in a high-resolution space where the resolution is greater than those of the standard image and the reference images. The image-processing device also includes a position-displacement-distribution calculating portion that calculates a distribution of position displacements between two comparative images that are individually formed of pixels corresponding to the different types of color filters in the generated combined image, a correlation-level calculating portion that calculates, for individual pixels, correlation levels between the two comparative images based on the calculated distribution, and an image correcting portion that corrects the combined image based on the calculated correlation levels.

Claims

1. An image-processing device comprising:

an image processor comprising circuitry or a hardware processor that operates under control of a stored program, the image processor being configured to execute processes comprising;

a high-resolution combining process that generates a combined image by combining a standard image and at least one reference image other than the standard image, which are acquired by capturing images of an imaging subject in time series by using an imaging element in which multiple types of color filters are arrayed in individual pixels, in a high-resolution space in which a resolution is greater than resolutions of the standard image and the at least one reference image;

a position-displacement-distribution calculating process that calculates a distribution of position displacements between two comparative images that are individually formed of pixels corresponding to the multiple types of color filters in the combined image generated in the high-resolution combining process;

a correlation-level calculating process that calculates, for individual pixels, correlation levels between the two comparative images based on the distribution of the position displacements calculated in the position-displacement-distribution calculating process; and

an image correcting process that corrects the combined image based on the correlation levels calculated in the correlation-level calculating process,

wherein the position-displacement-distribution calculating process calculates, for each of the two comparative images, positions of a maximum value and a minimum value of values obtained by adding up or averaging pixel values of two or more pixels for a plurality of small regions formed of two or more individually corresponding pixels, and calculates displacement amounts at the positions of the maximum value and the minimum value for each of the small regions corresponding between the two comparative images.

2. The image-processing device according to claim 1, wherein the image correcting process corrects the combined image by combining the combined image and the standard image by using combining ratios based on the correlation levels calculated in the correlation-level calculating process.

3. The image-processing device according to claim 1, wherein the image correcting process corrects the combined image by combining two images, which are generated by applying filters having different low-pass effects to the combined image, by using combining ratios based on the correlation levels calculated in the correlation-level calculating process.

4. The image-processing device according to claim 1, wherein the image correcting process comprises (i) a first correction process in which two images, which are generated by applying filters having different low-pass effects to the combined image, are combined by using combining ratios based on the correlation levels calculated in the correlation-level calculating process, and (ii) a second correction process in which the image obtained in the first correction process is combined with the standard image.

5. The image-processing device according to claim 1, wherein the position-displacement-distribution calculating process adds up or averages pixel values of a plurality of pixel groups formed of two or more pixels that are arrayed in one direction in the small regions.

6. The image-processing device according to claim 1, wherein the position-displacement-distribution calculating process adds up or averages pixel values of two or more pixels of a plurality of rectangular regions in the small regions.

7. The image-processing device according to claim 1, wherein the correlation-level calculating process calculates the correlation levels that are decreased with an increase in the displacement amounts.

8. The image-processing device according to claim 7, wherein the correlation-level calculating process calculates a number of the position displacements so as to serve as the displacement amounts.

9. An imaging device comprising:

the image-processing device according to claim 1; and

an image-acquisition portion comprising an imaging element that acquires the standard image and the at least one reference image other than the standard image by capturing the images of the imaging subject in time series by using the imaging element in which multiple types of color filters are arrayed in individual pixels,

wherein the image-processing device processes the standard image and the reference images acquired by the image-acquisition portion.

12. An imaging device comprising:

the image-processing device according to claim 2; and

an image-acquisition portion comprising an imaging element that acquires the standard image and the at least one reference image other than the standard image by capturing the images of the imaging subject in time series by using the imaging element in which multiple types of color filters are arrayed in individual pixels,

wherein the image-processing device processes the standard image and the reference images acquired by the image-acquisition portion.

13. An imaging device comprising:

the image-processing device according to claim 3; and

an image-acquisition portion comprising an imaging element that acquires the standard image and the at least one reference image other than the standard image by capturing the images of the imaging subject in time series by using the imaging element in which multiple types of color filters are arrayed in individual pixels,

wherein the image-processing device processes the standard image and the reference images acquired by the image-acquisition portion.

14. An imaging device comprising:

the image-processing device according to claim 4; and

an image-acquisition portion comprising an imaging element that acquires the standard image and the at least one reference image other than the standard image by capturing the images of the imaging subject in time series by using the imaging element in which multiple types of color filters are arrayed in individual pixels,

wherein the image-processing device processes the standard image and the reference images acquired by the image-acquisition portion.

15. An imaging device comprising:

the image-processing device according to claim 5; and

an image-acquisition portion comprising an imaging element that acquires the standard image and the at least one reference image other than the standard image by capturing the images of the imaging subject in time series by using the imaging element in which multiple types of color filters are arrayed in individual pixels,

wherein the image-processing device processes the standard image and the reference images acquired by the image-acquisition portion.

16. An imaging device comprising:

the image-processing device according to claim 6; and

an image-acquisition portion comprising an imaging element that acquires the standard image and the at least one reference image other than the standard image by capturing the images of the imaging subject in time series by using the imaging element in which multiple types of color filters are arrayed in individual pixels,

wherein the image-processing device processes the standard image and the reference images acquired by the image-acquisition portion.

17. An imaging device comprising:

the image-processing device according to claim 7; and

an image-acquisition portion comprising an imaging element that acquires the standard image and the at least one reference image other than the standard image by capturing the images of the imaging subject in time series by using the imaging element in which multiple types of color filters are arrayed in individual pixels,

wherein the image-processing device processes the standard image and the reference images acquired by the image-acquisition portion.

18. An imaging device comprising:

the image-processing device according to claim 8; and

an image-acquisition portion comprising an imaging element that acquires the standard image and the at least one reference image other than the standard image by capturing the images of the imaging subject in time series by using the imaging element in which multiple types of color filters are arrayed in individual pixels,

wherein the image-processing device processes the standard image and the reference images acquired by the image-acquisition portion.

10. An image-processing method comprising:

a high-resolution combining step of generating a combined image by combining a standard image and at least one reference image other than the standard image, which are acquired by capturing images of an imaging subject in time series by using an imaging element in which multiple types of color filters are arrayed in individual pixels, in a high-resolution space in which a resolution is greater than resolutions of the standard image and the at least one reference image;

a position-displacement-distribution calculating step of calculating a distribution of position displacements between two comparative images that are individually formed of pixels corresponding to the multiple types of color filters in the combined image generated in the high-resolution combining step;

a correlation-level calculating step of calculating, for individual pixels, correlation levels between the two comparative images based on the distribution of the position displacements calculated in the position-displacement-distribution calculating step; and

an image correcting step of correcting the combined image based on the correlation levels calculated in the correlation-level calculating step,

wherein the position-displacement-distribution calculating step comprises calculating, for each of the two comparative images, positions of a maximum value and a minimum value of values obtained by adding up or averaging pixel values of two or more pixels for a plurality of small regions formed of two or more individually corresponding pixels, and calculating displacement amounts at the positions of the maximum value and the minimum value for each of the small regions corresponding between the two comparative images.

11. A non-transitory computer-readable storage medium storing an image-processing program that causes a computer to execute:

a high-resolution combining process of generating a combined image by combining a standard image and at least one or mom reference image other than the standard image, which are acquired by capturing images of an imaging subject in time series by using an imaging element in which multiple types of color filters are arrayed in individual pixels, in a high-resolution space in which a resolution is greater than resolutions of the standard image and the at least one reference image;

a position-displacement-distribution calculating process of calculating a distribution of position displacements between two comparative images that are individually formed of pixels corresponding to the different types of color filters in the combined image generated in the high-resolution combining process;

a correlation-level calculating process of calculating, for individual pixels, correlation levels between the two comparative images based on the distribution of the position displacements calculated in the position-displacement-distribution calculating process; and

an image correcting process of correcting the combined image based on the correlation levels calculated in the correlation-level calculating process,

wherein the position-displacement-distribution calculating process calculates, for each of the two comparative images, positions of a maximum value and a minimum value of values obtained by adding up or averaging pixel values of two or more pixels for a plurality of small regions formed of two or more individually corresponding pixels, and calculates displacement amounts at the positions of the maximum value and the minimum value for each of the small regions corresponding between the two comparative images.