SUPER-RESOLUTION PROCESSOR AND SUPER-RESOLUTION PROCESSING METHOD

• US 20110206296A1
• Filed: 02/07/2011
• Published: 08/25/2011
• Est. Priority Date: 02/09/2010
• Status: Active Grant

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Takita, Kenji, Sakaguchi, Satoshi, Matsunobu, Toru
• Granted Patent: US 8,526,765 B2
• Time in Patent Office: Days
• US Class Current: 382/299
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

A super-resolution processor according to the present invention includes: an N enlargement unit that generates an N-enlarged image by enlarging the input image by a factor N; an M enlargement unit that generates an M-enlarged image by enlarging the input image by a factor M; a high-pass filter unit that extracts a high-frequency component of the M-enlarged image, as an M-enlarged high-frequency image; a patch extraction unit that extracts an estimated patch of a predetermined size from the M-enlarged high-frequency image, the estimated patch being a part of the M-enlarged high-frequency image; and an addition unit that adds the estimated patch to a processing target block of the predetermined size in the N-enlarged image, to generate the output image, where M is smaller than N.

Claims

1. A super-resolution processor that performs super-resolution processing on an input image to generate an output image of a higher resolution than the input image, said super-resolution processor comprising:

an N enlargement unit configured to generate an N-enlarged image by enlarging the input image by a factor N, where N is larger than 1;

an M enlargement unit configured to generate an M-enlarged image by enlarging the input image by a factor M, where M is larger than 1;

a high-pass filter unit configured to extract a high-frequency component of the M-enlarged image, as an M-enlarged high-frequency image;

a patch extraction unit configured to extract an estimated patch of a predetermined size from the M-enlarged high-frequency image, the estimated patch being a part of the M-enlarged high-frequency image; and

an addition unit configured to add the estimated patch to a processing target block of the predetermined size in the N-enlarged image, to generate the output image,

wherein M is smaller than N.

2. The super-resolution processor according to claim 1,

wherein said patch extraction unit is configured to extract the estimated patch, from a neighboring region of a position of the processing target block in the M-enlarged high-frequency image.

3. The super-resolution processor according to claim 1, further comprising:

a first feature value extraction unit configured to extract an input feature value, the input feature value being a feature value of the processing target block in the N-enlarged image; and

a second feature value extraction unit configured to extract a plurality of first neighboring feature values, each of the plurality of first neighboring feature values being a feature value of a different one of a plurality of patches in the M-enlarged image, the plurality of patches each having the predetermined size,

wherein said patch extraction unit is configured to calculate similarity between the input feature value and each of the plurality of first neighboring feature values, and extract, as the estimated patch, a region in the M-enlarged high-frequency image corresponding to any of;

(1) one first neighboring feature value having highest similarity;

(2) a predetermined number of first neighboring feature values in decreasing order of similarity; and

(3) first neighboring feature values each having similarity equal to or higher than a predetermined threshold, from among the plurality of first neighboring feature values.

4. The super-resolution processor according to claim 1, further comprising

a self-similarity ratio estimation unit configured to estimate a self-similarity ratio of the N-enlarged image,

wherein said M enlargement unit is configured to generate the M-enlarged image by enlarging the input image by the factor M where M=N/K, when the self-similarity ratio is 1/K.

5. The super-resolution processor according to claim 4,

wherein said self-similarity ratio estimation unit is configured to calculate a period of variation of an autocorrelation function of the input image or the N-enlarged image, determine a larger value of K when the period of variation is shorter, and estimate the self-similarity ratio as 1/K.

6. The super-resolution processor according to claim 4,

wherein said self-similarity ratio estimation unit is configured to calculate energy of a high-frequency component of the input image or the N-enlarged image, determine a larger value of K when the energy is smaller, and estimate the self-similarity ratio as 1/K.

7. The super-resolution processor according to claim 4,

wherein said self-similarity ratio estimation unit is configured to calculate a sum of absolute differences between adjacent pixels of the input image or the N-enlarged image, determine a larger value of K when the sum of absolute differences is smaller, and estimate the self-similarity ratio as 1/K.

8. The super-resolution processor according to claim 3,

wherein said M enlargement unit is further configured to generate an M1-enlarged image by enlarging the input image by a factor M1, where M1 is larger than 1,

M1 is smaller than N, and different from M,

said high-pass filter unit is further configured to extract a high-frequency component of the M1-enlarged image, as an M1-enlarged high-frequency image,

said second feature value extraction unit is further configured to extract a plurality of second neighboring feature values, each of the plurality of second neighboring feature values being a feature value of a different one of a plurality of patches in the M1-enlarged image, the plurality of patches each having the predetermined size,

said patch extraction unit is further configured to calculate similarity between the input feature value and each of the plurality of second neighboring feature values, and

said patch extraction unit is configured to extract the estimated patch from a region in the M-enlarged high-frequency image or the M1-enlarged high-frequency image corresponding to any of:

(1) one first neighboring feature value or second neighboring feature value having highest similarity;

(2) a predetermined number of first neighboring feature values or second neighboring feature values in decreasing order of similarity; and

(3) first neighboring feature values or second neighboring feature values each having similarity equal to or higher than a predetermined threshold, from among the plurality of first neighboring feature values and the plurality of second neighboring feature values.

9. The super-resolution processor according to claim 3,

wherein said patch extraction unit is configured to calculate, as the similarity, a sum of absolute differences or a sum of squared differences between the input feature value and each of the plurality of first neighboring feature values.

10. The super-resolution processor according to claim 1,

wherein said M enlargement unit is configured to set M to a first value in the case where N is larger than a predetermined threshold, and set M to a second value in the case where N is equal to or smaller than the predetermined threshold, the second value being smaller than the first value.

15. The super-resolution processor according to claim 1, further comprising:

an edge super-resolution unit configured to generate an edge high-resolution image by increasing a resolution of the input image;

an edge detection unit configured to detect an edge in the input image; and

an image combining unit configured to combine the output image and the edge high-resolution image,

wherein said image combining unit is configured to;

combine the output image and the edge high-resolution image at a ratio in which the edge high-resolution image is higher than the output image, for a region having a larger amount of edge than a predetermined threshold in the input image; and

combine the output image and the edge high-resolution image at a ratio in which the edge high-resolution image is lower than the output image, for a region having a smaller amount of edge than the predetermined threshold in the input image.

11. A super-resolution processor that performs super-resolution processing on an input image to generate an output image, said super-resolution processor comprising:

a 1/K reduction unit configured to generate a 1/K-reduced image by reducing the input image by 1/K;

a high-pass filter unit configured to extract a high-frequency component of the 1/K-reduced image, as a 1/K-reduced high-frequency image;

a patch extraction unit configured to extract an estimated patch of a predetermined size from the 1/K-reduced high-frequency image, the estimated patch being a part of the 1/K-reduced high-frequency image; and

an addition unit configured to add the estimated patch to a processing target block of the predetermined size in the 1/K-reduced image, to generate the output image.

12. The super-resolution processor according to claim 11,

wherein said patch extraction unit is configured to extract the estimated patch, from a neighboring region of a position of the processing target block in the 1/K-reduced high-frequency image.

13. The super-resolution processor according to claim 11, further comprising:

a first feature value extraction unit configured to extract an input feature value, the input feature value being a feature value of the processing target block in the input image; and

a second feature value extraction unit configured to extract a plurality of first neighboring feature values, each of the plurality of first neighboring feature values being a feature value of a different one of a plurality of patches in the 1/K-reduced image, the plurality of patches each having the predetermined size,

wherein said patch extraction unit is configured to calculate similarity between the input feature value and each of the plurality of first neighboring feature values, and extract, as the estimated patch, a region in the 1/K-reduced high-frequency image corresponding to any of;

one first neighboring feature value having highest similarity;

a predetermined number of first neighboring feature values in decreasing order of similarity; and

first neighboring feature values each having similarity equal to or higher than a predetermined threshold, from among the plurality of first neighboring feature values.

14. The super-resolution processor according to claim 11, further comprising

a self-similarity ratio estimation unit configured to estimate a self-similarity ratio of the input image,

wherein said 1/K reduction unit is configured to generate the 1/K-reduced image by reducing the input image by 1/K, when the self-similarity ratio is 1/K.

16. A super-resolution processing method for performing super-resolution processing on an input image to generate an output image of a higher resolution than the input image, said super-resolution processing method comprising:

generating an N-enlarged image by enlarging the input image by a factor N, where N is larger than 1;

generating an M-enlarged image by enlarging the input image by a factor M, where M is larger than 1;

extracting a high-frequency component of the M-enlarged image, as an M-enlarged high-frequency image;

extracting an estimated patch of a predetermined size from the M-enlarged high-frequency image, the estimated patch being a part of the M-enlarged high-frequency image; and

adding the estimated patch to a processing target block of the predetermined size in the N-enlarged image, to generate the output image,

wherein M is smaller than N.

18. A non-transitory computer-readable recording medium having a computer program recorded thereon for causing a computer to execute the super-resolution processing method according to claim 16.

17. A super-resolution processing method for performing super-resolution processing on an input image to generate an output image, said super-resolution processing method comprising:

generating a 1/K-reduced image by reducing the input image by 1/K;

extracting a high-frequency component of the 1/K-reduced image, as a 1/K-reduced high-frequency image;

extracting an estimated patch of a predetermined size from the 1/K-reduced high-frequency image, the estimated patch being a part of the 1/K-reduced high-frequency image; and

adding the estimated patch to a processing target block of the predetermined size in the 1/K-reduced image, to generate the output image.

19. A non-transitory computer-readable recording medium having a computer program recorded thereon for causing a computer to execute the super-resolution processing method according to claim 17.