Creating details in an image with adaptive frequency lifting

• US 9,536,288 B2
• Filed: 03/14/2014
• Issued: 01/03/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Tao, Li, Kim, Yeong-Taeg
• Primary Examiner(s): WU, XIAO MIN
• Assistant Examiner(s): Chen, Yu
• Application Number:
  US14/214,114
• Publication Number:
  US 20140267347A1
• Time in Patent Office: 1,026 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]

Abstract

Image details are created for an image. An image is processed to obtain image information. Detected texture samples are processed to obtain texture information. A processor device is used for transforming received image information for obtaining frequency domain information. The texture information is used for determining a lifting factor. A frequency distribution is determined for the frequency domain information using the lifting factor for creating particular frequency. An inverse transformation is performed on an updated frequency distribution for creating output image blocks. The output image blocks are combined to create image details for a lifted image. The lifted image is mixed with the image.

Claims

1. A method comprising:

determining, by a processor device, enhancement information based on texture information of input image information, the determining of the enhancement information comprising;

performing frequency lifting on a transform coefficient of the input image, the transform coefficient being representative of the input image at a first frequency, the frequency lifting corresponding to adjusting the transform coefficient from the first frequency to a second frequency greater than the first frequency, wherein the frequency lifting comprises;

determining a distance from the transform coefficient to a zero-frequency coefficient position;

determining a texture level of the texture information; and

determining the second frequency for the adjusting of the transform coefficient, the second frequency being determined based at least on the distance and the texture level; and

mixing the enhancement information with the input image information to generate an enhanced image.

2. The method of claim 1, further comprising:

processing an input image to obtain the input image information;

wherein the determining of the enhancement information comprises;

processing detected texture samples to obtain the texture information;

transforming the input image information for obtaining frequency domain information;

using the texture information for determining a lifting factor;

determining a frequency distribution for the frequency domain information using the lifting factor for creating particular frequency components;

performing an inverse transformation on an updated frequency distribution for creating one or more output image blocks; and

combining the one or more output image blocks to create the enhancement information that comprises image details for a lifted image.

3. The method of claim 2, further comprising:

inputting the input image;

dividing the input image into the input image information that comprises one or more input image blocks;

detecting texture in the image; and

dividing the detected texture into the texture information that comprises one or more texture blocks.

4. The method of claim 3, wherein an input image block and a texture block each has a block size larger than a block size of an output image block.

5. The method of claim 2, wherein determining the lifting factor comprises:

estimating a texture strength value based on number of texture samples in an input image block; and

computing the lifting factor based on texture sample thresholds as a function of the texture strength value.

6. The method of claim 5, wherein the frequency domain information comprises the transform coefficients, and the particular frequency components are created based on:

performing the frequency lifting on the transform coefficients based on the determined distances for lifting one or more frequency components to a higher frequency location in a transform spectrum, wherein the determined distances are adjusted using the lifting factor; and

converting the shifted transform coefficients to modified transform coefficients.

7. The method of claim 5, wherein the frequency domain information comprises the transform coefficients, and the particular frequency components are created based on:

using the shifted transform coefficient for determining the distance from the shifted transform coefficient to the zero-frequency coefficient position based on using associated transform coefficient row and column indexes, wherein said shifting of the transform coefficient is performed in a matrix;

performing the frequency lifting on the shifted transform coefficient based on the determined distance for lifting one or more frequency components to a higher frequency location in a transform spectrum, wherein the determined distance is adjusted using the lifting factor; and

converting the shifted transform coefficient to a modified transform coefficient.

8. The method of claim 7, further comprising:

moving the zero-frequency coefficient position to a center of the matrix after said shifting, wherein the zero-frequency coefficient position is used as an origin of a two-dimensional coordinate, and matrix elements are used as points on the two-dimensional coordinate.

9. The method of claim 8, wherein determining distances for coefficients is based on a pre-determined transfer function, and transforming received image information comprises using one of discrete Fourier transform processing, discrete cosine transform processing and discrete sign transform processing.

10. The method of claim 2, wherein an output image comprising the image details and the input image is displayed on an electronic device.

11. The method of claim 10, wherein the electronic device comprises a television device, a monitor device, a computing device or a mobile computing device.

12. The method of claim 1, wherein the processor device performs processing on a local television device, a set-top-box device, or a network server device.

13. A system comprising

a memory storing instructions;

one or more processors configured to execute the instructions to perform operations including texture processing to obtain texture information from an input image, and detail creation processing to;

determine enhancement information based on the texture information and performing frequency lifting on a transform coefficient of the input image, the transform coefficient being representative of the input image at a first frequency, the frequency lifting corresponding to adjusting the transform coefficient from the first frequency to a second frequency greater than the first frequency, wherein performing the frequency lifting comprises;

determining a distance from the transform coefficient to a zero frequency coefficient position;

determining a texture level of the texture information; and

determining the second frequency for the adjusting of the transform coefficient, the second frequency being determined based at least on the distance and the texture level; and

a mixer configured to mix the enhancement information with the input image.

14. The system of claim 13, wherein the one or more processors are configured to execute the instructions to perform further operations including:

texture processing to detect texture samples of the input image;

processing the texture samples to obtain the texture information;

transforming received input image information to obtain frequency domain information;

using the texture information to determine a lifting factor;

determining a frequency distribution for the frequency domain information using the lifting factor to create particular frequency components;

performing an inverse transformation on the particular frequency components to create one or more output image blocks;

and the system further comprises;

block constructing to combine the one or more output image blocks to create the enhancement information that comprises image details for a lifted image;

wherein the mixer is configured to mix the lifted image with the input image.

15. The system of claim 14, wherein the one or more processors are configured to execute the instructions to perform further operations including inputting the input image and dividing the input image into the input image information that comprises one or more input image blocks, and dividing detected texture samples into the texture information that comprises one or more texture blocks.

16. The system of claim 15, wherein an input image block and a texture block each has a block size larger than a block size of an output image block.

17. The system of claim 14, wherein the lifting factor is determined by the one or more processors that are configured to execute the instructions to perform further operations including:

estimating a texture strength value based on number of texture samples in an input image block; and

using texture sample thresholds as a function of the texture strength value for determining the lifting factor.

18. The system of claim 17, wherein the frequency domain information comprises the transform coefficient, and the one or more processors are configured to execute the instructions to perform further operations including creating the particular frequency components based on:

using the shifted transform coefficient to determine the distance from the shifted transform coefficient to the zero-frequency coefficient position based on using associated coefficient row and column indexes, wherein said shifting of the transform coefficient is performed in a matrix;

performing the frequency lifting on the shifted transform coefficient based on the determined distance to lift one or more frequency components to a higher frequency location in a transform spectrum, wherein the determined distance is adjusted using the lifting factor; and

converting the shifted transform coefficient to a modified transform coefficient.

19. The system of claim 17, wherein the frequency domain information comprises the transform coefficient, and the one or more processors are configured to execute the instructions to perform further operations including creating the particular frequency components based on:

using the shifted transform coefficient to determine the distance from the shifted transform coefficient to the zero-frequency coefficient position based on using associated coefficient row and column indexes, wherein said shifting of the transform coefficient is performed in a matrix;

performing the frequency lifting on the shifted transform coefficient based on the determined distance to lift one or more frequency components to a higher frequency location in a transform spectrum, wherein the determined distance is adjusted using the lifting factor; and

converting the shifted transform coefficient to a modified transform coefficient.

20. The system of claim 19, wherein the one or more processors are configured to execute the instructions to perform further operations including moving the zero-frequency coefficient position to a center of the matrix after shifting the transform coefficients position in the matrix, and the zero-frequency coefficient position is used as an origin of a two-dimensional coordinate, and matrix elements are used as points on the two-dimensional coordinate.

21. The system of claim 20, wherein the distances for the coefficients are determined based on a pre-determined transfer function, and transforming received image information comprises performing one of a discrete Fourier transform and a discrete cosine transform.

22. The system of claim 14, wherein an output image that comprises the image details and the input image is displayed on an electronic device that comprises a television device, a monitor device, a computing device or a mobile computing device.

23. The system of claim 14, wherein the one or more processors are configured to perform processing on a television device, a set-top-box device, or a network server device.

24. A non-transitory computer-readable medium having instructions which when executed on a computer perform a method comprising:

determining enhancement information based on texture information of input image information, the determining of the enhancement information comprising;

performing frequency lifting on a transform coefficient of the input image, the transform coefficient is representative of the input image at a first frequency, the frequency lifting corresponding to adjusting the transform coefficient from the first frequency to a second frequency greater than the first frequency, wherein the frequency lifting comprises;

determining a distance from the transform coefficient to a zero frequency coefficient position;

determining a texture level of the texture information; and

determining the second frequency for the adjusting of the transform coefficient, the second frequency being determined based at least on the distance and the texture level;

and mixing the enhancement information with the input image information to generate an enhanced image.

25. The non-transitory computer-readable medium of claim 24, further comprising:

processing the input image to obtain the input image information; and

processing detected texture samples of the input image to obtain the texture information;

wherein determining the enhancement information comprises;

transforming the input image information for obtaining frequency domain information;

using the texture information for determining a lifting factor;

determining a frequency distribution for the frequency domain information using the lifting factor for creating particular frequency components;

performing an inverse transformation on an updated frequency distribution for creating one or more output image blocks; and

combining the one or more output image blocks to create the enhancement information that comprises image details for a lifted image, and mixing the lifted image with the input image.

26. The non-transitory computer-readable medium of claim 25, further comprising:

dividing the input image into the input image information that comprises one or more input image blocks;

detecting texture samples in the input image; and

dividing the detected texture samples into the texture information that comprises one or more texture blocks, wherein an input image block and a texture block each has a block size larger than a block size of an output image block.

27. The non-transitory computer-readable medium of claim 25, wherein determining the lifting factor comprises:

estimating a texture strength value based on a number of texture samples in the input image; and

computing the lifting factor based on texture sample thresholds as a function of the texture strength value.

28. The non-transitory computer-readable medium of claim 27, wherein the frequency domain information comprises the transform coefficient, and the particular frequency components are created based on:

using the shifted transform coefficient for determining the distance from the shifted transform coefficient to the zero-frequency coefficient position based on using associated coefficient row and column indexes, wherein said shifting of the transform coefficient is performed in a matrix;

performing the frequency lifting on the shifted transform coefficient based on the determined distance for lifting one or more frequency components to a higher frequency location in a transform spectrum, wherein the determined distance is adjusted using determined lifting factors; and

converting the shifted transform coefficient to a modified transform coefficient.

29. The non-transitory computer-readable medium of claim 27, wherein the frequency domain information comprises the transform coefficient, and the particular frequency components are created based on:

using the shifted transform coefficient for determining the distance from the shifted transform coefficient to the zero-frequency coefficient position based on using associated coefficient row and column indexes;

performing the frequency lifting on the transform coefficient based on the determined distance for lifting one or more frequency components to a higher frequency location in a transform spectrum, wherein the determined distance is adjusted using determined lifting factors; and

converting the shifted transform coefficient to a modified transform coefficient.

30. The non-transitory computer-readable medium of claim 29, further comprising:

moving the zero-frequency coefficient position to a center of the matrix after said shifting, wherein the zero-frequency coefficient position is used as an origin of a two-dimensional dimensional coordinate, and matrix elements are used as points on the two-dimensional coordinate.

31. The non-transitory computer-readable medium of claim 30, wherein determining distances for transform coefficients is based on a pre-determined transfer function, and transforming received image information comprises using one of discrete Fourier transform processing, discrete cosine transform processing and discrete sign transform processing.

32. The non-transitory computer-readable medium of claim 25, wherein an output image that comprises the image details and the input image is displayed on an electronic device, and the electronic device comprises a television device, a monitor device, a computing device or a mobile computing device.

33. The non-transitory computer-readable medium of claim 25, wherein the computer performs processing on a local television device, a set-top-box device, or a network server device.