Image processing device, image processing method, image processing system, program, storage medium, and integrated circuit

• US 8,131,098 B2
• Filed: 07/03/2008
• Issued: 03/06/2012
• Est. Priority Date: 07/06/2007
• Status: Expired due to Fees

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Watanabe, Tatsumi, Ojima, Shuichi
• Primary Examiner(s): Newman, Michael A
• Application Number:
  US12/167,607
• Publication Number:
  US 20090022396A1
• Time in Patent Office: 1,342 Days
• Field of Search: 382/254, 382/106, 382/154, 382/171, 382/168, 382/181, 382/255, 382/266, 382/281, 345/422, 345/545
• US Class Current: 382/254
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/507 from shading G06T7/586 takes precedence
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06V 10/48 by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation

Abstract

The present invention executes color correction that improves the feeling of depth of a 2D image with ease and by using a preexisting device. Input image data is first converted into brightness information by a brightness information calculation portion. The interest level within the image is then estimated by an interest level estimation portion based on that information. The vanishing point is then estimated by a vanishing point estimation portion. Next, a depth estimation portion estimates the degree of depth based on the distance from the vanishing point to a pixel i and the interest level of the pixel i, and calculates a depth correction gain value. A corrected image, obtained by controlling a depth correction image process based on the depth correction gain value, is converted to a predetermined image format and outputted by an output portion.

Claims

1. An image processing device comprising:

a brightness information calculation portion operable to calculate brightness information of an image signal that can form an image made up of pixels, the calculation being performed per pixel or per pixel region made up of a plurality of pixels;

an interest level estimation portion operable to estimate an interest level indicating the degree to which the pixel or the pixel region is visually focused on in the image, based on the brightness information;

a vanishing point estimation portion operable to estimate vanishing point information of the image, based on the brightness information;

a depth estimation portion operable to (i) estimate a degree of depth based on the interest level and the vanishing point information and (ii) calculate a correction gain for performing a depth correction process on the image signal based on the degree of depth;

a depth correction portion operable to perform the depth correction process on the image signal based on the correction gain; and

a processor operable to control at least one of the brightness information calculation portion, that interest level estimation portion, the vanishing point estimation portion, the depth estimation portion, and the depth correction portion,

wherein the vanishing point estimation portion includes;

a border calculation portion operable to calculate border information based on the brightness information and extract characteristic points in border portions within the image based on the border information;

a Hough transform portion operable to convert straight lines in the image that connect the characteristic points into points within a Hough space by performing a Hough transform;

a distribution calculation portion operable to calculate a histogram distribution for the points within the Hough space; and

a vanishing point extraction portion operable to extract the vanishing point information of the image based on the histogram distribution for the points in the Hough space.

2. The image processing device according to claim 1,

wherein the interest level estimation portion includes:

a surrounding brightness information calculation portion operable to calculate surrounding brightness information AKi based on the brightness information of pixels surrounding a pixel of interest, the pixel of interest being the pixel in the image signal that is to be processed;

a brightness contrast amount calculation portion operable to calculate a brightness contrast amount RKi based on the surrounding brightness information AKi and brightness information of interest Ki, the brightness information of interest Ki being the brightness information of the pixel of interest; and

an interest level conversion portion operable to find the interest level based on the brightness contrast amount RKi.

3. The image processing device according to claim 2,

wherein the brightness contrast amount calculation portion calculates the brightness contrast amount RKi using the formula (brightness contrast amount RKi)=(brightness information of interest Ki)/(surrounding brightness information AKi), and

wherein the interest level conversion portion sets the interest level to a greater value as the value of the brightness contrast amount RKi increases.

4. The image processing device according to claim 1,

wherein the vanishing point extraction portion:

divides the image into a plurality of blocks; and

calculates, using the histogram distribution calculated by the distribution calculation portion, the number of straight lines within a block, the number being the number of times a straight line in the image corresponding to the points in the Hough space having a frequency value greater than a first threshold passes through the block, and uses the central point of the block as the vanishing point information of the image in the case where the number of straight lines within a block is greater than a predetermined value.

5. The image processing device according to claim 2,

wherein the depth estimation portion:

sets a degree of depth minimum value Min_fbi, which is the minimum value of the degree of depth, and a degree of depth maximum value Max_fbi (≧

Min_fbi), which is the maximum value of the degree of depth, to values in a relationship in which they monotonically increase with respect to the distance between the pixel of interest and the vanishing point information in the image;

sets the degree of depth (fbi) to a value, within the range between the degree of depth minimum value Min_fbi and the degree of depth maximum value Max_fbi, in a relationship that monotonically increases with respect to the interest level; and

sets the correction gain to a value in a relationship that monotonically increases with respect to the degree of depth (fbi), and

wherein the depth correction portion performs the depth correction process on the image signal so that the greater the value of the correction gain becomes, the greater the degree to which the pixel of interest is visually focused on becomes.

6. The image processing device according to claim 1, further comprising:

a color information calculation portion operable to calculate color information per pixel or per pixel region in the image signal,

wherein the interest level estimation portion estimates the interest level based on the brightness information and the color information.

7. The image processing device according to claim 1, further comprising:

a frequency information calculation portion operable to calculate frequency information per block made up of plural pixels or plural pixel regions in the image signal,

wherein the interest level estimation portion estimates the interest level based on the frequency information in addition to the brightness information.

8. The image processing device according to claim 7,

wherein the interest level estimation portion finds, per block, a high-frequency component sum sumHFreq, which is the sum of frequency components greater than or equal to a predetermined frequency, finds a ratio RBi of the high-frequency component sum sumHFreq to an all-frequency component sum sumAllFreq of the block (=sumHFreq/sumAllFreq), and estimates the interest level based on the ratio RBi.

9. The image processing device according to claim 1, further comprising:

a storage portion operable to store past frame images, which are the image in p number of frame units from one frame previous to the current frame up to p frames previous to the current frame (where p is a natural number), and the vanishing point information in the past frame images;

a vanishing point estimation judgment portion operable to (i) find a past frame image characteristic amount, which is an image characteristic amount of each of the p past frame images stored in the storage portion, (ii) find a current frame image characteristic amount, which is an image characteristic amount of the current frame image, the current frame image being the image in frame units in the current frame, (iii) identify the past frame image for which the difference between the current frame image characteristic amount and the past frame image characteristic amount is minimum as a minimum error past frame image, and (iv) output vanishing point replacement determination information in the case where the difference between the past frame image characteristic amount of the minimum error past frame image and the current frame image characteristic amount is less than a predetermined value; and

a vanishing point selection portion operable to set the vanishing point information of the minimum error past frame image as the vanishing point information of the current frame image in the case where the vanishing point determination information has been outputted by the vanishing point estimation judgment portion,

wherein the depth estimation portion estimates the degree of depth based on the vanishing point information set by the vanishing point selection portion and the interest level.

10. The image processing device according to claim 9,

wherein the vanishing point estimation portion does not execute the process for estimating the vanishing point information based on the current frame image in the case where the vanishing point determination information has been outputted by the vanishing point estimation judgment portion.

11. The image processing device according to claim 9,

wherein the vanishing point selection portion sets smoothed vanishing point information, in which the vanishing point information of the p past frame images has been smoothed, as the vanishing point information of the current frame image, in the case where the vanishing point determination information has not been outputted by the vanishing point estimation judgment portion, and

wherein the vanishing point estimation portion does not execute the process for estimating the vanishing point information based on the current frame image.

12. The image processing device according to claim 9,

wherein the vanishing point estimation judgment portion identifies the past frame image for which square error between the current frame image characteristic amount and the past frame image characteristic amount is minimum as the minimum error past frame image, and outputs vanishing point replacement determination information in the case where the square error between the past frame image characteristic amount of the minimum error past frame image and the current frame image characteristic amount is less than a predetermined value.

13. The image processing device according to claim 12,

wherein the image characteristic amount is calculated based on at least one of a brightness contrast amount, a color contrast amount, and an amount based on frequency information calculated per block made up of a plurality of pixels or a plurality of pixel regions.

14. The image processing device according to claim 13,

wherein the amount based on the frequency information is a ratio RBi of the high-frequency component sum sumHFreq, which is the sum of frequency components greater than or equal to a predetermined frequency found per block, to an all-frequency component sum sumAllFreq of the block (=sumHFreq/sumAllFreq).

15. The image processing device according to claim 1,

wherein the depth correction portion performs at least one of a brightness information correction process, a color information correction process, a border enhancement process, and a smoothing process as the depth correction process.

16. An image processing system comprising:

the image processing device according to claim 1;

a user mode selection portion operable to select a processing mode in accordance with an instruction from a user; and

a display portion operable to display the output from the image processing device as an image.

17. The image processing system according to claim 16,

wherein the user mode selection portion selects the processing mode that includes at least information regarding strength of the depth correction process executed by the image processing device.

18. An image processing method comprising:

calculating brightness information of an image signal that can form an image made up of pixels, the calculation being performed per pixel or per pixel region made up of a plurality of pixels;

estimating an interest level indicating the degree to which the pixel or the pixel region is visually focused on in the image, based on the brightness information;

estimating vanishing point information of the image, based on the brightness information;

estimating a degree of depth based on the interest level and the vanishing point information and calculating a correction gain for performing a depth correction process on the image signal based on the degree of depth; and

performing a depth correction process on the image signal based on the correction gain,

wherein the estimating vanishing point information includes;

calculating border information based on the brightness information and extracting characteristic points in portions within the image based on the border information;

converting straight lines in the image that connect the characteristic points into points within a Hough space by performing a Hough transform;

calculating a histogram distribution for the points within the Hough space; and

extracting the vanishing point information of the image based on the histogram distribution for the points in the Hough space.

19. A non-transitory computer-readable storage medium having stored thereon a program, wherein, when executed, the program causes a computer to function as:

a brightness information calculation portion operable to calculate brightness information of an image signal that can form an image made up of pixels, the calculation being performed per pixel or per pixel region made up of a plurality of pixels;

an interest level estimation portion operable to estimate an interest level indicating the degree to which the pixel or the pixel region is visually focused on in the image, based on the brightness information;

a vanishing point estimation portion operable to estimate vanishing point information of the image, based on the brightness information;

a depth estimation portion operable to (i) estimate a degree of depth based on the interest level and the vanishing point information and (ii) calculate a correction gain for performing a depth correction process on the image signal based on the degree of depth;

a depth correction portion operable to perform the depth correction process on the image signal based on the correction gain,

wherein the vanishing point estimation portion includes;

a border calculation portion operable to calculate border information based on the brightness information and extract characteristic points in border portions within the image based on the border information;

a Hough transform portion operable to convert straight lines in the image that connect the characteristic points into points within a Hough space by performing a Hough transform;

a distribution calculation portion operable to calculate a histogram distribution for the points within the Hough space; and

a vanishing point extraction portion operable to extract the vanishing point information of the image based on the histogram distribution for the points in the Hough space.