Image processing device, image processing method, program, storage medium and integrated circuit

• US 20070273686A1
• Filed: 05/23/2007
• Published: 11/29/2007
• Est. Priority Date: 05/23/2006
• Status: Active Grant

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Ojima, Shuichi, Watanabe, Tatsumi
• Granted Patent: US 8,164,594 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 15/60 Shadow generation

Abstract

It is an object to easily, and using existing devices, perform shadow enhancement that achieves an increase in the feeling of depth of 2D video. The input image data are first converted into brightness information by a brightness information calculation portion. Then, based on that brightness information, the normal direction and the edge information in the pixel targeted for processing are estimated by a normal direction estimation portion. A corrected gradation derivation portion then performs correction processing such as the addition of shadow component on the input image based on the brightness information and the estimated normal direction and edge information to create a processed image that has a feeling of depth, and then an output portion converts this to a predetermined image format and outputs it. In this way, it is possible to easily increase the feeling of depth of a 2D image through the addition of shadow, for example, in accordance with the characteristics of the input image.

Claims

1. An image processing device comprising:

a brightness information calculation portion that calculates brightness information of an image signal that is input and that can form a two-dimensional image made from pixels;

a normal direction estimation portion that estimates a normal direction for each pixel, or each region made of a plurality of pixels, from the brightness information in order to give a feeling of depth to the two-dimensional image that is formed by the image signal, and obtains a normal direction vector that is a three-dimensional vector for each pixel or for each region based on the normal direction that is estimated; and

a corrected gradation derivation portion that finds a correction amount of the brightness information based on the normal direction vector that is obtained by the normal direction estimation portion, and corrects the brightness information of the image signal based on the correction amount of the brightness information.

2. The image processing device according to claim 1, further comprising:

an output portion that outputs the image signal whose brightness information has been corrected by the corrected gradation derivation portion in a predetermined format.

3. The image processing device according to claim 1,

wherein the corrected gradation derivation portion sets at least one virtual light source and determines a light source position of the virtual light source, and corrects the brightness information of the image signal based on the normal direction vector and the positional relationship between the pixel position of the pixel and the light source position.

4. The image processing device according to claim 1,

wherein the corrected gradation derivation portion sets at least one virtual light source and finds a light source direction vector, which is a three-dimensional vector that has a direction toward the virtual light source, for each pixel or for each region, and corrects the brightness information of the image signal based on the normal direction vector and the light source direction vector.

5. The image processing device according to claim 4,

wherein the corrected gradation derivation portion corrects the brightness information of the image signal by adding shadow to the pixel value of the pixel.

6. The image processing device according to claim 4,

wherein the corrected gradation derivation portion corrects the brightness information of the image signal by adding light to the pixel value of the pixel.

7. The image processing device according to claim 1,

wherein the normal direction estimation portion determines the normal direction vector from the values of the three direction components, these being a first direction component value that is the value of the component of a first direction and a second direction component value that is the value of the component of a second direction, which serve as two-dimensional coordinate information for determining a pixel position of the pixel on the two-dimensional image, and a third direction component value that is a value of a component of a third direction that is not included on a plane that is formed by the two-dimensional image.

8. The image processing device according to claim 7,

wherein the normal direction estimation portion sets the third direction component value to a first value that is a predetermined value, and for a plurality of pixels sets the third direction component value to the same value as the first value and determines the normal direction value for each pixel or for each region.

9. The image processing device according to claim 7,

wherein the normal direction estimation portion determines the first direction component value of the normal direction vector based on a differential value for the first direction of the pixel value of the pixel.

10. The image processing device according to claim 7,

wherein the normal direction estimation portion determines the second direction component value of the normal direction vector based on a differential value for the second direction of the pixel value of the pixel.

11. The image processing device according to claim 7,

wherein the normal direction estimation portion determines the first direction component value of the normal direction vector based on a differential value of the first direction of the pixel value of the pixel, and determines the second direction component value of the normal direction vector based on a differential value of the second direction of the pixel value of the pixel;

and

wherein the corrected gradation derivation portion finds the correction amount of the brightness information based on the modulated normal direction vector, and corrects the brightness information of the image signal based on the correction amount of the brightness information.

12. The image processing device according to claim 11,

wherein the correction amount of the brightness information changes along with a change in the first direction component value or the second direction component value of the normal direction vector.

13. The image processing device according to claim 1,

wherein the normal direction estimation portion obtains a modulated normal direction vector by modulating the normal direction vector;

and

wherein the corrected gradation derivation portion finds a correction amount of the brightness information based on the modulated normal direction vector, and corrects the brightness information of the image signal based on the correction amount of the brightness information.

14. The image processing device according to claim 7,

wherein the normal direction estimation portion obtains a modulated normal direction vector by modulating the normal direction vector;

and

wherein the corrected gradation derivation portion finds a correction amount of the brightness information based on the modulated normal direction vector, and corrects the brightness information of the image signal based on the correction amount of the brightness information.

15. The image processing device according to claim 14,

wherein the normal direction estimation portion obtains the modulated normal direction vector by modulating the normal direction vector based on a differential value for the first direction of the pixel value of the pixel.

16. The image processing device according to claim 14,

wherein the normal direction estimation portion obtains the modulated normal direction vector by modulating the normal direction vector based on a differential value for the second direction of the pixel value of the pixel.

17. The image processing device according to claim 14,

wherein the normal direction estimation portion obtains the modulated normal direction vector by modulating the normal direction vector based on a differential value for the first direction and the second direction of the pixel value of the pixel.

18. The image processing device according to claim 14,

wherein the normal direction estimation portion calculates a brightness contrast amount based on the brightness information of a pixel of interest, which is the pixel that is targeted for processing in the image signal, and the brightness information of pixels surrounding the pixel of interest, obtains the modulated normal direction vector by changing the third direction component value of the normal direction vector based on the brightness contrast amount.

19. The image processing device according to claim 14,

wherein the normal direction estimation portion calculates a brightness contrast amount based on the brightness information of a pixel of interest, which is the pixel that is targeted for processing in the image signal, and the brightness information of the pixels surrounding the pixel of interest, calculates a first visual depth amount that indicates a degree of perception of visual depth based on the brightness contrast amount, and obtains the modulated normal direction vector by changing the first direction component value and the second direction component value of the normal direction vector based on the first visual depth amount.

20. The image processing device according to claim 14,

wherein the normal direction estimation portion calculates a color contrast amount based on color information of a pixel of interest, which is the pixel that is targeted for processing in the image signal, and the color information of pixels surrounding the pixel of interest, calculates a second visual depth amount that indicates a degree of perception of visual depth based on the color contrast amount, and obtains the modulated normal direction vector by changing the first direction component value and the second direction component value of the normal direction vector based on the second visual depth amount.

21. The image processing device according to claim 14,

wherein the normal direction estimation portion calculates a brightness contrast amount based on the brightness information of a pixel of interest, which is the pixel that is targeted for processing in the image signal, and the brightness information of pixels surrounding the pixel of interest, and calculates a first visual depth amount that indicates a degree of perception of visual depth based on the brightness contrast amount;

calculates a color contrast amount based on color information of a pixel of interest, which is the pixel that is targeted for processing in the image signal, and the color information of pixels surrounding the pixel of interest, and calculates a second visual depth amount that indicates a degree of perception of visual depth based on the color contrast amount; and

obtains the modulated normal direction vector by changing the first direction component value and the second direction component value of the normal direction vector based on the first visual depth amount and the second visual depth amount.

22. The image processing device according to claim 4,

wherein the normal direction estimation portion obtains a modulated normal direction vector by modulating the normal direction vector;

and

wherein the corrected gradation derivation portion finds a correction amount of the brightness information based on the modulated normal direction vector and the light source direction vector, and corrects the brightness information of the image signal based on the correction amount of the brightness information.

23. The image processing device according to claim 22,

wherein the normal direction estimation portion modulates the normal direction vector based on an angle α

that is formed between the normal direction vector and the light source direction vector.

24. The image processing device according to claim 23,

wherein the corrected gradation derivation portion increases the correction amount of the brightness information as the angle α

becomes larger.

25. The image processing device according to claim 23,

wherein the corrected gradation derivation portion

increases the correction amount of the brightness information as the angle α

becomes larger, in a case where the angle α

is larger than α

0, which is the angle that is formed by the normal vector and the light source direction vector of the two-dimensional image, and equal to or less than a first threshold angle α

1; and

decreases the correction amount as the angle α

becomes larger in a case where the angle α

is larger than the first threshold angle α

1.

26. The image processing device according to claim 25,

wherein the corrected gradation derivation portion corrects the brightness information of the image signal with the correction amount by adding shadow to the pixel value of the pixel.

27. The image processing device according to claim 13, further comprising:

an input portion that can input information for determining a characteristic by which to modulate the normal direction vector.

28. The image processing device according to claim 4,

wherein the corrected gradation derivation portion corrects the brightness information of the image signal based on the scalar product of the light source direction vector and the normal direction vector.

29. The image processing device according to claim 4,

wherein the corrected gradation derivation portion corrects the brightness information of the image signal by applying the light source direction vector and the normal direction vector to a mathematical model or lookup table to obtain a shadow-enhanced image signal and then synthesizing the shadow-enhanced image signal with the image signal.

30. The image processing device according to claim 4,

wherein the corrected gradation derivation portion corrects the brightness information of the image signal by adopting the light source direction vector and the normal direction vector in a mathematical model or lookup table to obtain a light-enhanced image signal and then synthesizing the light-enhanced image signal with the image signal.

31. The image processing device according to claim 4,

wherein the corrected gradation derivation portion corrects the brightness information of the image signal by adopting the light source direction vector and the normal direction vector in a mathematical model or lookup table to obtain a shadow-enhanced image signal and a light-enhanced image signal, and then synthesizing the shadow-enhanced image signal and the light-enhanced image signal with the image signal.

32. The image processing device according to claim 1,

wherein the normal direction estimation portion finds a smoothed normal direction vector that is obtained by smoothing the normal direction vector in the space direction;

and

wherein the corrected gradation derivation portion corrects the brightness information of the image signal based on the smoothed normal direction vector.

33. The image processing device according to claim 1,

wherein the normal direction estimation portion finds a smoothed normal direction vector that is obtained by smoothing the normal direction vector in the time direction;

and

wherein the corrected gradation derivation portion corrects the brightness information of the image signal based on the smoothed normal direction vector.

40. The image processing device according to claim 2, further comprising:

a user mode selection portion that allows a processing mode to be selected by user command; and

a display portion that displays the output from the output portion as an image.

41. The image processing device according to claim 40,

wherein the user mode selection portion allows selection of at least a processing mode that includes information relating to a strength of correction of the brightness information.

34. An image processing method, comprising steps of:

calculating brightness information of an image signal that is input and that can form a two-dimensional image made from pixels;

setting a coordinate (c1,c2,c3) (where c1, c2, and c3 are real numbers) of a light source position;

calculating a difference x between the brightness information of the pixel at a predetermined coordinate position (p1,p2) (where p1 and p2 are real numbers) on the plane formed by the two-dimensional image and the brightness information of a pixel that is adjacent in the x direction on the plane, and a difference y between the brightness information of the pixel at the predetermined coordinate position (p1,p2) and the brightness information of a pixel that is adjacent in the y direction on the plane;

calculating a normal direction vector (said difference x, said difference y, c3) from the difference x and the difference y that are calculated in the difference calculation step;

calculating a light source direction vector (c1-p1, c2-p2, c3) based on the light source position and the pixel position of the pixel (p1,p2); and

correcting the brightness information of the pixel based on the normal direction vector and the light source direction vector.

35. The image processing method according to claim 34,

wherein in a case where the light source position is at an infinite parallel position with respect to the pixel position of the pixel, such as sunlight, then the coordinate (c1,c2,c3) of the light source position is a value that has a predetermined size in the direction of the infinite parallel position, which is the light source position.

36. An image processing method, comprising steps of:

calculating brightness information of an image signal that is input and that can form a two-dimensional image made from pixels;

estimating a normal direction for each pixel or for each region made of a plurality of pixels, from the brightness information in order to give a feeling of depth to the two-dimensional image that is formed by the image signal, and obtaining a normal direction vector that is a three-dimensional vector for each pixel or for each region based on the normal direction that is estimated; and

finding a correction amount of the brightness information based on the normal direction vector that is obtained in the normal direction estimation step, and correcting the brightness information of the image signal based on the correction amount of the brightness information.

37. A program for causing a computer to function as:

a brightness information calculation portion that calculates brightness information of an image signal that is input and that can form a two-dimensional image made from pixels;

a normal direction estimation portion that estimates a normal direction for each pixel, or each region made of a plurality of pixels, from the brightness information in order to give a feeling of depth to the two-dimensional image that is formed by the image signal, and obtains a normal direction vector that is a three-dimensional vector for each pixel or for each region based on the normal direction that is estimated; and

a corrected gradation derivation portion that finds a correction amount of the brightness information based on the normal direction vector that is obtained by the normal direction estimation portion, and corrects the brightness information of the image signal based on the correction amount of the brightness information.

38. A computer-readable storage medium on which is stored a program for causing a computer to function as:

a brightness information calculation portion that calculates brightness information of an image signal that is input and that can form a two-dimensional image made from pixels;

a normal direction estimation portion that estimates a normal direction for each pixel, or each region made of a plurality of pixels, from the brightness information in order to give a feeling of depth to the two-dimensional image that is formed by the image signal, and obtains a normal direction vector that is a three-dimensional vector for each pixel or for each region based on the normal direction that is estimated; and

a corrected gradation derivation portion that finds a correction amount of the brightness information based on the normal direction vector that is obtained by the normal direction estimation portion, and corrects the brightness information of the image signal based on the correction amount of the brightness information.

39. An integrated circuit, comprising:

a brightness information calculation portion that calculates brightness information of an image signal that is input and that can form a two-dimensional image made from pixels;

a normal direction estimation portion that estimates a normal direction for each pixel, or each region made of a plurality of pixels, from the brightness information in order to give a feeling of depth to the two-dimensional image that is formed by the image signal, and obtains a normal direction vector that is a three-dimensional vector for each pixel or for each region based on the normal direction that is estimated; and

a corrected gradation derivation portion that finds a correction amount of the brightness information based on the normal direction vector that is obtained by the normal direction estimation portion, and corrects the brightness information of the image signal based on the correction amount of the brightness information.

42. An image processing device, comprising:

a brightness information calculation portion that derives brightness information from an image signal that forms a two-dimensional image;

a normal direction estimation portion that obtains a three-dimensional vector value for each pixel or for each region; and

a corrected gradation derivation portion that corrects a luminance value of the pixel or the region based on the three-dimensional vector value.