IMAGE PROCESSING APPARATUS AND METHOD

• US 20110129165A1
• Filed: 11/29/2010
• Published: 06/02/2011
• Est. Priority Date: 11/27/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Choe, Won-Hee, LIM, Jae-Guyn, Lee, Seong-Deok, Kang, Joo-Young, Park, Byung-Kwan
• Granted Patent: US 8,929,677 B2
• Time in Patent Office: Days
• US Class Current: 382/255
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

An image processing apparatus and image processing method capable of providing a high-resolution image for a desired position by use of light field data are provided. The image processing apparatus may determine a position of a portion of light field data among light field data corresponding to a scene, according to a desired focusing position, generate a refocusing first image by use of the position determined portion of light field data, generate a high-resolution image having a predetermined setting ratio relative to the refocusing first image, determine a ratio for local synthesis between the high-resolution image and an enlarged image of the refocusing first image by use of a similarity with respect to the position determined portion of light field data, and generate a synthesized image by synthesizing the high-resolution image and the enlarged image of the refocusing first image according to the local synthesis ratio.

Claims

1. An image processing apparatus, comprising:

a light field data position determiner configured to determine a position of a portion of light field data among light field data corresponding to a scene, according to a desired focusing position;

a refocusing image generator configured to generate a refocusing first image by use of the position determined portion of light field data;

a high resolution image generator configured to generate a high-resolution image comprising a predetermined setting ratio, relative to the refocusing first image;

a synthesis ratio determiner configured to determine a ratio for local synthesis between the high-resolution image and an enlarged image of the refocusing first image, by use of a similarity with respect to the position determined portion of light field data; and

an image synthesizer configured to generate a synthesized image by synthesizing the high-resolution image and the enlarged image of the refocusing first image, according to the local synthesis ratio.

2. The image processing apparatus of claim 1, wherein the light field data position determiner is further configured to:

set a second image, corresponding to a part of an image representing the scene, and comprising the desired focusing position;

generate a plurality of refocusing images for the second image while changing a varying alpha value, the varying alpha value representing a relationship between a photographic plane and a refocusing plane;

determine a final alpha value, from the varying alpha value, the final alpha value allowing a highest sharpness refocusing image to be generated for the second image; and

determine the position of the portion of light field data by use of the determined final alpha value.

3. The image processing apparatus of claim 1, wherein the synthesis ratio determiner is further configured to determine the ratio for local synthesis, such that the high-resolution image is synthesized with a higher synthesis ratio for a portion comprising a higher similarity with respect to the position-determined portion of light field data.

4. The image processing apparatus of claim 1, wherein the synthesis ratio determiner comprises:

a non-uniformity map generator configured to generate a non-uniformity map representing a non-uniformity of intensity of light field data at the determined position;

a ratio non-uniformity map generator configured to generate a ratio non-uniformity map comprising information about the ratio for local synthesis between the high-resolution image and the enlarged image of the refocusing first image, by use of the non-uniformity map; and

a ratio non-uniformity map enlarger configured to enlarge the ratio non-uniformity map at the setting ratio.

5. The image processing apparatus of claim 1, wherein the image synthesizer is further configured to:

enlarge the refocusing first image at the setting ratio; and

synthesize the enlarged image of the refocusing first image and the high-resolution image at the determined ratio for local synthesis.

6. The image processing apparatus of claim 2, wherein the high-resolution image generator comprises:

an image frame determiner configured to determine a standard image frame and at least one reference image frames, by use of the position determined portion of light field data comprising the position, the position being determined by the determined final alpha value;

a point spread function determiner configured to determine a point spread function, based on an amount of sub-pixel displacement of the at least one reference image frame with respect to the standard image frame;

an image interpolator configured to generate a high-resolution standard image frame comprising a higher resolution than the standard image frame by interpolating the standard image frame; and

an image restorer configured to restore the high-resolution image by updating the high-resolution standard image frame using the generated high-resolution standard image frame, the point spread function, and the at least one reference image frame.

7. The image processing apparatus of claim 6, wherein the amount of sub-pixel displacement represents a position difference between light field data corresponding to the standard image frame and light field data corresponding to each reference image frame.

8. The image processing apparatus of claim 6, wherein the image frame determiner is further configured to:

determine a view image seen from an angle as the standard image frame; and

determine at least one view image seen from another angle as the reference image frame by use of the position determined portion of light field data comprising the position, the position being determined by the final alpha value.

9. The image processing apparatus of claim 6, wherein the point spread function determiner is further configured to determine, as the point spread function, a two-dimensional Gaussian function, based on the amount of sub-pixel displacement of the each reference image frame with respect to the standard image frame.

10. The image processing apparatus of claim 6, wherein the image restorer comprises:

a residual value generator configured to generate a residual value by use of the generated high-resolution standard image frame, one of the reference image frames, and a point spread function based on the one reference image frame and the standard image frame; and

an image updater configured to update the high-resolution standard image frame by use of the residual value.

11. The image processing apparatus of claim 10, wherein the residual value comprises a value of the one reference image frame minus a convolution of the high-resolution standard image frame with the point spread function.

12. The image processing apparatus of claim 10, wherein, if the high-resolution standard image frame is updated, the residual value generator is further configured to generate a residual value by use of the updated high-resolution standard image frame, another one of the at least one reference image frame, and a point spread function based on the another one of the at least one reference image frame and the standard image frame.

13. The image processing apparatus of claim 12, wherein the residual value comprises a value of the another one of the at least one reference image frame minus a convolution of the updated high-resolution standard image frame with the point spread function.

14. The image processing apparatus of claim 1, further comprising a light field data capturer, the light field data capturer comprising:

a first optical unit configured to form an image of an object;

a photo sensor array configured to capture light rays; and

a second optical unit disposed between the first optical unit and the photo sensor array, and configured to direct the light rays toward the photo sensor array by separating the light rays based on the direction of the light rays.

15. An image processing method, the method comprising:

determining a position of a portion of light field data among light field data corresponding to a scene, according to a desired focusing position;

generating a refocusing first image by use of the position determined portion of light field data;

generating a high-resolution image comprising a predetermined setting ratio, relative to the refocusing first image;

determining a ratio for local synthesis between the high-resolution image and an enlarged image of the refocusing first image, by use of a similarity with respect to the position determined portion of light field data; and

generating a synthesized image by synthesizing the high-resolution image and the enlarged image of the refocusing first image, according to the local synthesis ratio.

16. The image processing method of claim 15, wherein the determining of the position of the portion of light field data comprising:

setting a second image corresponding to a part of an image representing the scene and comprising the desired focusing position;

generating a plurality of refocusing images for the second image while changing a varying alpha value, the varying alpha value representing a relationship between a photographic plane and a refocusing plane;

determining a final alpha value, from the varying alpha value, the final alpha value allowing a highest sharpness refocusing image to be generated for the second image; and

determining the position of the portion of light field data by use of the determined alpha value.

17. The image processing method of claim 15, wherein, in the determining of the ratio for local synthesis, the ratio for local synthesis is determined such that the high-resolution image is synthesized with a higher synthesis ratio for a portion comprising a higher similarity with respect to the position-determined portion of light field data.

18. The image processing method of claim 15, wherein the determining of the ratio for local synthesis comprises:

generating a non-uniformity map representing a non-uniformity of intensity of light field data at the position determined by an alpha value representing a relationship between a photographic plane and a refocusing plane;

generating a ratio non-uniformity map including information about the ratio for local synthesis between the high-resolution image and the enlarged image of the refocusing first image by use of the non-uniformity map; and

enlarging the ratio non-uniformity map at the setting ratio.

19. The image processing method of claim 15, wherein the generating of the synthesized image comprises:

enlarging the refocusing first image at the setting ratio; and

synthesizing the enlarged image of the refocusing first image and the high-resolution image at the determined ratio for local synthesis.

20. The image processing method of claim 16, wherein the generating of the high-resolution image comprises:

determining a standard image frame and at least one reference image frames, by use of the determined alpha value;

determining a point spread function, based on an amount of sub-pixel displacement of the at least one reference image frame with respect to the standard image frame;

generating a high-resolution standard image frame comprising a higher resolution than the standard image frame, by interpolating the standard image frame; and

restoring the high-resolution image by updating the high-resolution standard image frame using the generated high-resolution standard image frame, the point spread function, and the at least one reference image frame.

23. A computer-readable information storage medium storing a program for causing a computer to implement the method of claim 15.

21. A method of processing an acquired four-dimensional (4D) light field, the method comprising:

determining a desired focusing position and estimating an alpha value for the focusing position, the alpha value representing a relationship between a photographic plane and a refocusing plane;

generating a refocusing image from angular data according to the alpha value;

generating a high-resolution image by increasing a resolution of a focusing region, based on a sub-pixel displacement of the angular data according to the alpha value;

dividing the focusing region from an out-of-focus region by checking a similarity of the angular data according to the alpha value; and

performing fusion on the refocusing image and the high-resolution image.

22. The method of claim 21, wherein the alpha value is determined by varying the alpha value until a highest sharpness refocusing image is generated.

24. A computer-readable information storage medium storing a program for causing a computer to implement the method of claim 21.

25. An image processing method, comprising:

from light field data corresponding to a scene, generating a refocusing image corresponding to a part of an image representing the scene, the refocusing image including a desired focusing position;

generating an enlarged image of the refocusing image;

generating a high resolution image corresponding to the enlarged image using the light field data corresponding to the part of the image; and

generating a synthesized image based on the high resolution image and the enlarged image.

26. The method of claim 25, wherein the refocusing image comprises a highest sharpness among a plurality of images corresponding to the part of the image.

27. The method of claim 25, wherein the synthesized image is generated by synthesizing the high resolution image and the enlarged image according to a calculated ratio.