Image processing apparatus, image processing method, and computer-readable device having instructions for generating focused image

• US 9,503,629 B2
• Filed: 03/17/2011
• Issued: 11/22/2016
• Est. Priority Date: 03/19/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Suzuki, Naho
• Primary Examiner(s): He, Weiming
• Application Number:
  US13/050,340
• Publication Number:
  US 20110227950A1
• Time in Patent Office: 2,077 Days
• Field of Search: 348/340, 348/231.99
• US Class Current: 1/1
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

An image processing apparatus includes: a display section having the function of detecting a position touched by a pointer; an icon display control section for identifying a pixel of interest which is a pixel associated with the position of the pointer from among pixels forming an image displayed at the display section and displaying an icon for indicating the pixel of interest on the image displayed at the display section; and an image processing section for generating an image focused in a part thereof associated with the pixel of interest using distance data associated with the pixels forming the image.

Claims

1. An image processing apparatus comprising:

a display controller configured to;

control recording of a first image and metadata including depth information associated with the first image into a storage medium;

control display of the first image;

control display of an icon having a predetermined size and shape at a first pointed position on the first image at which a pointer is detected as having been pointed, the first pointed position corresponding to a first pixel of an object in the first image;

control display of the icon with the predetermined size and shape at a second pointed position on the first image at which the pointer is detected as having been pointed, the second pointed position corresponding to a second pixel of the object in the first image;

control display of a sub screen including an enlarged image of a first region of the first image, the first region including the second pixel of the object in the first image, wherein the sub screen is displayed overlapping the first image; and

control display of a sub screen icon identifying on the sub screen a position of the enlarged image corresponding to the second pointed position; and

an image processor configured to;

calculate a respective defocus amount to be applied to at least two pixels of a plurality of pixels of the first image based on depth information representing a plurality of distances between a lens and a plurality of portions of the object in the first image; and

generate a second image at least by applying the respective defocus amount to the at least two pixels such that at least a portion of the object in the second image is in focus, the at least a portion of the object in the second image corresponding to the second pointed position on the first image,

wherein;

the display controller is further configured to control display of the second image,

the display controller is further configured to control recording of the second image into the storage medium,

the display controller is further configured to, before the image processor generates the second image, control display of a prompt to a user to confirm selection of the second pointed position, and

the first image comprises an original image and the second image comprises a defocused image corresponding to the original image.

2. An image processing apparatus according to claim 1, wherein the sub screen is configured to display the enlarged image by altering display of a part of the first image.

3. An image processing apparatus according to claim 1, wherein the display controller is further configured to change a displayed position of the sub screen icon.

4. An image processing apparatus according to claim 1, wherein the display controller is further configured to control display of contour information, the contour information indicating a contour of the object, the contour enclosing the second pointed position.

5. An image processing apparatus according to claim 4, wherein the contour information is generated by using the depth information.

6. An image processing apparatus according to claim 5, wherein the contour information is generated based on a differentiated value of a depth information distribution of the first image being greater than a predetermined threshold.

7. An image processing apparatus according to claim 5, wherein the contour information is generated based on a double differentiated value of a depth information distribution of the first image being zero.

8. An image processing apparatus according to claim 4, wherein the contour information is generated by using luminance information of the first image.

9. An image processing apparatus according to claim 8, wherein the contour information is generated by detecting a pixel having luminance resulting in a luminance difference of greater than or equal to a predetermined value compared with an adjacent pixel.

10. An image processing apparatus according to claim 1, wherein at least a portion of the object in the first image is not in focus and corresponds to the first pointed position.

11. An image processing apparatus according to claim 1, wherein the image processor is configured to generate the second image by controlling blur.

12. An image processing apparatus according to claim 11, wherein the image processor is configured to control the blur in the second image based on the second pointed position.

13. An image processing apparatus according to claim 11, wherein the image processor is configured to control a value of the blur based on an operation input by a user.

14. An image processing apparatus according to claim 13, wherein the display controller is further configured to control display of a blur indicator indicating a value of the blur.

15. An image processing apparatus according to claim 1, wherein the first image comprises a frame of a movie.

16. An image processing apparatus according to claim 1, further comprising a display configured to display information received from the display controller and a touch sensor configured to detect the pointer.

17. An image processing apparatus according to claim 1, wherein at least a portion of the object in the first image is not in focus and corresponds to the second pointed position.

18. An image processing method comprising:

controlling recording of a first image and metadata including depth information associated with the first image into a storage medium;

controlling display of the first image;

controlling display of an icon having a predetermined size and shape at a first pointed position on the first image at which a pointer is detected as having been pointed, the first pointed position corresponding to a first pixel of an object in the first image;

controlling display of the icon with the predetermined size and shape at a second pointed position on the first image at which the pointer is detected as having been pointed, the second pointed position corresponding to a second pixel of the object in the first image;

controlling display of a sub screen including an enlarged image of a first region of the first image, the first region including the second pixel of the object in the first image, wherein the sub screen is displayed overlapping the first image;

controlling display of a sub screen icon identifying on the sub screen a position of the enlarged image corresponding to the second pointed position;

calculating a respective defocus amount to be applied to at least two pixels of a plurality of pixels of the first image based on depth information representing a plurality of distances between a lens and a plurality of portions of the object in the first image;

before generating a second image, controlling display of a prompt to a user to confirm selection of the second pointed position;

generating the second image at least by applying the respective defocus amount to the at least two pixels such that at least a portion of the object in the second image is in focus, the at least a portion of the object in the second image corresponding to the second pointed position on the first images;

controlling display of the second image; and

controlling recording of the second image into the storage medium,

wherein the first image comprises an original image and the second image comprises a defocused image corresponding to the original image.

19. An image processing method according to claim 18, further comprising displaying the enlarged image on the sub screen by altering display of a part of the first image.

20. An image processing method according to claim 18, further comprising changing a displayed position of the sub screen icon.

21. An image processing method according to claim 18, further comprising controlling display of contour information, the contour information indicating a contour of the object, the contour enclosing the second pointed position.

22. An image processing method according to claim 21, further comprising generating the contour information by using the depth information.

23. An image processing method according to claim 22, wherein generating the contour information comprises basing the contour information on a differentiated value of a depth information distribution of the first image being greater than a predetermined threshold.

24. An image processing method according to claim 22, wherein generating the contour information comprises basing the contour information on a double differentiated value of a depth information distribution of the first image being zero.

25. An image processing method according to claim 21, wherein generating the contour information comprises detecting a pixel having luminance resulting in a luminance difference of greater than or equal to a predetermined value compared with an adjacent pixel.

26. An image processing method according to claim 21, wherein the contour information is generated by using luminance information of the first image.

27. An image processing method according to claim 18, wherein at least a portion of the object in the first image is not in focus and corresponds to the first pointed position.

28. An image processing method according to claim 18, further comprising generating the second image by controlling blur.

29. An image processing method according to claim 28, further comprising controlling the blur in the second image based on the second pointed position.

30. An image processing method according to claim 28, further comprising controlling a value of the blur based on an operation input by a user.

31. An image processing method according to claim 30, further comprising controlling display of a blur indicator indicating a value of the blur.

32. An image processing method according to claim 18, wherein the first image comprises a frame of a movie.

33. An image processing method according to claim 18, wherein at least a portion of the object in the first image is not in focus and corresponds to the second pointed position.

34. A computer-readable device having instructions that, when executed by a processor, perform a method comprising:

controlling recording of a first image and metadata including depth information associated with the first image into a storage medium;

controlling display of the first image;

controlling display of an icon having a predetermined size and shape at a first pointed position on the first image at which a pointer is detected as having been pointed, the first pointed position corresponding to a first pixel of an object in the first image;

controlling display of the icon with the predetermined size and shape at a second pointed position on the first image at which the pointer is detected as having been pointed, the second pointed position corresponding to a second pixel of the object in the first image;

controlling display of a sub screen including an enlarged image of a first region of the first image, the first region including the second pixel of the object in the first image, wherein the sub screen is displayed overlapping the first image;

controlling display of a sub screen icon identifying on the sub screen a position of the enlarged image corresponding to the second pointed position;

calculating a respective defocus amount to be applied to at least two pixels of a plurality of pixels of the first image based on depth information representing a plurality of distances between a lens and a plurality of portions of the object in the first image;

before generating a second image, controlling display of a prompt to a user to confirm selection of the second pointed position;

generating the second image at least by applying the respective defocus amount to the at least two pixels such that at least a portion of the object in the second image is in focus, the at least a portion of the object in the second image corresponding to the second pointed position on the first image;

controlling display of the second image; and

controlling recording of the second image into the storage medium,

wherein the first image comprises an original image and the second image comprises a defocused image corresponding to the original image.

35. A computer-readable device according to claim 34, wherein the method further comprises displaying the enlarged image on the sub screen by altering display of a part of the first image.

36. A computer-readable device according to claim 34, wherein the method further comprises changing a displayed position of the sub screen icon.

37. A computer-readable device according to claim 34, wherein the method further comprises controlling display of contour information, the contour information indicating a contour of the object, the contour enclosing the second pointed position.

38. A computer-readable device according to claim 37, wherein the method further comprises generating the contour information by using the depth information.

39. A computer-readable device according to claim 38, wherein generating the contour information comprises basing the contour information on a differentiated value of a depth information distribution of the first image being greater than a predetermined threshold.

40. A computer-readable device according to claim 39, wherein generating the contour information comprises basing the contour information on a double differentiated value of a depth information distribution of the first image being zero.

41. A computer-readable device according to claim 38, wherein generating the contour information comprises detecting a pixel having luminance resulting in a luminance difference of greater than or equal to a predetermined value compared with an adjacent pixel.

42. A computer-readable device according to claim 37, wherein the contour information is generated by using luminance information of the first image.

43. A computer-readable device according to claim 34, wherein at least a portion of the object in the first image is not in focus and corresponds to the first pointed position.

44. A computer-readable device according to claim 34, wherein the method further comprises generating the second image by controlling blur.

45. A computer-readable device according to claim 44, wherein the method further comprises controlling the blur in the second image based on the second pointed position.

46. A computer-readable device according to claim 44, wherein the method further comprises controlling a value of the blur based on an operation input by a user.

47. A computer-readable device according to claim 46, wherein the method further comprises controlling display of a blur indicator indicating a value of the blur.

48. A computer-readable device according to claim 34, wherein the first image comprises a frame of a movie.

49. A computer-readable device according to claim 34, wherein at least a portion of the object in the first image is not in focus and corresponds the second pointed position.