Eye beautification

• US 8,520,089 B2
• Filed: 06/30/2010
• Issued: 08/27/2013
• Est. Priority Date: 07/30/2008
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Ciuc, Mihai, Capata, Adrian, Florea, Corneliu
• Primary Examiner(s): AGGARWAL, YOGESH K
• Application Number:
  US12/827,868
• Publication Number:
  US 20110002506A1
• Time in Patent Office: 1,154 Days
• Field of Search: 348/222.1, 382/118, 382/531, 382254-275
• US Class Current: 348/222.1
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30216 Redeye defect
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/181 involving edge growing; involving edge linking
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 1/62 Retouching, i.e. modification of isolated colours only or in isolated picture areas only
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

Sub-regions within a face image are identified to be enhanced by applying a localized smoothing kernel to luminance data corresponding to the sub-regions of the face image. An enhanced face image is generated including an enhanced version of the face that includes certain original pixels in combination with pixels corresponding to the one or more enhanced sub-regions of the face.

Claims

1. A method of enhancing an appearance of a face within a digital image, comprising using a processor in:

acquiring a digital image of a scene including a face, including capturing the image using a lens and an image sensor of a processing device that includes the processor, or receiving said image following capture by another device that includes a lens and an image sensor, or a combination thereof;

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the iris at an iris side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof; and

wherein the method further comprises finding a border point as a maximum gradient point along a horizontal line through an upright eye, and growing the border including adding immediate neighboring points characterized by largest horizontal intensity gradients.

2. The method of claim 1, wherein the enhancing of the iris comprises linear contrast stretching.

3. The method of claim 1, wherein the enhancing of the iris comprises spatial blurring.

4. The method of claim 1, wherein the method further comprises applying one or more localized color smoothing kernels to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

5. The method of claim 1, wherein the method further comprises applying noise reduction or enhancement, or both, to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

6. The method of claim 1, wherein the method further comprises identifying one or more further segments of a periphery of the sclera.

7. The method of claim 6, wherein the method further comprises enhancing pixels of the sclera within the periphery.

8. The method of claim 1, wherein the method further comprises enhancing pixels of the sclera at a sclera side of the border.

9. The method of claim 1, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

10. A method of enhancing an appearance of a face within a digital image, comprising using a processor in:

acquiring a digital image of a scene including a face, including capturing the image using a lens and an image sensor of a processing device that includes the processor, or receiving said image following capture by another device that includes a lens and an image sensor, or a combination thereof;

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the iris at an iris side of the border; and

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof; and

wherein the method further comprises identifying one or more further segments of a periphery of the sclera; and

wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

11. The method of claim 10, wherein the enhancing of the iris comprises linear contrast stretching.

12. The method of claim 10, wherein the enhancing of the iris comprises spatial blurring.

13. The method of claim 10, wherein the method further comprises applying one or more localized color smoothing kernels to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

14. The method of claim 10, wherein the method further comprises applying noise reduction or enhancement, or both, to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

15. The method of claim 10, wherein the method further comprises enhancing pixels of the sclera within the periphery.

16. The method of claim 10, wherein the method further comprises enhancing pixels of the sclera at a sclera side of the border.

17. The method of claim 10, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

18. A method of enhancing an appearance of a face within a digital image, comprising using a processor in:

acquiring a digital image of a scene including a face, including capturing the image using a lens and an image sensor of a processing device that includes the processor, or receiving said image following capture by another device that includes a lens and an image sensor, or a combination thereof;

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the sclera at a sclera side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof, and

wherein the method further comprises identifying one or more further segments of a periphery of the sclera; and

wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

19. The method of claim 18, wherein the method further comprises applying one or more localized color smoothing kernels to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

20. The method of claim 18, wherein the method further comprises applying noise reduction or enhancement, or both, to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

21. The method of claim 18, wherein the enhancing pixels of the sclera comprises utilizing information regarding said periphery.

22. The method of claim 18, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

23. A digital image acquisition device, comprising

a lens,

an image sensor and

a processor, and

a processor-readable memory having embodied therein processor-readable code for programming the processor to perform a method of enhancing an appearance of a face within a digital image, wherein the method comprises:

acquiring a digital image of a scene including a face, including capturing the image using the lens and the image sensor, or receiving said image following capture by another device with its own lens and image sensor, or a combination thereof;

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the iris at an iris side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof; and

wherein the method further comprises finding a border point as a maximum gradient point along a horizontal line through an upright eye, and growing the border including adding immediate neighboring points characterized by largest horizontal intensity gradients.

24. The device of claim 23, wherein the enhancing of the iris comprises linear contrast stretching.

25. The device of claim 23, wherein the enhancing of the iris comprises spatial blurring.

26. The device of claim 23, wherein the method further comprises applying one or more localized color smoothing kernels to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

27. The device of claim 23, wherein the method further comprises applying noise reduction or enhancement, or both, to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

28. The device of claim 23, wherein the method further comprises identifying one or more further segments of a periphery of the sclera.

29. The device of claim 28, wherein the method further comprises enhancing pixels of the sclera within the periphery.

30. The device of claim 23, wherein the method further comprises enhancing pixels of the sclera at a sclera side of the border.

31. The device of claim 23, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

32. A digital image acquisition device, comprising

a lens,

an image sensor and

a processor, and

a processor-readable memory having embodied therein processor-readable code for programming the processor to perform a method of enhancing an appearance of a face within a digital image, wherein the method comprises:

acquiring a digital image of a scene including a face, including capturing the image using the lens and the image sensor, or receiving said image following capture by another device with its own lens and image sensor, or a combination thereof;

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the iris at an iris side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof,

wherein the method further comprises identifying one or more further segments of a periphery of the sclera; and

wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

33. The device of claim 32, wherein the method further comprises applying one or more localized color smoothing kernels to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

34. The device of claim 32, wherein the method further comprises applying noise reduction or enhancement, or both, to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

35. The device of claim 32, wherein the method further comprises identifying one or more further segments of a periphery of the sclera.

36. The device of claim 35, wherein the method further comprises enhancing pixels of the sclera within the periphery.

37. The device of claim 32, wherein the method further comprises enhancing pixels of the sclera at a sclera side of the border.

38. The device of claim 32, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

39. A digital image acquisition device, comprising

a lens,

an image sensor and

a processor, and

a processor-readable memory having embodied therein processor-readable code for programming the processor to perform a method of enhancing an appearance of a face within a digital image, wherein the method comprises:

acquiring a digital image of a scene including a face, including capturing the image using the lens and the image sensor, or receiving said image following capture by another device, or a combination thereof;

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the sclera at a sclera side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof and

wherein the method further comprises identifying one or more further segments of a periphery of the sclera; and

wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

40. The device of claim 39, wherein the method further comprises applying one or more localized color smoothing kernels to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

41. The device of claim 39, wherein the method further comprises applying noise reduction or enhancement, or both, to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

42. The device of claim 39, wherein the enhancing pixels of the sclera comprises utilizing information regarding said periphery.

43. The device of claim 39, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

44. One or more non-transitory computer-readable storage media having embodied therein code for programming one or more processors to perform a method of enhancing an appearance of a face within a digital image that has been acquired with a device that includes a lens and an image sensor, wherein the method comprises:

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the iris at an iris side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof, and

wherein the method further comprises finding a border point as a maximum gradient point along a horizontal line through an upright eye, and growing the border including adding immediate neighboring points characterized by largest horizontal intensity gradients.

45. The one or more computer-readable storage media of claim 44, wherein the enhancing of the iris comprises linear contrast stretching.

46. The one or more computer-readable storage media of claim 44, wherein the enhancing of the iris comprises spatial blurring.

47. The one or more computer-readable storage media of claim 44, wherein the method further comprises applying one or more localized color smoothing kernels to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

48. The one or more computer-readable storage media of claim 44, wherein the method further comprises applying noise reduction or enhancement, or both, to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

49. The one or more computer-readable storage media of claim 44, wherein the method further comprises identifying one or more further segments of a periphery of the sclera.

50. The one or more computer-readable storage media of claim 49, wherein the method further comprises enhancing pixels of the sclera within the periphery.

51. The one or more computer-readable storage media of claim 44, wherein the method further comprises enhancing pixels of the sclera at a sclera side of the border.

52. The one or more computer-readable storage media of claim 44, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

53. One or more non-transitory computer-readable storage media having embodied therein code for programming one or more processors to perform a method of enhancing an appearance of a face within a digital image that has been acquired with a device that includes a lens and an image sensor, wherein the method comprises:

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the iris at an iris side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereofwherein the method further comprises identifying one or more further segments of a periphery of the sclera

wherein the method further comprises identifying one or more further segments of a periphery of the sclera; and

wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

54. The one or more computer-readable storage media of claim 53, wherein the enhancing of the iris comprises linear contrast stretching.

55. The one or more computer-readable storage media of claim 53, wherein the enhancing of the iris comprises spatial blurring.

56. The one or more computer-readable storage media of claim 53, wherein the method further comprises applying one or more localized color smoothing kernels to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

57. The one or more computer-readable storage media of claim 53, wherein the method further comprises applying noise reduction or enhancement, or both, to the iris, such that the iris of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

58. The one or more computer-readable storage media of claim 53, wherein the method further comprises finding a border point as a maximum gradient point along a horizontal line through an upright eye, and growing the border including adding immediate neighboring points characterized by largest horizontal intensity gradients.

59. The one or more computer-readable storage media of claim 53, wherein the method further comprises enhancing pixels of the sclera within the periphery.

60. The one or more computer-readable storage media of claim 53, wherein the method further comprises enhancing pixels of the sclera at a sclera side of the border.

61. The one or more computer-readable storage media of claim 53, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

62. One or more non-transitory computer-readable storage media having embodied therein code for programming one or more processors to perform a method of enhancing an appearance of a face within a digital image that has been acquired with a device that includes a lens and an image sensor, wherein the method comprises:

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the sclera at a sclera side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof; and

wherein the method further comprises identifying one or more further segments of a periphery of the sclera; and

wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

63. The one or more computer-readable storage media of claim 62, wherein the method further comprises applying one or more localized color smoothing kernels to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

64. The one or more computer-readable storage media of claim 62, wherein the method further comprises applying noise reduction or enhancement, or both, to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

65. The one or more computer-readable storage media of claim 62, wherein the enhancing pixels of the sclera comprises utilizing information regarding said periphery.

66. The one or more computer-readable storage media of claim 62, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

67. A method of enhancing an appearance of a face within a digital image, comprising using a processor in:

acquiring a digital image of a scene including a face, including capturing the image using a lens and an image sensor of a processing device that includes the processor, or receiving said image following capture by another device that includes a lens and an image sensor, or a combination thereof;

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the sclera at a sclera side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof; and

wherein the method further comprises finding a border point as a maximum gradient point along a horizontal line through an upright eye, and growing the border including adding immediate neighboring points characterized by largest horizontal intensity gradients.

68. The method of claim 67, wherein the method further comprises applying one or more localized color smoothing kernels to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

69. The method of claim 67, wherein the method further comprises applying noise reduction or enhancement, or both, to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

70. The method of claim 67, wherein the method further comprises identifying one or more further segments of a periphery of the sclera.

71. The method of claim 67, wherein the enhancing pixels of the sclera comprises utilizing information regarding said periphery.

72. The method of claim 70, wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

73. The method of claim 67, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

74. The device of claim 67, wherein the enhancing of the iris comprises linear contrast stretching.

75. The device of claim 67, wherein the enhancing of the iris comprises spatial blurring.

76. A digital image acquisition device, comprising

a lens,

an image sensor and

a processor, and

a processor-readable memory having embodied therein processor-readable code for programming the processor to perform a method of enhancing an appearance of a face within a digital image, wherein the method comprises:

acquiring a digital image of a scene including a face, including capturing the image using the lens and the image sensor, or receiving said image following capture by another device, or a combination thereof;

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the sclera at a sclera side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof;

wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

77. The device of claim 76, wherein the method further comprises applying one or more localized color smoothing kernels to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

78. The device of claim 76, wherein the method further comprises applying noise reduction or enhancement, or both, to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

79. The device of claim 76, wherein the method further comprises identifying one or more further segments of a periphery of the sclera.

80. The device of claim 79, wherein the enhancing pixels of the sclera comprises utilizing information regarding said periphery.

81. The device of claim 79, wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

82. The device of claim 76, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.

83. One or more non-transitory computer-readable storage media having embodied therein code for programming one or more processors to perform a method of enhancing an appearance of a face within a digital image that has been acquired with a device that includes a lens and an image sensor, wherein the method comprises:

detecting the face within the digital image;

segmenting skin tone portions of the face from face features including one or two eyes;

within at least one of the eyes, identifying a border between an iris and a sclera of the eye using luminance information;

enhancing pixels of the sclera at a sclera side of the border;

displaying, transmitting, communicating or digitally storing or otherwise outputting the enhanced image or a further processed version, or combinations thereof; and

wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

84. The one or more computer-readable storage media of claim 83, wherein the method further comprises applying one or more localized color smoothing kernels to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized color smoothing.

85. The one or more computer-readable storage media of claim 83, wherein the method further comprises applying noise reduction or enhancement, or both, to the sclera, such that the sclera of the enhanced image comprises pixels modified from original pixels of the face at least by localized noise reduction or enhancement, or both.

86. The one or more computer-readable storage media of claim 83, wherein the method further comprises identifying one or more further segments of a periphery of the sclera.

87. The one or more computer-readable storage media of claim 86, wherein the enhancing pixels of the sclera comprises utilizing information regarding said periphery.

88. The one or more computer-readable storage media of claim 86, wherein the identifying of one or more further segments of a periphery of the sclera comprises determining points along multiple horizontal lines, wherein the points are located each at a next local intensity minimum that follows a local intensity maximum.

89. The one or more computer-readable storage media of claim 83, wherein the identifying said border between said iris and said sclera of the eye comprises not using color information.