METHOD, SYSTEM AND COMPUTER PROGRAM PRODUCT FOR AUTOMATIC AND SEMI-AUTOMATIC MODIFICATION OF DIGITAL IMAGES OF FACES

• US 20070258656A1
• Filed: 05/04/2007
• Published: 11/08/2007
• Est. Priority Date: 05/05/2006
• Status: Active Grant

• Current Assignee: L'Oreal SA
• Original Assignee: Parham Aarabi
• Inventors: Aarabi, Parham
• Granted Patent: US 8,660,319 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 1/00132 in a digital photofinishing system, i.e. a system where digital photographic images undergo typical photofinishing processing, e.g. printing ordering
• CPC Class Codes: H04N 1/00167 Processing or editing H04N1/00196 - H04N1/00201 take precedence

Abstract

The present invention is directed at modifying digital images of faces automatically or semi-automatically. In one aspect, a method of detecting faces in digital images and matching and replacing features within the digital images is provided. Techniques for blending, recoloring, shifting and resizing of portions of digital images are disclosed. In other aspects, methods of virtual “face lifts” and methods of detecting faces within digital image are provided. Advantageously, the detection and localization of faces and facial features, such as the eyes, nose, lips and hair, can be achieved on an automated or semi-automated basis. User feedback and adjustment enables fine tuning of modified images. A variety of systems for matching and replacing features within digital images and detection of faces in digital images is also provided, including implementation as a website, through mobile phones, handheld computers, or a kiosk. Related computer program products are also disclosed.

Claims

1. A method for modifying digital images comprising:

(a) detecting a face in a first digital image and optionally detecting a face in a second digital image, if the location of face(s) in the first digital image or the second digital image has not already been established;

(b) establishing regions of interest in the face in the first digital image and optionally establishing regions of interest in the face in the second digital image;

(c) detecting features in the regions of interest in the face in the first digital image and optionally detecting features in the regions of interest in the face in the second digital image; and

(d) modifying the first digital image by either;

(i) matching and replacing one or more of the features in the face in the first digital image with the one or more features in the regions of interest in the face in the second digital image, thereby defining a modified digital image;

or

(ii) isolating from modification the regions of interest in the first digital image, modifying the first digital image other than the regions of interest, and replacing the regions of interest into the modified first digital image.

2. The method of claim 1 wherein the features include eyes, eyebrows, nose, mouth, lips or hair.

3. The method of claim 1 further comprising blending, recoloring, shifting or resizing the one or more features in the face in the second digital image.

4. The method of claim 1 further comprising adjusting size and location of the one or more features in the face in the second digital image in the modified digital image to increase the perceived beauty of a face in the modified digital image.

5. The method of claim 1 further comprising color adjusting the one or more features in the face in the second digital image to correspond with the face in the first digital image, or color adjusting the features of the face in the first digital image to correspond to the modified first digital image.

6. The method of claim 5 wherein the color adjusting is achieved by performing color histogram transformations.

7. The method of claim 6 wherein the color histograms favour outside areas of the one or more features.

8. The method of claim 1 further comprising blending the one or more features in the face in the second digital image with the face in the first digital image.

9. The method of claim 8 wherein the blending is achieved by gradient filling using a blending mask.

10. The method of claim 9 wherein the blending mask corresponds to the regions of interest in the face in the first digital image.

11. The method of claim 1 further comprising resizing the one or more features in the face in the second digital image to correspond with the one or more features in the face in the first digital image prior to matching and replacing.

12. The method of claim 1 further comprising shifting the one or more features in the face in the second digital image to correspond with the one or more features in the face in the first digital image prior to matching and replacing.

13. The method of claim 1 further comprising blending the face in the first digital image prior to matching and replacing one or more of the features in the face in the first digital image with the one or more features in the face in the second digital image.

14. The method of claim 1 wherein location of the face in the first digital image and/or location of the face in the second digital image is determined including by using user input.

15. The method of claim 1 wherein the modified first image is used to provide a virtual facelift of the face of the first digital image.

16. The method of claim 1 wherein the location of the face in the first digital image and optionally the location of the face in the second digital image are determined by calculating edge intensities and by using a set of deterministic rules for edges within a face.

17. The method of claim 1 wherein the regions of interest in the face in the first digital image and optionally the regions of interest in the face in the second digital image are established using a generic face template.

18. The method of claim 17 wherein the generic face template is proportional in size to the face in the first digital image.

19. The method of claim 18 wherein the features in the regions of interest in the face in the first digital image and optionally the features in the regions of interest in the face in the second digital image are detected by calculating gradient intensities and a gradient template.

20. The method of claim 19 wherein the features in the regions of interest in the face in the first digital image, and optionally the features in the regions of interest in the face in the second digital image, are also detected by deviation from skin color.

21. The method of claim 1 wherein selected regions of the face in the first digital image are modified to produce a facelift effect in those selected regions.

22. The method of claim 1 further comprising the step of making the modified digital image or modified first digital image available on a computer or wireless device via a computer network or wireless network.

23. The method of claim 1 wherein the face in the first digital image and optionally the face in the second digital image are detected using a fusion face detection means.

24. The method of claim 23 wherein the fusion detection means consists of one or more of symmetry-based face detection, template subtraction face detection, template ratio face detection, skin-detector-based face detection, eye-lip total symmetry face detection, eye total symmetry face detection, or lip total symmetry face detection.

29. The method of claim 1 wherein the face in the first digital image and optionally the face in the second digital image is detected using a fusion detection means.

30. The method of claim 29 wherein the fusion detection means consists of one or more of symmetry-based face detection, template subtraction face detection, template ratio face detection, skin-detector-based face detection, eye-lip total symmetry face detection, eye total symmetry face detection, or lip total symmetry face detection.

46. The method of claim 1 wherein the location of the features are used to re-evaluate the location of the face, thereby enabling interactive face detection and facial feature detection.

47. The method of claim 1 wherein after the facial features are detected, a three dimensional projective transformation is performed on the face to make the face frontal, followed by face modification.

25. A method for modifying a digital image comprising:

(a) detecting a face in the digital image;

(b) establishing regions of interest in the face in the digital image;

(c) detecting features in the regions of interest in the face in the digital image;

(d) augmenting the face in the digital image by smoothing selective regions; and

(e) replacing the features in the face in the digital image, thereby defining a modified digital image.

26. The method of claim 25 wherein the regions of interest in the face in the digital image are established using a generic template.

27. The method of claim 26 wherein the generic template is proportional in size to the face in the digital image.

28. The method of claim 27 wherein the features in the regions of interest in the face in the digital image are detected by calculating gradient intensities using one or more boxes in the generic template.

31. A system for modifying digital images comprising:

(a) a computer linked to a database, the computer including or being linked to a utility for enabling one or more users upload, store, retrieve, email, display and/or manage digital images;

(b) a modification utility linked to the computer, the modification utility being operable to provide instructions to the computer that enable the computer to;

(i) detect a face in a first digital image and optionally detect a face in a second digital image, if the location of the faces in the first digital image or the second digital image has not already been established;

(ii) establish regions of interest in the face in the first digital image and optionally establish regions of interest in the face in the second digital image;

(iii) detect features in the regions of interest in the face in the first digital image and optionally detect features in the regions of interest in the face in the second digital image; and

(iv) modify the first digital image by either;

(A) matching and replacing one or more of the features in the face in the first digital image with the one or more features in the face in the second digital image, thereby defining a modified digital image;

or

(B) isolating from modification the regions of interest in the first digital image, modifying the first digital image other than the regions of interest, and replacing the regions of interest into the modified first digital image.

32. The system of claim 31 wherein the modification utility further includes means for processing digital images consisting of one or more of the following operations performed on the digital images:

resizing, histogram equalization, compressing, histogram transformation, color adjustment, recoloring, correlating in two dimensions, convolving in two dimensions, blending, edge extraction, cropping, smoothing, or blending based on a template.

33. The system of claim 31 wherein the computer is a server computer, and the one or more users are associated with remote computers linked to the server computer via a communication network, and the server computer is enable to provide access from the remote computers to computer resources that enable the one or more users to upload, store, retrieve email, display or manage digital images, and wherein the server computer is operable to receive image modification instructions from the remote computers.

34. The system of claim 33 wherein the server computer is interoperable with remote computers consisting of a personal computer, mobile phone, Internet device, handheld computer or a kiosk.

35. The system of claim 34 wherein the server computer enables the one or more users to interactively access digital images via the remote computer, and the server computer is operable to receive image modification instructions from the remote computer.

36. The system of claim 31 wherein the computer is part of a kiosk.

37. The system of claim 31 wherein the features include eyes, eyebrows, nose, mouth, lips or hair.

38. The system of claim 37 wherein modification utility is operable to enable the one or more users to select the features to match and replace.

39. The system of claim 33 wherein at least one of the remote computers is a wireless device, and the server computer is operable to receive image modification instructions from a user via the wireless device, and the server computer is operable to deliver the modified digital image or modified first digital image to the wireless device via a wireless network.

40. The system of claim 31 wherein the computer is a personal computer and the database is an online database linked to a web server, the web server being operable to enable the personal computer to upload, store, retrieve, email, display and/or manage digital images stored to the online database.

41. The system of claim 31 wherein the modification utility is embedded in field programmable gate arrays.

42. The system of claim 31 further comprising a camera to generate digital images for modification.

43. The system of claim 31, wherein the modification utility is operable to enable the computer to:

(i) permit the one or more users to interactively access the digital images and select one or more digital images for modification;

(ii) detect a face in the digital image;

(iii) establish regions of interest in the face in the digital image;

(iv) detect features in the regions of interest in the face in the digital image;

(v) blend the face in the digital image;

(vi) replace the features in the face in the digital image, thereby defining a modified digital image; and

(vii) display the modified digital image to the user.

44. The system of claim 31 wherein the system further comprises a detection utility linked to the computer being operable to provide instructions to the computer that enable the computer:

(a) To progressively scan a digital image with a detection box of varying size; and

(b) using two or more detection techniques embodied in the detection utility, such detection techniques consisting of (i) symmetry-based detection, (ii) template subtraction detection, (iii) template ratio detection, (iv) skin-detector-based detection, (v) eye-lip total symmetry detection, (vi) eye total symmetry detection, or (vii) lip total symmetry detection, establishing face scores and face coordinates by operation of the detection techniques, thereby detect the existence of a face in the digital image.

45. The system of claim 44 wherein the server computer is operable to remotely return the face score and the face coordinates to a user.

48. The system of claim 31 wherein the computer is operable to perform automatically virtual facelift operations on digital images of faces upon a user uploading or selecting a digital image of a face.

49. The system of claim 31 wherein the modification utility is operable to simulate the effects of different cosmetics and plastic surgery products automatically using computer vision and artificial intelligence techniques.

50. The system of claim 31 wherein the modification utility is operable to simulate the post operation appearance of a face prior to an operation taking place by selectively modifying a digital image of the face using computer vision and artificial intelligence techniques.

51. The system of claim 31 wherein revenue generation is linked to image modification by one or more entities receiving a fee by permitting one or more users to pay a fee to modify the digital image of a face provided to the system.

52. The system of claim 31 wherein the one or more users submit a digital image of a face with a set of image modification criteria, or linked to a set of image modification criteria, and the computer is operable to provide an Internet link to a dynamic image that changes over time based on the set of image modification criteria, or a subset of such image modification criteria selected by the one or more users.

53. The system of claim 31 wherein the system is operable to enable automatic extraction of facial features (eyes, eyebrows, nose, and mouths/lips), followed by optional face smoothing and selective replacement of the facial features in order to perform a virtual facelift and skin reconditioning on a digital image of a face.

54. The system of claim 52 wherein the system is operable to receive user adjustment instructions to assist in modification of the digital image of the face.

55. The system of claim 31 wherein the computer includes a digital pad, or electronic device including a display screen and an input means for entering user requested face modifications;

and the system includes a camera.

56. The system of claim 31 wherein the modification utility is operable to enable the one or more users to select the areas of a digital image of a face for modification, and the degree of modification to be applied to selected areas.

57. The system of claim 31 wherein the modification utility is operable to apply one or more virtual face lift operations in one or more areas of the digital image of the face including the eyebrows, the mid-brow, forehead, below the eye, around the eye, inner cheek, outer cheek, nose, below nose, jaw, chin, lip, eye, hair, mouth, and below mouth areas.

58. The system of claim 31 wherein the modification utility embodies a fusion face detection means for detecting a face in a digital image.

59. The system of claim 57 wherein the fusion detection means embodies symmetry-based face detection, template subtraction face detection, template ratio face detection, skin-detector-based face detection, eye-lip total symmetry face detection, eye total symmetry face detection, or lip total symmetry face detection, and dynamically applies to or more such detection techniques.

60. The system of claim 31 wherein the digital images include one or more video frames.

61. A computer program product for enabling the modification of digital images comprising:

(a) a computer readable medium bearing software instructions; and

(b) the software instructions for enabling the computer to perform predetermined operations, the predetermined operations including the steps of;

(i) detecting a face in a first digital image and optionally detecting a face in a second digital image, if the location of the faces in the first digital image or the second digital image has not already been established;

(ii) establishing regions of interest in the face in the first digital image and optionally establishing regions of interest in the face in the second digital image;

(iii) detecting features in the regions of interest in the face in the first digital image and optionally detecting features in the regions of interest in the face in the second digital image; and

(iv) modifying the first digital image by either;

(A) matching and replacing one or more of the features in the face in the first digital image with the one or more features in the regions of interest in the face in the second digital image, thereby defining a modified digital image;

or

(B) isolating from modification the regions of interest in the first digital image, modifying the first digital image other than the regions of interest, and replacing the regions of interest into the modified first digital image.