System and method for creating and sharing personalized virtual makeovers

• US 9,058,765 B1
• Filed: 03/17/2009
• Issued: 06/16/2015
• Est. Priority Date: 03/17/2008
• Status: Expired due to Fees

• Current Assignee: TAAZ Incorporated
• Original Assignee: TAAZ Incorporated
• Inventors: Mallick, Satya, Barnes, Kevin, Kriegman, David
• Primary Examiner(s): Richer, Aaron M
• Assistant Examiner(s): NGUYEN, ANH TUAN V
• Application Number:
  US12/406,099
• Time in Patent Office: 2,282 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 30/0256 User search
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0275 Auctions
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G09G 2320/0666 for control of colour parameters, e.g. colour temperature
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 5/06 using colour palettes, e.g. look-up tables
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing

Abstract

A system and methods for creating and sharing personalized virtual makeovers comprising algorithms for applying virtual cosmetics, hairstyles and accessories to a digital portrait of an individual using described techniques that produce a revised image with a realistic appearance. The system may calculate positions of the boundaries of facial features such as eyes, lips and hair, or provide the user a sophisticated adjustment method. The invention places and shapes virtual cosmetics, hairstyles and accessories to precisely calculated locations, blending colors and shading for a realistic and lifelike result. Accessories such as eyeglasses and jewelry are also realistically applied. The system comprises individual makeover editors for skin, eyes, lips, hair and accessories. Virtual makeovers may be shared, including allowing multiple users modify the same image simultaneously.

Claims

1. A non-transitory computer-readable medium comprising computer program product for creating and sharing a personalized virtual makeover image, said computer program product comprising computer readable instructions, wherein execution of said computer readable instructions by one or more processors causes said one or more processors to:

preprocess an image using a computing device;

accept a first input selecting a makeover item from a set of makeover items associated with a predefined order;

obtain locations of placement points for said makeover item on said image;

automatically transform said makeover item using affine transformation so that its position conforms to said locations of said placement points;

create a first graphics layer and a first mask for said transformed makeover item;

accept a second input selecting at least one additional makeover item from said set of makeover items, obtain locations of placement points for said at least one additional makeover item on said image, automatically transform said at least one additional makeover item using affine transformation so that its position conforms to said locations of said placement points for said at least one additional makeover item, and create at least one additional graphics layer and a corresponding mask for said transformed at least one additional makeover item;

merge said first graphics layer and said at least one additional graphics layer onto said image in a layering order using said predefined order to create a virtual makeover image on a display comprising adjustment handles on said makeover item and said at least one additional makeover item for adjustment and transformation control of said makeover item and said at least one additional makeover item, wherein said first graphics layer and said at least one additional graphics layer are operated upon individually to adjust and transform said makeover item and said at least one additional makeover item using said adjustment handles;

accept further inputs from said adjustment handles to generate an adjusted and transformed first graphics layer and an at least one adjusted and transformed additional graphics layer; and

generate the virtual makeover image by rendering said adjusted and transformed first graphics layer through said first mask and said at least one adjusted and transformed additional graphics layer through said corresponding mask onto said image in said layering order using said predefined order.

2. The non-transitory computer-readable medium of claim 1, wherein said set of makeover items comprises at least one face layer makeover item, at least one mouth layer makeover item and at least one eye layer makeover item, wherein said predefined order indicates layering of said at least one face layer makeover item below said at least one mouth layer makeover item and said least one eye layer makeover item.

3. The non-transitory computer-readable medium of claim 2, wherein said set of makeover items further comprises at least one hair layer makeover item, wherein said predefined order indicates layering of said at least one hair layer makeover item above said at least one face layer makeover item, said at least one mouth layer makeover item and said least one eye layer makeover item.

4. The non-transitory computer-readable medium of claim 1, wherein said placement points are automatically detected.

5. The non-transitory computer-readable medium of claim 1, wherein said set of makeover items comprises at least one foundation and at least one blush, wherein said predefined order indicates layering of said at least one foundation below said at least one blush.

6. The non-transitory computer-readable medium of claim 1, wherein said create a first layer for said transformed makeover item comprises applying said transformed makeover item through a skin mask.

7. The non-transitory computer-readable medium of claim 1, wherein said virtual makeover image is displayed on a display coupled to said computing device.

8. A non-transitory computer-readable medium comprising computer program product for creating a personalized virtual makeover image, said computer program product comprising computer readable instructions, wherein execution of said computer readable instructions by one or more processors causes said one or more processors to:

preprocess a base image using a computing device;

accept a first input selecting a hairstyle for application to said base image;

obtain locations of placement points for said hairstyle on said base image;

automatically transform said hairstyle using affine transformation so that its position conforms to said locations of said placement points, wherein said transform results in warping said hairstyle;

generate a hair layer and a mask for said transformed hairstyle;

accept a second input selecting at least one additional makeover item from a set of makeover items associated with a predefined order, obtain locations of placement points for said at least one additional makeover item on said base image, automatically transform said at least one additional makeover item using affine transformation so that its position conforms to said locations of said placement points for said at least one additional makeover item, and generate at least one additional graphics layer and a corresponding mask for said transformed at least one additional makeover item;

merge said hair layer and said at least one additional graphics layer on said base image in a layering order using said predefined order to create a virtual makeover image on a display comprising adjustment handles on said hairstyle and said at least one additional makeover item for adjustment and transformation control of said hairstyle and said at least one additional makeover item, wherein said hair layer and said at least one additional graphics layer are operated upon individually to adjust and transform said hairstyle and said at least one additional makeover item using said adjustment handles;

accept further inputs from said adjustment handles to generate an adjusted and transformed hair layer and an at least one adjusted and transformed additional graphics layer; and

render said adjusted and transformed hair layer through said mask and said at least one adjusted and transformed additional graphics layer through said corresponding mask onto said base image in said layering order using said predefined order to create the virtual makeover image.

9. The non-transitory computer-readable medium of claim 8, wherein said transformation control comprises a global transformation control.

10. The non-transitory computer-readable medium of claim 9, wherein said global transformation comprises six degrees of freedom.

11. The non-transitory computer-readable medium of claim 10, wherein said global transformation comprises an affine transformation.

12. The non-transitory computer-readable medium of claim 9, wherein said global transformation control comprises a bounding box.

13. The non-transitory computer-readable medium of claim 8, wherein said transformation control comprises a local transformation control.

14. The non-transitory computer-readable medium of claim 13, wherein said local transformation control comprises a control for morphing said hairstyle.

15. The non-transitory computer-readable medium of claim 13, wherein said local transformation control comprises a control for orienting said hairstyle.

16. An article of manufacture comprising:

a computing device;

said computing device informationally coupled to at least one display device;

said computing device informationally coupled to at least one input device;

said computing device informationally coupled to at least one tangible form of computer data storage;

said computing device comprising computer-readable program instructions, said computer readable program instructions comprising a virtual makeover method, said virtual makeover method comprising;

preprocessing an image on a computing device;

accepting a first input selecting a makeover item from a set of makeover items associated with a predefined order;

obtaining locations of placement points for said makeover item on said image;

automatically transforming said makeover item using affine transformation so that its position conforms to said locations of said placement points;

generating a graphics layer and a first mask for said transformed makeover item;

accepting a second input selecting at least one additional makeover item from said set of makeover items, obtaining locations of placement points for said at least one additional makeover item on said image, automatically transforming said at least one additional makeover item using affine transformation so that its position conforms to said locations of said placement points for said at least one additional makeover item, and generating at least one additional graphics layer and a corresponding mask for said at least one additional makeover item;

merging said graphics layer and said at least one additional graphics layer onto said image in a layering order using said predefined order to create a virtual makeover image on a display comprising adjustment handles on said makeover item and said at least one additional makeover item for adjustment and transformation control of said makeover item and said at least one additional makeover item, wherein said first graphics layer and said at least one additional graphics layer are operated upon individually to adjust and transform said makeover item and said at least one additional makeover item using said adjustment handles;

accepting further inputs from said adjustment handles to generate an adjusted and transformed graphics layer and at least one adjusted and transformed additional graphics layer; and

rendering said adjusted and transformed graphics layer through said first mask and said at least one adjusted and transformed additional graphics layer through said corresponding mask onto said image in said layering order using said predefined order to create the virtual makeover image.

17. The article of manufacture of claim 16 wherein said computing device further comprises a kiosk.

18. The article of manufacture of claim 16 wherein said computing device further comprises a handheld computing device.

19. The article of manufacture of claim 16 wherein said computing device further comprises a toy.

20. A virtual makeover computer system comprising:

a computing device informationally coupled to a user input device and a display device;

a portrait stored in a tangible computer readable memory medium coupled to said computing device;

said computing device comprising a computer usable memory medium specially programmed with computer readable program code, wherein said computer readable program code comprises a method to;

detect a face in said portrait, said face delineated by a facial boundary;

detect facial features boundaries within said facial boundary;

detect color properties associated with said face in said portrait;

obtain a plurality of makeover items from a recommended list of makeover items to apply with respect to said face, wherein said plurality of makeover items are associated with a predefined order, wherein said recommended list is based on said color properties, said facial features, user preferences, user attributes, or combinations thereof;

at least one image processing function configured to automatically transform each one of said plurality of makeover items using affine transformation so that its position conforms to locations of at least one of said facial boundary and said facial features and further configured to generate a plurality of graphics layers and a plurality of masks, wherein each one of said plurality of graphics layers and a corresponding one of said plurality of masks represent one of said plurality of makeover items, wherein said image processing function is further configured to merge said plurality of graphics layers onto said portrait in a layering order using said predefined order to create a virtual makeover image on a display comprising adjustment handles on said plurality of makeover items for adjustment and transformation control of said plurality of makeover items, wherein said plurality of graphics layers is operated upon individually to adjust and transform each one of said plurality of makeover items using said adjustment handles, wherein said image processing function is further configured to accept further inputs from said adjustment handles and generate an adjusted and transformed plurality of graphics layers; and

a rendering function configured to create the virtual makeover image from said adjusted and transformed plurality of graphics layers through said plurality of masks onto said portrait, wherein said adjusted and transformed plurality of graphics layers is rendered in said layering order using said predefined order.

21. The virtual makeover computer system of claim 20 wherein said face detection is performed automatically.

22. The virtual makeover computer system of claim 20 wherein said facial features detection is performed automatically.

23. The virtual makeover computer system of claim 20 wherein said facial features detection is performed semi-automatically.

24. The virtual makeover computer system of claim 20 wherein said facial features boundaries comprise boundaries of a mouth within said face in said portrait.

25. The virtual makeover computer system of claim 20 wherein said facial features boundaries comprise boundaries of eyes within said face in said portrait.

26. The virtual makeover computer system of claim 20 wherein said color properties comprise skin tone.

27. The virtual makeover computer system of claim 20 wherein said color properties comprise light color.

28. The virtual makeover computer system of claim 20 wherein said color properties comprise light direction.

29. The virtual makeover computer system of claim 20 wherein a color space that decomposes color into chromacity and luminance is used to detect said color properties.

30. The virtual makeover computer system of claim 20 wherein said computing device is coupled to a computer server over a computer network.

31. The virtual makeover computer system of claim 20 wherein said computing device is coupled to a computer server over a World Wide Web via an Internet;

and wherein said makeover image is displayed in a web browser.

32. The virtual makeover computer system of claim 20 wherein said plurality of graphics layers is operated upon by at least two users.

33. The virtual makeover computer system of claim 20 wherein said plurality of graphics layers is operated upon simultaneously by at least two users.

34. The virtual makeover computer system of claim 20 wherein said rendering function is further configured to:

obtain a stored look describing at least one makeover item;

automatically transform said stored look to said detected facial features of said detected facial boundary; and

apply said stored look to said portrait before rendering said makeover image.

35. A virtual makeover computer system comprising:

a computer server;

a computing device informationally coupled to said computer server, said computing device coupled to a user input device and a display device;

a portrait stored in a tangible computer readable memory medium coupled to said computing device;

said computing device comprising a computer usable memory medium specially programmed with computer readable program code, wherein said computer readable program code comprises;

a face detection function configured to automatically detect facial boundaries of a face in said portrait;

a facial feature detection function configured to detect facial features boundaries within said facial boundaries;

a recommendation engine configured to provide a plurality of makeover items to apply with respect to said face, wherein said plurality of makeover items are associated with a predefined order;

at least one image processing function configured to automatically transform each one of said plurality of makeover items using affine transformation so that its position conforms to locations of at least one of said facial boundaries and said facial features and further configured to generate a plurality of graphics layers and a plurality of masks, wherein each one of said plurality of graphics layers and a corresponding one of said plurality of masks represent one of said plurality of makeover items, wherein said image processing function is further configured to merge said plurality of graphics layers onto said portrait in a layering order using said predefined order to create a virtual makeover image on a display comprising adjustment handles on said plurality of makeover items for adjustment and transformation control of said plurality of makeover items, wherein said plurality of graphics layers is operated upon individually to adjust and transform each one of said plurality of makeover items using said adjustment handles, wherein said image processing function is further configured to accept further inputs from said adjustment handles and generate an adjusted and transformed plurality of graphics layers; and

a rendering function configured to create the virtual makeover image comprising said adjusted and transformed plurality of graphics layers through said plurality of masks onto said portrait, wherein said adjusted and transformed plurality of graphics layers is rendered in said layering order using said predefined order.

36. A virtual makeover computer system comprising:

a computing device informationally coupled to a user input device and a display device;

a portrait stored in a tangible computer readable memory medium coupled to said computing device;

said computing device comprising a computer usable memory medium specially programmed with computer readable program code, wherein said computer readable program code comprises;

a facial boundary detection function for detecting a face in said portrait;

a facial features detection function for detecting facial features within said facial boundary;

a plurality of makeover items to apply with respect to said portrait, wherein said plurality of makeover items are associated with a predefined order;

a recommendation engine configured to generate selected ones from said plurality of makeover items;

at least one image processing function configured to automatically transform each one of said plurality of makeover items using affine transformation so that its position conforms to locations of at least one of said facial boundaries and said facial features and further configured to generate a plurality of graphics layers and a plurality of masks, wherein each one of said plurality of graphics layers and a corresponding one of said plurality of masks represent one of said plurality of makeover items, wherein said image processing function is further configured to merge said plurality of graphics layers onto said portrait in a layering order using said predefined order to create a virtual makeover image on a display comprising adjustment handles on said plurality of makeover items for adjustment and transformation control of said plurality of makeover items, wherein said plurality of graphics layers is operated upon individually to adjust and transform each one of said plurality of makeover items using said adjustment handles, wherein said image processing function is further configured to accept further inputs from said adjustment handles and generate an adjusted and transformed plurality of graphics layers; and

a rendering function configured to generate the makeover image by merging said selected ones of said adjusted and transformed plurality of graphics layers through said plurality of masks onto said portrait, wherein said adjusted and transformed plurality of graphics layers is rendered in said layering order using said predefined order.

37. The virtual makeover computer system of claim 36 wherein said selected ones comprise a product.

38. The virtual makeover computer system of claim 36 wherein said selected ones comprise a style.

39. The virtual makeover computer system of claim 36 wherein said selected ones comprise a technique.

40. The virtual makeover computer system of claim 36 wherein said selected ones comprise a color.

41. The virtual makeover computer system of claim 36 wherein said recommendation engine is an expert system.

42. The virtual makeover computer system of claim 36 wherein said recommendation engine is a learning system.

43. The virtual makeover computer system of claim 36 wherein said selected ones are generated by considering said facial features.

44. The virtual makeover computer system of claim 36 wherein said selected ones are generated by considering the personal characteristics of said portrait.

45. The virtual makeover computer system of claim 36 wherein said selected ones are generated by considering popularity of said selected ones.

46. The virtual makeover computer system of claim 36 wherein said selected ones are generated based on a scored gallery of choices.

47. The virtual makeover computer system of claim 36 wherein said selected ones comprise a recommended look.