Head-mounted object image combining method, makeup image combining method, headmounted object image combining device, makeup image composition device, and program

• US 20050175234A1
• Filed: 09/02/2003
• Published: 08/11/2005
• Est. Priority Date: 09/03/2002
• Status: Abandoned Application

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Sakamoto, Shizuo
• Application Number:
  US10/513,354
• Publication Number:
  US 20050175234A1
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: A45D 44/005 for selecting or displaying personal cosmetic colours or hairstyle
• CPC Class Codes: A47F 10/00 Furniture or installations specially adapted to particular types of service systems, not otherwise provided for conveyor aspects B65G; building aspects E04H; coin-freed apparatus G07F
• CPC Class Codes: G02C 5/00 Constructions of non-optical parts
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

An eyeglasses simulation device, provided as an embodiment of a head-mounted object image synthesis device, measures, when combining eyeglasses with a face image of a user, a three-dimensional shape of a face of the person and its reflection characteristics, fits the eyeglasses selected by the user to the three-dimensional shape, estimates a position and an orientation of a face in the face image obtained by photographing the face of the person, produces an eyeglasses image in which the eyeglasses are arranged so as to match the face in the face image by the use of the estimated position and orientation, combines together the eyeglasses image and the face image, and displays a combined image. This creates a combined image of the face of the user and the eyeglasses like one as if the user actually put on the eyeglasses and confirmed the mounted state of the eyeglasses reflected in a mirror.

Claims

1. A head-mounted object image synthesis method comprising:

a step of measuring a three-dimensional shape of a face of a person;

a step of fitting eyeglasses to said measured three-dimensional shape;

a step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person; and

a step of producing the eyeglasses in the image so as to match the face by the use of said estimated position and orientation of the face, and displaying it.

4. The head-mounted object image synthesis method according to claim 1, wherein the step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person and the step of producing the eyeglasses in the image so as to match the face by the use of said estimated position and orientation of the face, and displaying it, are successively carried out.

5. The head-mounted object image synthesis method according to claim 1, wherein, instead of said eyeglasses, an earring, a headgear, a hairpiece, or another mount object for a head, or a combination thereof is combined with the image.

2. A head-mounted object image synthesis method comprising:

a step of measuring a three-dimensional shape of a face of a person and its reflection characteristics;

a step of fitting eyeglasses to said measured three-dimensional shape;

a step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person, by the use of said three-dimensional shape and said reflection characteristics; and

a step of producing the eyeglasses in the image so as to match the face by the use of said estimated position and orientation of the face, and displaying it.

3. The head-mounted object image synthesis method according to claim 2, comprising, instead of the step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person, by the use of said three-dimensional shape and said reflection characteristics:

a step of producing bases in advance from said three-dimensional shape and said reflection characteristics, said bases adapted for estimating positions and orientations of the face in various postures and under various illumination conditions; and

a step of estimating the position and the orientation of the face in the image by the use of said bases.

6. The head-mounted object image synthesis method according to claim 2, wherein a texture image with a little shade is used instead of said reflection characteristics.

29. The head-mounted object image synthesis method according to claim 2, wherein the step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person and the step of producing the eyeglasses in the image so as to match the face by the use of said estimated position and orientation of the face, and displaying it, are successively carried out.

30. The head-mounted object image synthesis method according to claim 2, wherein, instead of said eyeglasses, an earring, a headgear, a hairpiece, or another mount object for a head, or a combination thereof is combined with the image.

7. A makeup image synthesis method comprising:

a step of measuring a three-dimensional shape of a face of a person;

a step of applying makeup to said measured three-dimensional shape;

a step of fitting to said measured three-dimensional shape a face standard three-dimensional shape with a modification rule held in advance;

a step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person;

a step of estimating an expression change of the face in said image; and

a step of generating makeup in the image so as to match the face by the use of said estimated position and orientation of the face and said estimated expression change, and displaying it.

8. A makeup image synthesis method comprising:

a step of measuring a three-dimensional shape of a face of a person and its reflection characteristics;

a step of applying makeup to said measured three-dimensional shape;

a step of fitting to said measured three-dimensional shape a face standard three-dimensional shape with a modification rule held in advance;

a step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person, by the use of said measured three-dimensional shape and reflection characteristics;

a step of estimating an expression change of the face in said image; and

a step of generating makeup in the image so as to match the face by the use of said estimated position and orientation of the face and said estimated expression change, and displaying it.

9. The makeup image synthesis method according to claim 8, comprising, instead of the step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person, by the use of said three-dimensional shape and said reflection characteristics:

a step of producing bases in advance from said three-dimensional shape and said reflection characteristics, said bases adapted for estimating positions and orientations of the face in various postures and under various illumination conditions; and

a step of estimating the position and the orientation of the face in the image by the use of said bases.

10. The makeup image synthesis method according to claim 8, comprising, instead of the step of estimating an expression change of the face in the image obtained by photographing the face of said person:

a step of producing bases in advance from said measured three-dimensional shape and reflection characteristics, said bases adapted for estimating positions and orientations of the face in various postures and under various illumination conditions; and

a step of estimating the expression change of the face in the image by the use of said bases and said face standard three-dimensional shape with said modification rule.

11. The makeup image synthesis method according to any one of claim 8, wherein a texture image with a little shade is used instead of said reflection characteristics.

12. A head-mounted object image synthesis device comprising:

three-dimensional shape measuring means for measuring a three-dimensional shape of a face of a person;

eyeglasses selection and fitting means for fitting selected eyeglasses to said measured three-dimensional shape;

face position and orientation estimation means for estimating a position and an orientation of a face in a face image obtained by photographing the face of said person;

eyeglasses image generation means for generating an eyeglasses image in which the eyeglasses are arranged so as to match the face in said face image by the use of said estimated position and orientation of the face; and

face and eyeglasses combination display means for combining together said face image and said eyeglasses image and displaying a combined image.

15. The head-mounted object image synthesis device according to claim 12, wherein said face position and orientation estimation means, said eyeglasses image generation means, and said face and eyeglasses combination display means are successively carried out.

16. The head-mounted object image synthesis device according to claim 12, wherein, instead of said eyeglasses, an earring, a headgear, a hairpiece, or another mount object for a head, or a combination thereof is combined with the image.

13. A head-mounted object image synthesis device comprising:

three-dimensional shape and reflection characteristic measuring means for measuring a three-dimensional shape of a face of a person and its reflection characteristics;

eyeglasses selection and fitting means for fitting selected eyeglasses to said measured three-dimensional shape;

face position and orientation estimation means for estimating a position and an orientation of a face in a face image obtained by photographing the face of said person, by the use of said three-dimensional shape and said reflection characteristics;

eyeglasses image generation means for generating an eyeglasses image in which the eyeglasses are arranged so as to match the face in said face image by the use of said estimated position and orientation of the face; and

face and eyeglasses combination display means for combining together said face image and said eyeglasses image and displaying a combined image.

14. The head-mounted object image synthesis device according to claim 13, comprising, instead of said face position and orientation estimation means:

geodetic illumination calculation means for producing geodetic illumination bases in advance from said three-dimensional shape and said reflection characteristics, said bases adapted for estimating positions and orientations of the face in various postures and under various illumination conditions; and

face position and orientation estimation means for estimating the position and the orientation of the face in the face image by the use of said bases.

17. The head-mounted object image synthesis device according to claim 13, wherein a texture image with a little shade is used instead of said reflection characteristics.

31. The head-mounted object image synthesis device according to claim 13, wherein said face position and orientation estimation means, said eyeglasses image generation means, and said face and eyeglasses combination display means are successively carried out.

32. The head-mounted object image synthesis device according to claim 13, wherein, instead of said eyeglasses, an earring, a headgear, a hairpiece, or another mount object for a head, or a combination thereof is combined with the image.

18. A makeup image synthesis device comprising:

three-dimensional shape measuring means for measuring a three-dimensional shape of a face of a person;

makeup means for applying makeup to said measured three-dimensional shape;

face standard three-dimensional model and modification rule storage means for storing a face standard three-dimensional shape with a modification rule held in advance;

three-dimensional model fitting means for fitting said face standard three-dimensional model to said measured three-dimensional shape;

face position and orientation estimation means for estimating a position and an orientation of a face in a face image obtained by photographing the face of said person;

face component modification estimation means for estimating an expression change of the face in said face image;

makeup image generation means for generating a makeup image in which the makeup applied by said makeup means is modified so as to match the face in said face image by the use of said estimated position and orientation of the face and said estimated expression change; and

face and makeup combination display means for combining together said face image and said makeup image and displaying a combined image.

19. A makeup image synthesis device comprising:

three-dimensional shape measuring means for measuring a three-dimensional shape of a face of a person and its reflection characteristics;

makeup means for applying makeup to said measured three-dimensional shape;

face standard three-dimensional model and modification rule storage means for storing a face standard three-dimensional shape with a modification rule held in advance;

three-dimensional model fitting means for fitting said face standard three-dimensional model to said measured three-dimensional shape;

face position and orientation estimation means for estimating a position and an orientation of a face in a face image obtained by photographing the face of said person, by the use of said measured three-dimensional shape and reflection characteristics;

face component modification estimation means for estimating an expression change of the face in said face image;

makeup image generation means for generating a makeup image in which the makeup applied by said makeup means is modified so as to match the face in said face image by the use of said estimated position and orientation of the face and said estimated expression change; and

face and makeup combination display means for combining together said face image and said makeup image and displaying a combined image.

20. The makeup image synthesis device according to claim 19, comprising, instead of said face position and orientation estimation means:

geodetic illumination base calculation means for producing geodetic illumination bases in advance from said three-dimensional shape and said reflection characteristics, said bases adapted for estimating positions and orientations of the face in various postures and under various illumination conditions; and

face position and orientation estimation means for estimating the position and the orientation of the face in the face image by the use of said bases.

21. The makeup image synthesis device according to claim 19, comprising, instead of said face component modification estimation means:

geodetic illumination base calculation means for producing geodetic illumination bases in advance from said measured three-dimensional shape and reflection characteristics, said bases adapted for estimating positions and orientations of the face in various postures and under various illumination conditions; and

face component modification estimation means for estimating the expression change of the face in the face image by the use of said bases and said face standard three-dimensional shape with said modification rule.

22. The makeup image synthesis device according to claim 19, wherein a texture image with a little shade is used instead of said reflection characteristics.

23. A program for causing a computer to execute:

a step of measuring a three-dimensional shape of a face of a person;

a step of fitting eyeglasses to said measured three-dimensional shape;

a step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person; and

a step of producing the eyeglasses in the image so as to match the face by the use of said estimated position and orientation of the face, and displaying it.

24. A program for causing a computer to execute:

a step of measuring a three-dimensional shape of a face of a person and its reflection characteristics;

a step of fitting eyeglasses to said measured three-dimensional shape;

a step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person, by the use of said three-dimensional shape and said reflection characteristics; and

a step of producing the eyeglasses in the image so as to match the face by the use of said estimated position and orientation of the face, and displaying it.

25. The program according to claim 24, comprising, instead of the step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person, by the use of said three-dimensional shape and said reflection characteristics:

a step of producing bases in advance from said three-dimensional shape and said reflection characteristics, said bases adapted for estimating positions and orientations of the face in various postures and under various illumination conditions; and

a step of estimating the position and the orientation of the face in the image by the use of said bases.

26. A program for causing a computer to execute:

a step of measuring a three-dimensional shape of a face of a person;

a step of applying makeup to said measured three-dimensional shape;

a step of fitting to said measured three-dimensional shape a face standard three-dimensional shape with a modification rule held in advance;

a step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person;

a step of estimating an expression change of the face in said image; and

a step of generating makeup in the image so as to match the face by the use of said estimated position and orientation of the face and said estimated expression change, and displaying it.

27. A program for causing a computer to execute:

a step of measuring a three-dimensional shape of a face of a person and its reflection characteristics;

a step of applying makeup to said measured three-dimensional shape;

a step of fitting to said measured three-dimensional shape a face standard three-dimensional shape with a modification rule held in advance;

a step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person, by the use of said measured three-dimensional shape and reflection characteristics;

a step of estimating an expression change of the face in said image; and

a step of generating makeup in the image so as to match the face by the use of said estimated position and orientation of the face and said estimated expression change, and displaying it.

28. The program according to claim 27, comprising, instead of the step of estimating a position and an orientation of a face in an image obtained by photographing the face of said person, by the use of said three-dimensional shape and said reflection characteristics:

a step of producing bases in advance from said three-dimensional shape and said reflection characteristics, said bases adapted for estimating positions and orientations of the face in various postures and under various illumination conditions; and

a step of estimating the position and the orientation of the face in the image by the use of said bases.