Using computed facial feature points to position a product model relative to a model of a face

• US 9,990,780 B2
• Filed: 10/03/2016
• Issued: 06/05/2018
• Est. Priority Date: 10/03/2016
• Status: Active Grant

• Current Assignee: Ditto Technologies
• Original Assignee: Ditto Technologies
• Inventors: Kornilov, Dmitry, Nechaev, Andrey
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US15/284,247
• Publication Number:
  US 20180096537A1
• Time in Patent Office: 610 Days
• Field of Search: None
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Using computed facial feature points to position a product model relative to a model of a face is disclosed, comprising: obtaining a three-dimensional (3D) model of a user'"'"'s face, wherein the 3D model of the user'"'"'s face comprises a plurality of 3D points; determining a face normal that is normal to a plane that is determined based at least in part on a first subset of 3D points from the plurality of 3D points; determining a set of computed bridge points based at least in part on a second subset of 3D points from the plurality of 3D points and the face normal; and using the set of computed bridge points to determine an initial placement of a 3D model of a glasses frame relative to the 3D model of the user'"'"'s face.

Claims

1. A system, comprising:

a processor configured to;

obtain a three-dimensional (3D) model of a user'"'"'s face, wherein the 3D model of the user'"'"'s face comprises a plurality of 3D points;

determine a face normal that is normal to a plane, wherein the plane is determined based at least in part on a first subset of 3D points from the plurality of 3D points;

determine a set of computed bridge points based at least in part on a second subset of 3D points from the plurality of 3D points and the face normal; and

use the set of computed bridge points to determine an initial placement of a 3D model of a glasses frame relative to the 3D model of the user'"'"'s face; and

a memory coupled to the processor and configured to store the 3D model of the user'"'"'s face.

2. The system of claim 1, wherein the 3D model of the user'"'"'s face is determined from a set of images that includes the user'"'"'s face.

3. The system of claim 1, wherein the plane is determined based at least in part on a corresponding distance between the plane and each 3D point in the first subset of 3D points.

4. The system of claim 1, wherein the first subset of 3D points comprises a set of internal eye corner points and a set of ear juncture points, wherein the second subset of 3D points comprises the set of internal eye corners points and a set of internal eyebrow points, wherein to determine the set of computed bridge points based at least in part on the second subset of 3D points and the face normal comprises to:

compute a distance along the face normal between the set of internal eyebrow points and the set of internal eye corner points;

determine a bridge shift based at least in part on the distance; and

apply the bridge shift to each of the set of internal eye corner points along the face normal.

5. The system of claim 1, wherein to use the set of computed bridge points to determine the initial placement of the 3D model of the glasses frame relative to the 3D model of the user'"'"'s face comprises to:

align the set of computed bridge points and a set of bridge points of the 3D model of the glasses frame; and

adjust a set of temple bend points of the 3D model of the glasses frame based at least in part on a set of ear juncture points from the 3D model of the user'"'"'s face.

6. The system of claim 1, wherein the processor is further configured to:

morph a predetermined 3D face to correspond to the 3D model of the user'"'"'s face to obtain a morphed 3D face, wherein the predetermined 3D face includes a plurality of 3D points along a facial feature; and

determine a morphed set of 3D points along a morphed facial feature of the morphed 3D face based at least in part on the plurality of 3D points along the facial feature.

7. The system of claim 6, wherein the processor is further configured to adjust the initial placement of the 3D model of the glasses frame relative to the 3D model of the user'"'"'s face based at least in part on the morphed set of 3D points along the morphed facial feature of the morphed 3D face.

8. The system of claim 6, wherein the facial feature comprises a nose curvature, wherein the processor is further configure to:

determine a segment that is closest to the initial placement of the 3D model of the glasses frame, wherein the segment is defined between two points from the morphed set of 3D points along the morphed facial feature of the morphed 3D face;

determine a nose curvature normal based at least in part on a normal of the segment;

adjust the initial placement of the 3D model of the glasses frame along the nose curvature normal of the segment towards the segment; and

adjust a set of temple bend points of the initial placement of the 3D model of the glasses frame based at least in part on a set of ear juncture points from the 3D model of the user'"'"'s face.

9. The system of claim 1, wherein the processor is further configured to:

render a two-dimensional (2D) image associated with the glasses frame corresponding to an image of the user'"'"'s face using at least the initial placement of the 3D model of the glasses frame relative to the 3D model of the user'"'"'s face, a set of extrinsic information corresponding to the image of the user'"'"'s face, and a 3D face; and

overlay the 2D image of the glasses frame over the image of the user'"'"'s face.

10. The system of claim 9, wherein the set of extrinsic information corresponding to the image of the user'"'"'s face describes an orientation of the user'"'"'s face in the image.

21. The system of claim 1, wherein the determination of an initial placement of a 3D model of a glasses frame relative to the 3D model of the user'"'"'s face is based at least in part on aligning the set of computed bridge points with bridge points of the 3D model of the glasses frame.

11. A method, comprising:

obtaining a three-dimensional (3D) model of a user'"'"'s face, wherein the 3D model of the user'"'"'s face comprises a plurality of 3D points;

determining a face normal that is normal to a plane, wherein the plane is determined based at least in part on a first subset of 3D points from the plurality of 3D points;

determining a set of computed bridge points based at least in part on a second subset of 3D points from the plurality of 3D points and the face normal; and

using the set of computed bridge points to determine an initial placement of a 3D model of a glasses frame relative to the 3D model of the user'"'"'s face.

12. The method of claim 11, wherein the 3D model of the user'"'"'s face is determined from a set of images that includes the user'"'"'s face.

13. The method of claim 11, wherein the plane is determined based at least in part on a corresponding distance between the plane and each 3D point in the first subset of 3D points.

14. The method of claim 11, wherein the first subset of 3D points comprises a set of internal eye corner points and a set of ear juncture points, wherein the second subset of 3D points comprises the set of internal eye corners points and a set of internal eyebrow points, wherein determining the set of computed bridge points based at least in part on the second subset of 3D points and the face normal comprises:

computing a distance along the face normal between the set of internal eyebrow points and the set of internal eye corner points;

determining a bridge shift based at least in part on the distance; and

applying the bridge shift to each of the set of internal eye corner points along the face normal.

15. The method of claim 11, wherein using the set of computed bridge points to determine the initial placement of the 3D model of the glasses frame relative to the 3D model of the user'"'"'s face comprises:

aligning the set of computed bridge points and a set of bridge points of the 3D model of the glasses frame; and

adjusting a set of temple bend points of the 3D model of the glasses frame based at least in part on a set of ear juncture points from the 3D model of the user'"'"'s face.

16. The method of claim 11, further comprising:

morphing a predetermined 3D face to correspond to the 3D model of the user'"'"'s face to obtain a morphed 3D face, wherein the predetermined 3D face includes a plurality of 3D points along a facial feature; and

determining a morphed set of 3D points along a morphed facial feature of the morphed 3D face based at least in part on the plurality of 3D points along the facial feature.

17. The method of claim 16, further comprising adjusting the initial placement of the 3D model of the glasses frame relative to the 3D model of the user'"'"'s face based at least in part on the morphed set of 3D points along the morphed facial feature of the morphed 3D face.

18. The method of claim 16, wherein the facial feature comprises a nose curvature, and further comprising:

determining a segment that is closest to the initial placement of the 3D model of the glasses frame, wherein the segment is defined between two points from the morphed set of 3D points along the morphed facial feature of the morphed 3D face;

determining a nose curvature normal based at least in part on a normal of the segment;

adjusting the initial placement of the 3D model of the glasses frame along the nose curvature normal of the segment towards the segment; and

adjusting a set of temple bend points of the initial placement of the 3D model of the glasses frame based at least in part on a set of ear juncture points from the 3D model of the user'"'"'s face.

19. The method of claim 11, further comprising:

rendering a two-dimensional (2D) image associated with the glasses frame corresponding to an image of the user'"'"'s face using at least the initial placement of the 3D model of the glasses frame relative to the 3D model of the user'"'"'s face, a set of extrinsic information corresponding to the image of the user'"'"'s face, and a 3D face; and

overlaying the 2D image of the glasses frame over the image of the user'"'"'s face.

20. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

obtaining a three-dimensional (3D) model of a user'"'"'s face, wherein the 3D model of the user'"'"'s face comprises a plurality of 3D points;

determining a face normal that is normal to a plane, wherein the plane is determined based at least in part on a first subset of 3D points from the plurality of 3D points;

determining a set of computed bridge points based at least in part on a second subset of 3D points from the plurality of 3D points and the face normal; and

using the set of computed bridge points to determine an initial placement of a 3D model of a glasses frame relative to the 3D model of the user'"'"'s face.