Process and method for real-time physically accurate and realistic-looking glasses try-on

• US 10,042,188 B2
• Filed: 02/23/2016
• Issued: 08/07/2018
• Est. Priority Date: 02/23/2015
• Status: Active Grant

• Current Assignee: FittingBox S.A.
• Original Assignee: FittingBox S.A.
• Inventors: Choukroun, Ariel, Guenard, Jerome, Le Gallou, Sylvain, Dehais, Christophe
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): ROBINSON, TERRELL M
• Application Number:
  US15/051,325
• Publication Number:
  US 20160246078A1
• Time in Patent Office: 896 Days
• Field of Search: 345632
• CPC Class Codes: G02C 13/003 Measuring during assembly or fitting of spectacles
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method for generating a real-time realistic-looking image of a pair of virtual glasses positioned on a real user face image issued from a single image, a bundle of pictures, a video clip or a real-time camera streams. A face analysis element estimates the 3D rotation matrix, the 3D translation vector, morphology parameters of the user'"'"'s face. A glasses positioning element associated with a try-on mode for look or real-size simulation estimates the 3D rotation matrix, the 3D translation vector, global and local deformation parameters of the virtual glasses, and arms articulation, adapted to the morphology parameters computed by the face analysis element. A realistic rendering element mixes the real user face image with the virtual glasses positioned by the glasses positioning element.

Claims

1. A method for generating a real-time realistic-looking image of a pair of virtual glasses defined as a 3D object comprising two arms, and positioned on a real user face image issued from an image, which is one of a single image, a bundle of pictures, a video clip or real-time camera streams, comprising the steps of:

estimating facial parameters comprising a 3D rotation matrix, a 3D translation vector and morphology parameters, of a user'"'"'s face by a face analysis element;

estimating glasses positioning parameters comprising a 3D rotation matrix, a 3D translation vector, global and local deformation parameters of the virtual glasses, and arms articulation, adapted to the morphology parameters, by glasses positioning element associated with a try-on mode for look or real-size simulation;

mixing the real user face image with the virtual glasses positioned in accordance with the glasses positioning parameters by a realistic rendering element;

applying an opacity correction formula to translucent parts of the glasses; and

wherein global and local deformation parameters of the virtual glasses comprise non-rigid deformation of arms of the virtual glasses.

2. The method according to claim 1, wherein the facial parameters further comprise expression parameters;

and further comprises the step of estimating the expression parameters of the user'"'"'s face by the face analysis element.

3. The method according to claim 1, further comprising the step of estimating intrinsic camera parameters by the face analysis element.

4. The method according to claim 1, further comprising the step of providing parameterization, segmentation into parts, and mesh correction by a glasses 3D model analysis element of the glasses positioning element.

5. The method according to claim 1, wherein the step of estimating the glasses positioning parameters are performed by two glasses positioning elements.

6. The method according to claim 1, further comprising the step of providing a position, a rotation, and a face parameters refinement from multiple images in real-time by a 3D face tracking element.

7. The method according to claim 1, further comprising the step of estimating opacity correction and tone mapping parameters by a realistic rendering element.

8. The method according to claim 1, further comprising the steps of:

performing an image acquisition decorrelation, a scene analysis, and rendering in parallel;

recovering a 3D face pose; and

providing, at each frame, a smooth 3D pose to the realistic rendering element by a head movement and control filter.

9. The method according to claim 2, further comprising the step of estimating the facial parameters of the user'"'"'s face using a user face model, image contours, and 3D semantic points in correspondence with detected features from the image by the face analysis element.

10. The method according to claim 9, further comprising the step of estimating the facial parameters of the user'"'"'s face using the user face model, the image contours and the 3D semantic points in correspondence with the detected features from more than one image by the face analysis element.

11. The method according to claim 1, further comprising the steps of:

offline processing to generate a symmetrized and segmented object of the virtual glasses, comprising arms and a frame, from a 3D model, and to reposition the object in a reference system;

online real-time processing to estimate the glasses positioning parameters without interpenetration, comprising the 3D rotation matrix, the 3D translation vector, an arm opening angle, and the global and local deformation parameters, by mixing mechanical constraints and constraints linked to properties of the object.

12. The method according to claim 11, wherein the offline processing step further comprising the steps of:

automatic segmentation of the object;

extracting an object skeleton by slicing the object and linking barycenters;

computing a reference frame and a non-linear deformation field that maximizes a global symmetry in accordance with a symmetrization of the object skeleton; and

computing a displacement of object vertices in accordance with an influence function of a displacement field of the object skeleton.

13. The method according to claim 11, wherein the online processing step comprises the step of fitting the virtual glasses to the user'"'"'s face where a scale between the virtual glasses and the user'"'"'s face is ignored, and the frame is positioned on the user'"'"'s face with an arms opening of 5 degrees to account for a widening at temples of the user'"'"'s face.

14. The method according to claim 11, wherein the online processing step comprises the step of fitting the virtual glasses to the user'"'"'s face with a relative size where a face size of the user'"'"'s face is not known, the virtual glasses are positioned and scaled for an average face size of a predetermined face category, and an arms opening is computed in accordance with the global and local deformation parameters of the virtual glasses utilizing the relative size.

15. The method according to claim 11, wherein the online processing step comprises the step of fitting the virtual glasses to the user'"'"'s face with an absolute size where a face size of the user'"'"'s face is known.

16. The method according to claim 1, further comprising the step of modifying an opacity value α

into a corrected value α

_LDR of objects using the formula;

17. A method for generating a real-time realistic-looking image of a pair of virtual glasses defined as a 3D object comprising two arms, and positioned on a real user face image issued from an image, which is one of a single image, a bundle of pictures, a video clip or real-time camera streams, comprising the steps of:

estimating facial parameters comprising a 3D rotation matrix, a 3D translation vector and morphology parameters, of a user'"'"'s face by a face analysis element;

estimating glasses positioning parameters comprising a 3D rotation matrix, a 3D translation vector, global and local deformation parameters of the virtual glasses, and arms articulation, adapted to the morphology parameters, by glasses positioning element associated with a try-on mode for look or real-size simulation;

mixing the real user face image with the virtual glasses positioned in accordance with the glasses positioning parameters by a realistic rendering element;

modifying an opacity value α

into a corrected value α

_LDR of objects using the formula;

18. The method according to claim 17, wherein the facial parameters further comprise expression parameters;

and further comprises the step of estimating the expression parameters of the user'"'"'s face by the face analysis element.

19. The method according to claim 17, further comprising the step of estimating intrinsic camera parameters by the face analysis element.

20. The method according to claim 17, further comprising the step of providing parameterization, segmentation into parts, and mesh correction by a glasses 3D model analysis element of the glasses positioning element.

21. The method according to claim 17, wherein the step of estimating the glasses positioning parameters are performed by two glasses positioning elements.

22. The method according to claim 17, further comprising the step of providing a position, a rotation, and a face parameters refinement from multiple images in real-time by a 3D face tracking element.

23. The method according to claim 17, further comprising the step of estimating opacity correction and tone mapping parameters by a realistic rendering element.

24. The method according to claim 17, further comprising the steps of:

performing an image acquisition decorrelation, a scene analysis, and rendering in parallel;

recovering a 3D face pose; and

providing, at each frame, a smooth 3D pose to the realistic rendering element by a head movement and control filter.

25. The method according to claim 18, further comprising the step of estimating the facial parameters of the user'"'"'s face using a user face model, image contours, and 3D semantic points in correspondence with detected features from the image by the face analysis element.

26. The method according to claim 25, further comprising the step of estimating the facial parameters of the user'"'"'s face using the user face model, the image contours and the 3D semantic points in correspondence with the detected features from more than one image by the face analysis element.

27. The method according to claim 17, further comprising the steps of:

offline processing to generate a symmetrized and segmented object of the virtual glasses, comprising arms and a frame, from a 3D model, and to reposition the object in a reference system;

online real-time processing to estimate the glasses positioning parameters without interpenetration, comprising the 3D rotation matrix, the 3D translation vector, an arm opening angle, and the global and local deformation parameters, by mixing mechanical constraints and constraints linked to properties of the object.

28. The method according to claim 27, wherein the offline processing step further comprising the steps of:

automatic segmentation of the object;

extracting an object skeleton by slicing the object and linking barycenters;

computing a reference frame and a non-linear deformation field that maximizes a global symmetry in accordance with a symmetrization of the object skeleton; and

computing a displacement of object vertices in accordance with an influence function of a displacement field of the object skeleton.

29. The method according to claim 27, wherein the online processing step comprises the step of fitting the virtual glasses to the user'"'"'s face where a scale between the virtual glasses and the user'"'"'s face is ignored, and the frame is positioned on the user'"'"'s face with an arms opening of 5 degrees to account for a widening at temples of the user'"'"'s face.

30. The method according to claim 27, wherein the online processing step comprises the step of fitting the virtual glasses to the user'"'"'s face with a relative size where a face size of the user'"'"'s face is not known, the virtual glasses are positioned and scaled for an average face size of a predetermined face category, and an arms opening is computed in accordance with the global and local deformation parameters of the virtual glasses utilizing the relative size.

31. The method according to claim 27, wherein the online processing step comprises the step of fitting the virtual glasses to the user'"'"'s face with an absolute size where a face size of the user'"'"'s face is known.

32. The method according to claim 17, further comprising the step of applying an opacity correction formula to the translucent parts of the glasses.