Systems and methods for visualizing eyewear on a user

• US 10,685,457 B2
• Filed: 11/15/2018
• Issued: 06/16/2020
• Est. Priority Date: 11/15/2018
• Status: Active Grant

• Current Assignee: Vision Service Plan (VSP Global, Inc.)
• Original Assignee: Vision Service Plan (VSP Global, Inc.)
• Inventors: Cook, Andrew, Crain, Patrick V.
• Primary Examiner(s): Hon, Ming Y
• Application Number:
  US16/191,979
• Publication Number:
  US 20200160556A1
• Time in Patent Office: 579 Days
• CPC Class Codes: G02C 13/003 Measuring during assembly or fitting of spectacles
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G09G 2380/08 Biomedical applications
• CPC Class Codes: G09G 3/00 Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes

Abstract

An eyewear fit visualization system and method enable a user to visualize the fit and look of eyewear on the user'"'"'s face without physically wearing the eyewear. An image of a user'"'"'s face is converted to a three-dimensional map that is used to determine contact points of the eyewear to the user'"'"'s face. Using these contact points, the system and method displays the eyewear precisely in place on a reflected or displayed image of the user'"'"'s face to accurately simulate the look and fit of the eyewear.

Claims

1. A computer-implemented method for providing an eyewear visualization experience, the method comprising:

receiving an image of a user'"'"'s face positioned a distance from a first side of a viewing surface of a visualization device;

utilizing the image of the user'"'"'s face to create a 3D map of the image, the 3D map comprising a positioning of a plurality of features of the user'"'"'s face according to a plurality of three-dimensional measurements associated with the plurality of features of the user'"'"'s face;

utilizing the 3D map to create a view of the user'"'"'s face comprising a plurality of normal vectors;

analyzing the normal vectors in a given area for a disorganization of the plurality of normal vectors;

detecting the disorganization of the plurality of normal vectors indicative of a feature of the user'"'"'s face to identify the position of a feature of the user'"'"'s face;

utilizing the identification of the position of the feature of the user'"'"'s face, determining a position of remaining features of the user'"'"'s face of the plurality of features;

receiving an image of eyewear;

utilizing the image of the eyewear and the locations of the plurality of features of the user'"'"'s face, identifying a plurality of contact points of the eyewear to the user'"'"'s face;

utilizing the plurality of contact points of the eyewear to the user'"'"'s face to position the image of the eyewear on the display according to a position of the visualized image of the user'"'"'s face on the first side of the viewing surface; and

displaying the image of the eyewear on a display positioned a distance from a second side of the viewing surface at the eyewear placement position.

2. The computer-implemented method of claim 1, wherein receiving the image of the user'"'"'s face comprises receiving the image from a depth camera.

3. The computer-implemented method of claim 2, wherein the depth camera is secured to the visualization device.

4. The computer-implemented method of claim 1, wherein the image of eyewear comprises a 3D map of the eyewear, the 3D map of the eyewear comprises a plurality of three-dimensional measurements associated with a plurality of features of the eyewear.

5. The computer-implemented method of claim 1, further comprising:

determining a user'"'"'s eyewear size category according to the plurality of contact points of the eyewear to the user'"'"'s face;

selecting a subset of eyewear for presentation to the user according to the eyewear size category; and

providing the subset of eyewear to the user for visualization.

6. The computer-implemented method of claim 1, wherein utilizing the plurality of contact points of the eyewear to the user'"'"'s face to position the image of the eyewear on the display according to the position of the visualized image of the user'"'"'s face comprises:

determining the position of the visualized image of the user'"'"'s face according to the distance of the user'"'"'s face from the first side of the viewing surface and a positioning of the user'"'"'s face with respect to the viewing surface and the display;

determining positioning of a plurality of image contact points corresponding to the plurality of contact points on the visualized image of the user'"'"'s face; and

displaying the image of the eyewear on the display according to the positioning of the plurality of image contact points and a position of the display at the distance from the second side of the viewing surface.

7. The computer-implemented method of claim 6, wherein displaying the image of the eyewear on the display according to the positioning of the plurality of image contact points and the position of the display at the distance from the second side of the viewing surface comprises:

determining the distance from the second side of the viewing surface;

determining a size and orientation of the image of the eyewear to display;

displaying the image of the eyewear on the display at the determined distance from the second side of the viewing surface and according to the determined size and orientation.

8. The computer-implemented method of claim 7, further comprising:

detecting a change in the distance of the user'"'"'s face from the first side of the viewing surface; and

in response to detecting the change in the distance of the user'"'"'s face from the first side of the viewing surface,

determining the distance from the second side of the viewing surface;

determining the size and orientation of the image of the eyewear to display;

displaying the image of the eyewear on the display at the determined distance from the second side of the viewing surface and according to the determined size and orientation.

9. The computer-implemented method of claim 6, wherein determining positioning of a plurality of image contact points corresponding to the plurality of contact points on the visualized image of the user'"'"'s face comprises:

utilizing the distance of the user'"'"'s face from the first side of the viewing surface and a position of one or more of the user'"'"'s eyes, determining a configuration of the visualized image of the user'"'"'s face from a perspective of the one or more of the user'"'"'s eyes; and

determining the positioning of the plurality of image contact points according to the position of the visualized image of the user'"'"'s face and the configuration of the visualized image of the user'"'"'s face.

10. The computer-implemented method of claim 1, wherein the viewing surface comprises a reflective surface and the visualized image comprises a reflection of the user'"'"'s face.

11. The computer-implemented method of claim 1, wherein the viewing surface comprises a screen positioned in front of the display, wherein the screen is at least partially transparent, and wherein the visualized image comprises a displayed image of the user'"'"'s face utilizing a camera adjacent to or coupled to the visualization device.

12. An eyewear visualization system, comprising:

a visualization device comprising a viewing surface that is at least partially reflective and at least partially transmissive and having a first side a second side opposite the first side;

at least one camera configured to capture an image of a user'"'"'s face positioned a distance from the first side of the viewing surface;

a display positioned adjacent to and a determined distance from the second side of the viewing surface;

a processor communicatively coupled to the at least one camera and the display, and operative to;

receive the image of the user'"'"'s face from the at least one camera;

utilize the image of the user'"'"'s face to create a 3D map of the image, the 3D map comprising a positioning of a plurality of features of the user'"'"'s face according to a plurality of three-dimensional measurements associated with the plurality of features of the user'"'"'s face;

utilize the 3D map to create a view of the user'"'"'s face comprising a plurality of normal vectors;

analyze the normal vectors in a given area for a disorganization of the plurality of normal vectors;

detect the disorganization of the plurality of normal vectors indicative of a feature of the user'"'"'s face to identify the position of a feature of the user'"'"'s face;

utilize the identification of the position of the feature of the user'"'"'s face to determine a position of remaining features of the user'"'"'s face of the plurality of features;

receive an image of eyewear;

utilize the image of the eyewear and the 3D map of the image of the user'"'"'s face to determine an eyewear placement position corresponding to a positioning of a reflected image of the user'"'"'s face on the first side of the viewing surface; and

display the image of the eyewear on the display at an eyewear placement position according to the determined distance of the display from the second side of the viewing surface.

13. The eyewear visualization system of claim 12, wherein the image of eyewear comprises a 3D map of the eyewear, the 3D map of the eyewear comprises a plurality of three-dimensional measurements associated with a plurality of features of the eyewear.

14. The eyewear visualization system of claim 13, wherein determining the eyewear placement position corresponding to the positioning of the reflected image of the user'"'"'s face on the first side of the viewing surface comprises:

utilizing the 3D map of the eyewear and the 3D map of the image of the user'"'"'s face to identify a plurality of contact points of the eyewear to the user'"'"'s face; and

utilizing the plurality of contact points of the eyewear to the user'"'"'s face to position the image of the eyewear on the display according to a position of the reflected image of the user'"'"'s face.

15. The eyewear visualization system of claim 14, wherein utilizing the plurality of contact points of the eyewear to the user'"'"'s face to position the image of the eyewear on the display according to the position of the reflected image of the user'"'"'s face comprises:

determining the position of the reflected image of the user'"'"'s face according to the distance of the user'"'"'s face from the first side of the viewing surface and a positioning of the user'"'"'s face with respect to the viewing surface and the display;

determining positioning of a plurality of image contact points corresponding to the plurality of contact points on the reflected image of the user'"'"'s face; and

displaying the image of the eyewear on the display according to the positioning of the plurality of image contact points and a position of the display at the determined distance from the second side of the viewing surface.

16. The eyewear visualization system of claim 15, wherein displaying the image of the eyewear on the display according to the positioning of the plurality of image contact points and the position of the display at the determined distance from the second side of the viewing surface comprises:

determining the distance from the second side of the viewing surface;

determining a size and orientation of the image of the eyewear to display;

displaying the image of the eyewear on the display at the determined distance from the second side of the viewing surface and according to the determined size and orientation.

17. The eyewear visualization system of claim 16, further comprising:

detecting a change in the distance of the user'"'"'s face from the first side of the viewing surface; and

in response to detecting the change in the distance of the user'"'"'s face from the first side of the viewing surface,

determining the distance from the second side of the viewing surface;

determining the size and orientation of the image of the eyewear to display;

displaying the image of the eyewear on the display at the determined distance from the second side of the viewing surface and according to the determined size and orientation.

18. The eyewear visualization system of claim 15, wherein determining positioning of a plurality of image contact points corresponding to the plurality of contact points on the reflected image of the user'"'"'s face comprises:

utilizing the distance of the user'"'"'s face from the first side of the viewing surface and a position of one or more of the user'"'"'s eyes, determining a configuration of the reflected image of the user'"'"'s face from a perspective of the one or more of the user'"'"'s eyes; and

determining the positioning of the plurality of image contact points according to the position of the reflected image of the user'"'"'s face and the configuration of the reflected image of the user'"'"'s face.

19. An eyewear visualization system, comprising:

a visualization device comprising a screen having a first side a second side opposite the first side;

at least one camera configured to capture an image of a user'"'"'s face positioned a distance from the first side of the screen;

a display positioned adjacent to and a distance from the second side of the screen;

a processor communicatively coupled to the at least one camera and the display, and operative to;

receive the image of the user'"'"'s face from the at least one camera;

utilize the image of the user'"'"'s face to create a 3D map of the image, the 3D map comprising a positioning of a plurality of features of the user'"'"'s face according to a plurality of three-dimensional measurements associated with the plurality of features of the user'"'"'s face;

utilize the 3D map to create a mesh view of the user'"'"'s face comprising a plurality of normal vectors;

analyze the normal vectors in a given area for a disorganization of the plurality of normal vectors;

detect the disorganization of the plurality of normal vectors indicative of a feature of the user'"'"'s face to identify the position of a feature of the user'"'"'s face;

utilizing the identification of the position of the feature of the user'"'"'s face and the mesh view, determine a position of remaining features of the user'"'"'s face of the plurality of features;

receive an image of eyewear;

utilizing the image of the eyewear and the locations of the plurality of features of the user'"'"'s face, determine an eyewear placement position corresponding to a positioning of a displayed image of the user'"'"'s face on the first side of the screen; and

display the image of the eyewear on the display at an eyewear placement position.

20. The eyewear visualization system of claim 19, wherein the image of eyewear comprises a 3D map of the eyewear, the 3D map of the eyewear comprises a plurality of three-dimensional measurements associated with a plurality of features of the eyewear.

21. The eyewear visualization system of claim 20, wherein determining the eyewear placement position corresponding to the positioning of the displayed image of the user'"'"'s face on the first side of the screen comprises:

utilizing the 3D map of the eyewear and the locations of the plurality of features of the user'"'"'s face to identify a plurality of contact points of the eyewear to the user'"'"'s face; and

utilizing the plurality of contact points of the eyewear to the user'"'"'s face to position the image of the eyewear on the display according to a position of the displayed image of the user'"'"'s face.

22. The eyewear visualization system of claim 21, wherein utilizing the plurality of contact points of the eyewear to the user'"'"'s face to position the image of the eyewear on the display according to the position of the displayed image of the user'"'"'s face comprises:

determining the position of the displayed image of the user'"'"'s face according to the distance of the user'"'"'s face from the first side of the screen and a positioning of the user'"'"'s face with respect to the screen and the display;

determining positioning of a plurality of image contact points corresponding to the plurality of contact points on the displayed image of the user'"'"'s face; and

displaying the image of the eyewear on the display according to the positioning of the plurality of image contact points and a position of the display at the distance from the second side of the screen.

23. The eyewear visualization system of claim 22, wherein displaying the image of the eyewear on the display according to the positioning of the plurality of image contact points and the position of the display at the distance from the second side of the screen comprises:

determining the distance from the second side of the screen;

determining a size and orientation of the image of the eyewear to display;

displaying the image of the eyewear on the display at the determined distance from the second side of the screen and according to the determined size and orientation.

24. The eyewear visualization system of claim 23, further comprising:

detecting a change in the distance of the user'"'"'s face from the first side of the screen; and

in response to detecting the change in the distance of the user'"'"'s face from the first side of the screen,

determining the distance from the second side of the screen;

determining the size and orientation of the image of the eyewear to display;

displaying the image of the eyewear on the display at the determined distance from the second side of the screen and according to the determined size and orientation.