2D IMAGE-BASED 3D GLASSES VIRTUAL TRY-ON SYSTEM

• US 20160035133A1
• Filed: 07/31/2014
• Published: 02/04/2016
• Est. Priority Date: 07/31/2014
• Status: Active Grant

• Current Assignee: ULSee Inc.
• Original Assignee: ULSee Inc.
• Inventors: Ye, Zhou, Chang, Chih-Ming, Lu, Ying-Ko, Hsu, Yi-Chia
• Granted Patent: US 9,665,984 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/12 Digital output to print unit , e.g. line printer, chain printer
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/16 Cloth

Abstract

Method to create try-on experience wearing virtual 3D eyeglasses is provided using 2D image data of eyeglasses. Virtual 3D eyeglasses are constructed using set of 2D images for eyeglasses. Virtual 3D eyeglasses is configured onto 3D face or head model and being simulated as being fittingly worn by the wearer. Each set of 2D images for eyeglasses includes a pair of 2D lens images, a frontal frame image, and at least one side frame image. Upon detection of a movement of the face and head of wearer in real-time, the 3D face or head model and the configuration and alignment of virtual 3D eyeglasses are modified or adjusted accordingly. Features such as trimming off of portion of the glasses frame, shadow creating and environment mapping are provided to the virtual 3D eyeglasses in response to translation, scaling, and posture changes made to the head and face of the wearer in real-time.

Claims

1. A method to create real-time try-on experience of wearing virtual 3D eyeglasses by a wearer, comprising:

obtaining a plurality of 2D images for a plurality of pairs of eyeglasses, the pairs of eyeglasses are organized into a group of 2D images, each pair of eyeglasses has a set of 2D images;

when one designed pair of eyeglasses from the group of 2D images is selected by the wearer, constructing a pair of virtual 3D eyeglasses using the set of 2D images for the designed pair of eyeglasses;

constructing a 3D face or head model of the wearer based upon one or more facial or head images of the wearer; and

fitting the pair of virtual 3D eyeglasses onto the 3D face or head model of the wearer, with the pair of virtual 3D eyeglasses being simulated as being worn by the wearer;

wherein each set of the 2D images for each pair of eyeglasses comprises a frontal frame image and at least one side frame image.

2. The method as claimed in claim 1, wherein the step of fitting the pair of virtual 3D eyeglasses onto the 3D face or head model of the wearer comprises:

rotating the virtual 3D eyeglasses together with the 3D face or head model according to a change in real-time posture of the face of the wearer.

3. The method as claimed in claim 1, wherein the step of fitting the pair of virtual 3D eyeglasses onto the 3D face or head model of the wearer comprises:

moving the virtual 3D eyeglasses together with the 3D face or head model according to the real-time posture of the face of the wearer and one reference point of the face.

4. The method as claimed in claim 1, wherein the step of fitting the pair of virtual 3D eyeglasses onto the 3D face or head model of the wearer comprises:

zooming in and out the virtual 3D eyeglasses together with the 3D face or head model according to a change in real-time scaling size of the face of the wearer.

5. The method as claimed in claim 1, wherein while the pair of virtual 3D eyeglasses is being simulated as being fittingly worn by the wearer, a right-side glasses frame portion inside the face is trimmed off upon a rotation angle of the 3D face or head model of the wearer is at least 5 degrees to the right.

6. The method as claimed in claim 1, wherein while the pair of virtual 3D eyeglasses is being simulated as being fittingly worn by the wearer, a left-side glasses frame portion inside the face is trimmed off upon a rotation angle of the 3D face or head model of the wearer is at least 5 degrees to the left.

7. The method as claimed in claim 1, wherein each set of the 2D images for each pair of eyeglasses further comprises a pair of 2D lens images;

and the pair of 2D lens images of the set of 2D images are based on 2D image that are set at a designated alpha channel value to change the transparency of the lenses of the pair of 3D virtual eyeglasses.

8. The method as claimed in claim 1, wherein each set of the 2D images for each pair of eyeglasses further comprises a pair of 2D lens images;

and an environmental mapping effect is applied to the pair of lens images to configure and show a transient modified lens image in place of the 2D lens image in real-time upon satisfying a set criteria according to the real-time posture of the face of the wearer.

9. The method as claimed in claim 1, wherein a shadow effect is applied to the face image of the wearer in real-time using the 3D face or head model.

10. An augmented reality system for real-time try-on of 3D virtual eyeglasses by a wearer, comprising:

a server, the server having a first memory, a plurality of 2D images for a plurality of pairs of eyeglasses are organized into a group of 2D images, the group of 2D images are stored in the first memory; and

a host, the host comprising a 3D virtual eyeglasses try-on simulation program, a processor, a camera, a display device and a second memory,

wherein each pair of eyeglasses has a set of 2D images in the group of 2D images, the set of 2D images is stored in the second memory;

the processor is configured to run the 3D virtual eyeglasses try-on simulation program for constructing a pair of virtual 3D eyeglasses using the set of 2D images for one designed pair of eyeglasses retrieved from the second memory when the designed pair of eyeglasses from the group of 2D images is selected by the wearer, constructing a 3D face or head model of the wearer based upon one or more facial and head images of the wearer captured by the camera in real-time, configuring the pair of virtual 3D eyeglasses onto the 3D face or head model of the wearer, with the pair of virtual 3D eyeglasses being simulated as being fittingly worn by the wearer;

the display device is configured to display the wearer trying-on the virtual 3D eyeglasses in real-time; and

each set of the 2D images for each pair of eyeglasses stored in the first memory and in the second memory, respectively, comprising a frontal frame image and at least one side frame image.

11. The augmented reality system as claimed in claim 10, wherein the processor is configured to run the 3D virtual eyeglasses try-on simulation program for rotating the virtual 3D eyeglasses together with the 3D face or head model according to a change in real-time posture of the face of the wearer.

12. The augmented reality system as claimed in claim 10, wherein the processor is configured to run the 3D virtual eyeglasses try-on simulation program for moving the virtual 3D eyeglasses together with the 3D face or head model according to the real-time posture of the face of the wearer and one reference point of the face.

13. The augmented reality system as claimed in claim 10, wherein the processor is configured to run the 3D virtual eyeglasses try-on simulation program for zooming in and out the virtual 3D eyeglasses together with the 3D face or head model according to a change in real-time scaling size of the face of the wearer.

14. The augmented reality system as claimed in claim 10, wherein while the pair of virtual 3D eyeglasses is being simulated as being fittingly worn by the wearer, a right-side glasses frame portion inside the face is trimmed off upon a rotation angle of the 3D face or head model of the wearer is at least 5 degrees to the right.

15. The augmented reality system as claimed in claim 10, wherein while the pair of virtual 3D eyeglasses is being simulated as being fittingly worn by the wearer, a left-side glasses frame portion inside the face is trimmed off upon a rotation angle of the 3D face or head model of the wearer is at least 5 degrees to the left.

16. The augmented reality system as claimed in claim 10, wherein each set of the 2D images for each pair of eyeglasses further comprises a pair of 2D lens images;

and the pair of 2D lens images of the set of 2D images are based on 2D image that are set at a designated alpha channel value to change the transparency of the lenses of the pair of 3D virtual eyeglasses.

17. The augmented reality system as claimed in claim 10, wherein each set of the 2D images for each pair of eyeglasses further comprises a pair of 2D lens images;

and an environmental mapping effect is applied to the pair of lens images to configure and show a transient modified lens image in place of the 2D lens image in real-time upon satisfying a set criteria according to the real-time posture of the face of the wearer.

18. The augmented reality system as claimed in claim 10, wherein a shadow effect is applied to the face image of the wearer in real-time using the 3D face or head model.

19. The augmented reality system as claimed in claim 10, wherein the 3D virtual eyeglasses try-on simulation program is an app, and the host is a mobile phone or an electronic wireless device.

20. The augmented reality system as claimed in claim 10, wherein the 3D virtual eyeglasses try-on simulation program is a webpage, the webpage comprising a virtual mirror for trying-on 3D virtual eyeglasses by the wearer, the virtual minor is configured under a webcam mode or a virtual model mode, upon operating under the webcam mode, real-time facial tracking is performed and a 3D face or head model of the wearer based upon one or more facial or head images of the wearer is generated based upon the real-time facial tracking results.