Systems and methods for generating a 3-D model of a user for a virtual try-on product

• US 10,147,233 B2
• Filed: 04/28/2015
• Issued: 12/04/2018
• Est. Priority Date: 05/23/2012
• Status: Active Grant

• Current Assignee: Luxottica Retail North America Incorporated (EssilorLuxottica SA)
• Original Assignee: Glasses.com (EssilorLuxottica SA)
• Inventors: Engle, Ryan, Turetzky, Darren
• Primary Examiner(s): Broome, Said
• Assistant Examiner(s): Doan, Phuc N
• Application Number:
  US14/698,655
• Publication Number:
  US 20150235428A1
• Time in Patent Office: 1,316 Days
• CPC Class Codes: G02C 13/003 Measuring during assembly or fitting of spectacles
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/30 Polynomial surface description
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2210/16 Cloth
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

A computer-implemented method for generating a three-dimensional (3-D) model of a user. A plurality of images of a user are obtained. An angle of view relative to the user pictured in at least one of the plurality of images is calculated. It is determined whether the calculated angle of view matches a predetermined viewing angle. Upon determining the calculated angle of view matches the predetermined viewing angle, at least one of the plurality of images is selected.

Claims

1. A computing device configured to generate a three-dimensional (3-D) model of a user, comprising:

a processor; and

memory in electronic communication with the processor, wherein the memory is configured to store a plurality of geometry files and instructions, the instructions being executable by the processor to;

obtain a plurality of images of the user, wherein the images of the user have one or more angles of view;

generate the 3-D model of the user using the plurality of images, the instructions to generate comprising instructions to;

select one or more physical characteristics of the user from the plurality of images of the user;

calculate a respective coefficient for each of the plurality of geometry files based on the one or more physical characteristics of the user;

track the one or more physical characteristics of the user across at least two images of the plurality of images of the user, based on the plurality of geometry files and an algorithm; and

combine the plurality of geometry files based on at least one of the respective coefficients to create the 3-D model to represent the user; and

generate a combined image of the user together with a product, the combined image comprising the 3-D model of the user and a rendered image of a 3-D model of the product.

2. The device of claim 1, wherein the instructions to generate the 3-D model of the user further comprise instructions to provide a universal morphable model;

and

instructions to adjust the universal morphable model based on the respective coefficients of each of the plurality of geometry files.

3. The device of claim 1, wherein the one or more physical characteristics of the user correspond to at least one of the plurality of images of the user;

wherein the instructions to select the one or more physical characteristics of the user comprise;

instructions to detect a feature of the user; and

instructions to track the detected feature of the user between at least two of the plurality of images of the user.

4. The device of claim 1, wherein the processor is further configured to calculate an angle of view of a camera used to capture the plurality of images relative to the user.

5. The device of claim 3, wherein the processor is further configured to perform a cross-correlation algorithm to track the detected feature.

6. The computing device of claim 1, wherein each of the plurality of geometry files comprises a different polygon mesh model with different features.

7. The computing device of claim 6, wherein the plurality of geometry files further comprises at least one first polygon mesh depicting a first set of facial characteristics and at least one second polygon mesh depicting a second set of facial characteristics.

8. The computing device of claim 1, further comprising instructions stored in the memory, the instructions being executable by the processor to linearly combine the plurality of geometry files based on each respective coefficient.

9. The computing device of claim 8, wherein an effect of each of the geometry files on the 3-D model of the user is weighted, based on each respective coefficient.

10. The computing device of claim 6, wherein each polygon mesh model comprises at least one of a plurality of vertices and a plurality of texture coordinates.

11. The computing device of claim 1, further comprising a universal morphable model, wherein the universal morphable model comprises at least one of a plurality of vertices and a plurality of texture coordinates corresponding to at least one of the geometry files.

12. The computing device of claim 1, wherein the instructions being executable by the processor to select one or more physical characteristics of the user further comprise instructions executable by the processor to:

determine a 3-D structure of the user from the plurality of images; and

select one or more physical characteristics of the user from the determined 3-D structure of the user.

13. The computing device of claim 1, wherein the algorithm to track the one or more physical characteristics comprises at least one of a cross-correlation algorithm and a structure-from-motion algorithm.

14. The computing device of claim 1, wherein the processor is further configured to determine whether an angle of view of a camera used to capture an image of the plurality of images relative to the user is one of a predetermined set of viewing angles.

15. A method comprising:

obtaining a plurality of images of a user, wherein the images of the user have one or more angles of view;

generating, by at least one processor, a 3-D model of the user using the plurality of images, the generating comprising;

selecting one or more physical characteristics of the use from the plurality of images of the user;

accessing a plurality of geometry files stored in a memory;

calculating a respective coefficient for each of the plurality of geometry files based on the one or more physical characteristics of the user;

tracking the one or more physical characteristics of the user across at least two images of the plurality of images of the user, based on the plurality of geometry files and an algorithm; and

combining the plurality of geometry files based on at least one of the respective coefficients to create the 3D model of the user; and

generating, by the at least one processor, a combined image of the user together with a product, the combined image comprising the generated 3-D model of the user and a rendered image of a 3-D model of the product.

16. The method of claim 15, wherein the generating the 3-D model of the user further comprises providing a universal morphable model;

and

adjusting the universal morphable model based on at least one of the respective coefficients of at least one of the plurality of geometry files.

17. The method of claim 15, wherein the one or more physical characteristics of the user correspond to at least one of the plurality of images of the user, wherein selecting the one or more physical characteristics of the user comprises:

detecting a feature of the user.

18. The method of claim 15, wherein each of the plurality of geometry files comprises a different generic polygon mesh model with different features.

19. The method of claim 18, wherein the plurality of geometry files include at least one polygon mesh depicting a first set of facial characteristics and at least one polygon mesh depicting a second set of facial characteristics.

20. The method of claim 15, further comprising the processor linearly combining the plurality of geometry files based on each respective coefficient.

21. The method of claim 20, further comprising the processor weighting, based on the respective coefficients, an effect of each of the geometry files on the 3-D model of the user.

22. The method of claim 15, wherein selecting one or more physical characteristics of the user further comprises:

determining a 3-D structure of the user from the plurality of images; and

selecting one or more physical characteristics of the user from the determined 3-D structure of the user.

23. The method of claim 15, wherein the algorithm to track the one or more physical characteristics comprises at least one of a cross-correlation algorithm and a structure-from-motion algorithm.

24. The method of claim 15, wherein the processor is further configured to determine whether an angle of view of a camera used to capture an image of the plurality of images relative to the user is one of a predetermined set of viewing angles.

25. A computer-program product comprising a non-transitory computer-readable medium storing instructions thereon, the instructions being executable by the processor to:

obtain a plurality of images of a user, wherein the images of the user have one or more angles of view;

generate a 3-D model of the user using the plurality of images, the instructions to generate comprising instructions to;

select one or more physical characteristics of the user from the plurality of images of the user;

calculate a respective coefficient for each of a plurality of geometry files, based on the one or more physical characteristics of the user;

track the one or more physical characteristics of the user across at least two images of the plurality of images of the user, based on the plurality of geometry files and an algorithm; and

combine the plurality of geometry files based on the respective coefficients to create the 3-D model to represent the user; and

generate a combined image of the user with a product, the combined image comprising the 3-D model of the user and a rendered image of a 3-D model of the product.

26. The computer-program product of claim 25, wherein the instructions to generate the 3-D model of the user further comprises instructions being executable by the processor to:

provide a universal morphable model; and

adjust the universal morphable model based on the respective coefficients of each of the plurality of geometry files.

27. The computer-program product of claim 26, wherein the instructions to select one or more physical characteristics are executable by the processor to:

detect a feature of the user.

28. The computer-program product of claim 27, wherein the instructions to generate the 3-D model are further executable by the processor to:

linearly combine the plurality of geometry files based on each respective coefficient; and

weight an effect of each of the geometry files on the 3-D model to represent the user.

29. The computer-program product of claim 25, wherein the instructions to select one or more physical characteristics of the user further comprise instructions executable by the processor to:

determine a 3-D structure of the user from the plurality of images; and

select one or more physical characteristics of the user from the determined 3-D structure of the user.

30. The computer-program product of claim 25, wherein the algorithm to track the one or more physical characteristics comprises at least one of a cross-correlation algorithm and a structure-from-motion algorithm.

31. The computer-program product of claim 25, wherein the processor is further configured to determine whether an angle of view of a camera used to capture an image of the plurality of images relative to the user is one of a predetermined set of viewing angles.