Methods and systems to modify a two dimensional facial image to increase dimensional depth and generate a facial image that appears three dimensional

• US 10,055,880 B2
• Filed: 12/06/2016
• Issued: 08/21/2018
• Est. Priority Date: 12/06/2016
• Status: Active Grant

• Current Assignee: Activision Publishing Incorporated (Microsoft Corporation)
• Original Assignee: Activision Publishing Incorporated (Microsoft Corporation)
• Inventors: Yan, Paul D, Vorachart, Thitikun
• Primary Examiner(s): CRADDOCK, ROBERT J
• Application Number:
  US15/370,166
• Publication Number:
  US 20180158230A1
• Time in Patent Office: 623 Days
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

The specification describes methods and systems for increasing a dimensional depth of a two-dimensional image of a face to yield a face image that appears three dimensional. The methods and systems identify key points on the 2-D image, obtain a texture map for the 2-D image, determines one or more proportions within the 2-D image, and adjusts the texture map of the 3-D model based on the determined one or more proportions within the 2-D image.

Claims

1. A computer-implemented method for increasing a dimensional depth of a two-dimensional image of a face to yield a face image that appears three dimensional, said method being implemented in a computer having a processor and a random access memory, wherein said processor is in data communication with a display and with a storage unit, the method comprising:

acquiring from the storage unit the two-dimensional image of the face;

acquiring from the storage unit a three-dimensional mesh image;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, identifying a plurality of key points on the two-dimensional image of the face;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, generating a texture map of the two-dimensional image of the face;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, projecting said texture map of the two-dimensional image of the face onto the three-dimensional mesh image;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, determining a first set of one or more proportions within the two-dimensional image of the face;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, determining a second set of one or more proportions within the three-dimensional mesh image;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, determining a plurality of scaling factors, wherein each of said scaling factors is a function of one of said first set of one or more proportions and a corresponding one of said second set of one or more proportions;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, adjusting the three-dimensional mesh image based on the determined plurality of scaling factors to yield the face image that appears three dimensional; and

using said computer, outputting the face image that appears three dimensional.

2. The computer-implemented method of claim 1, wherein the key points include points representative of a plurality of anatomical locations on the face, wherein said anatomical locations include points located on the eyebrows, eyes, nose, and lips.

3. The computer-implemented method of claim 1, wherein the texture map comprises a plurality of non-overlapping, triangular regions.

4. The computer-implemented method of claim 1, wherein each of said plurality of scaling factors is a ratio of one of said first set of one or more proportions to the corresponding one of said second set of one or more proportions.

5. The computer-implemented method of claim 1, wherein the determining the first set of one or more proportions within the two-dimensional image comprises determining proportions from measurements between at least two anatomical positions on the face.

6. The computer-implemented method of claim 1, wherein the determining a first set of one or more proportions within the two-dimensional image comprises determining a first anatomical distance and dividing said first anatomical distance by a second anatomical distance.

7. The computer-implemented method of claim 6, wherein the first anatomical distance is at least one of a lateral face width, a lateral jaw width, a lateral temple width, a lateral eyebrow width, a lateral chin width, a lateral lip width, and a lateral nose width and wherein the second anatomical distance is a distance between two temples of the face.

8. The computer-implemented method of claim 6, wherein the first anatomical distance is at least one of a vertically defined lip thickness, a vertical distance between a nose and a nose bridge, a vertical distance between a lip and a nose bridge, a vertical distance between a chin and a nose bridge, a vertical eye length, and a vertical distance between a jaw and a nose bridge and wherein the second anatomical distance is at least one of a distance between two anatomical positions on said face and a distance between two temples of the face.

9. The computer-implemented method of claim 6, wherein the first anatomical distance is a distance between two anatomical positions on said face and the second anatomical distance is a distance between a point located proximate a left edge of a left eyebrow of the face and a point located proximate a right edge of a right eyebrow of the face.

10. The computer-implemented method of claim 1, wherein the determining a second set of one or more proportions within the three-dimensional mesh image comprises determining a first anatomical distance and dividing said first anatomical distance by a second anatomical distance.

11. The computer-implemented method of claim 10, wherein the first anatomical distance is at least one of a lip thickness, a distance between a nose and a nose bridge, a distance between a lip and a nose bridge, a distance between a chin and a nose bridge, an eye length, and a distance between a jaw and a nose bridge of the three-dimensional mesh image and wherein the second anatomical distance is a distance between two anatomical positions on said three-dimensional mesh image.

12. The computer-implemented method of claim 10, wherein the first anatomical distance is a distance between two anatomical positions on said three-dimensional mesh image and the second anatomical distance is a distance between a point located proximate a left edge of a left eyebrow of the three-dimensional mesh image and a point located proximate a right edge of a right eyebrow of the three-dimensional mesh image.

13. The computer-implemented method of claim 1, wherein, prior to identifying the plurality of key points on the two-dimensional image of the face, processing the two-dimensional image to validate a presence of a frontal image of the face.

14. A computer readable non-transitory medium comprising a plurality of executable programmatic instructions wherein, when said plurality of executable programmatic instructions are executed by a processor, a process for increasing a dimensional depth of a two-dimensional image of a face to yield a face image that appears three dimensional is performed, said plurality of executable programmatic instructions comprising:

programmatic instructions, stored in said computer readable non-transitory medium, for acquiring from the storage unit the two-dimensional image of the face;

programmatic instructions, stored in said computer readable non-transitory medium, for acquiring from the storage unit a three-dimensional mesh image;

programmatic instructions, stored in said computer readable non-transitory medium, for identifying a plurality of key points on the two-dimensional image of the face;

programmatic instructions, stored in said computer readable non-transitory medium, for generating a texture map of the two-dimensional image of the face;

programmatic instructions, stored in said computer readable non-transitory medium, for translating said texture map of the two-dimensional image of the face onto the three-dimensional mesh image;

programmatic instructions, stored in said computer readable non-transitory medium, for determining a first set of one or more proportions within the two-dimensional image of the face;

programmatic instructions, stored in said computer readable non-transitory medium, for determining a second set of one or more proportions within the three-dimensional mesh image;

programmatic instructions, stored in said computer readable non-transitory medium, for determining a plurality of scaling factors, wherein each of said scaling factors is a function of one of said first set of one or more proportions and a corresponding one of said second set of one or more proportions; and

programmatic instructions, stored in said computer readable non-transitory medium, for adjusting the three-dimensional mesh image based on the determined plurality of scaling factors to yield the face image that appears three dimensional.

15. The computer readable non-transitory medium of claim 14, wherein the key points include points representative of a plurality of anatomical locations on the face, wherein said anatomical locations include points located on the eyebrows, eyes, nose, and lips.

16. The computer readable non-transitory medium of claim 14, wherein the texture map comprises a plurality of non-overlapping, triangular regions.

17. The computer readable non-transitory medium of claim 14, wherein the determining one or more proportions within the two-dimensional image comprises determining proportions from measurements between at least two anatomical positions on the face.

18. The computer readable non-transitory medium of claim 14, wherein each of said plurality of scaling factors is a ratio of one of said first set of one or more proportions to the corresponding one of said second set of one or more proportions.

19. The computer readable non-transitory medium of claim 14, wherein the determining a first set of one or more proportions within the two-dimensional image comprises determining a first anatomical distance and dividing said first anatomical distance by a second anatomical distance.

20. The computer readable non-transitory medium of claim 19, wherein the first anatomical distance is at least one of a lateral face width, a lateral jaw width, a lateral temple width, a lateral eyebrow width, a lateral chin width, a lateral lip width, and a lateral nose width and wherein the second anatomical distance is a distance between two temples of the face.

21. The computer readable non-transitory medium of claim 19, wherein the first anatomical distance is at least one of a vertically defined lip thickness, a vertical distance between a nose and a nose bridge, a vertical distance between a lip and a nose bridge, a vertical distance between a chin and a nose bridge, a vertical eye length, and a vertical distance between a jaw and a nose bridge and wherein the second anatomical distance is a distance between two temples of the face.

22. The computer readable non-transitory medium of claim 19, wherein the first anatomical distance is a distance between two anatomical positions on said face and the second anatomical distance is a distance between a point located proximate a left edge of a left eyebrow of the face and a point located proximate a right edge of a right eyebrow of the face.

23. The computer readable non-transitory medium of claim 14, wherein the determining a second set of one or more proportions within the three-dimensional mesh image comprises determining a first anatomical distance and dividing said first anatomical distance by a second anatomical distance.

24. The computer readable non-transitory medium of claim 23, wherein the first anatomical distance is at least one of a lip thickness, a distance between a nose and a nose bridge, a distance between a lip and a nose bridge, a distance between a chin and a nose bridge, an eye length and a distance between a jaw and a nose bridge of the three-dimensional mesh image and wherein the second anatomical distance is a distance between two anatomical positions on said three-dimensional mesh image.

25. The computer readable non-transitory medium of claim 23, wherein the first anatomical distance is a distance between two anatomical positions on said three-dimensional mesh image and the second anatomical distance is a distance between a point located proximate a left edge of a left eyebrow of the three-dimensional mesh image and a point located proximate a right edge of a right eyebrow of the three-dimensional mesh image.

26. The computer readable non-transitory medium of claim 14, further comprising programmatic instructions, stored in said computer readable non-transitory medium, for processing the two-dimensional image to validate a presence of a frontal image of the face prior to identifying the plurality of key points on the two-dimensional image of the face.