FACIAL ANIMATION USING FACIAL SENSORS WITHIN A HEAD-MOUNTED DISPLAY

• US 20170352178A1
• Filed: 06/03/2016
• Published: 12/07/2017
• Est. Priority Date: 06/03/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Oculus VR, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Katz, Dov, Toksvig, Michael John, Wang, Ziheng, Omernick, Timothy Paul, Herndon, Torin Ross
• Granted Patent: US 10,430,988 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

A facial tracking system generates a virtual rendering of a portion of a face of a user wearing a head-mounted display (HMD). The facial tracking system illuminates portions of the face inside the HMD. The facial tracking system captures a plurality of facial data of the portion of the face using one or more facial sensors located inside the HMD. A plurality of planar sections of the portion of the face are identified based at least in part on the plurality of facial data. The plurality of planar sections are mapped to one or more landmarks of the face. Facial animation information is generated based at least in part on the mapping, the facial animation information describing a portion of a virtual face corresponding to the portion of the user'"'"'s face.

Claims

1. A method comprising:

illuminating, via a plurality of light sources, portions of a face, inside a head mounted display (HMD), of a user wearing the HMD;

capturing a plurality of facial data of the portion of the face using one or more facial sensors located inside the HMD, wherein the plurality of facial data describes frames of an image including a plurality of pixels, each pixel associated with a different coordinate location of the image;

for each coordinate location of the image;

identifying a planar section corresponding to the brightest pixel value at the coordinate location;

identifying a position of one of the plurality of light sources corresponding to the identified planar section;

generating a virtual surface describing orientation of the portion of the face by aggregating the identified planar sections based at least in part on the identified positions;

mapping the virtual surface to one or more landmarks of the face; and

generating facial animation information based at least in part on the mapping and the virtual surface, the facial animation information describing a portion of a virtual face corresponding to the portion of the face.

2. The method of claim 1, further comprising:

updating a virtual face of an avatar using the facial animation information; and

providing the virtual face to a display element of the HMD for presentation to the user.

3. The method of claim 1, further comprising:

providing instructions to the user to perform one or more facial expressions in a calibration process;

generating the calibration attributes responsive to the user performing at least one of the one or more facial expressions; and

identifying the one or more landmarks of the face of the user based at least in part on the calibration attributes.

4. The method of claim 3, further comprising storing the calibration attributes in an online database including global calibration attributes received from a plurality of HMDs.

5. The method of claim 3, wherein the one or more landmarks of the face describe the location of one or more of the following:

an eye of the user, an eyebrow of the user, a nose of the user, a mouth of the user, and a cheek of the user.

6. The method of claim 1, wherein the facial animation information describes a three-dimensional virtual representation of the portion of the user'"'"'s face.

8. The method of claim 1, wherein the one or more facial sensors are selected from a group consisting of:

cameras, audio sensors, strain gauges, electromagnetic sensors, and proximity sensors.

9. The method of claim 1, wherein illuminating the portions of the face comprises:

providing instructions to the plurality of light sources to illuminate the portions of the face; and

providing instructions to the one or more facial sensors, coordinated with the instructions to the plurality of light sources, to capture the one or more facial data.

10. The method of claim 9, wherein the plurality of light sources are positioned in a ring arrangement, and wherein the instructions provided to the plurality of light sources are coordinated such that only one light source of the plurality of light sources illuminates the portions of the face at any given time.

11. The method of claim 1, further comprising:

receiving specular reflection information from the plurality of facial data; and

identifying a coordinate position of an eye of the user based at least in part on the specular reflection information.

12. The method of claim 11, wherein the facial animation information is further based on the coordinate position of the eye of the user.

21. The method of claim 1, further comprising:

for each coordinate location of the image, determining a normal vector to the corresponding identified planar section, wherein the virtual surface is generated further based on the normal vectors.

7. (canceled)

13. A method comprising:

receiving calibration attributes including one or more landmarks of a face, inside a head mounted display (HMD), of a user wearing the HMD;

capturing a plurality of facial data of a portion of the face using a plurality of light sources and one or more facial sensors located inside the HMD and off a line of sight of the user, wherein the plurality of facial data describes frames of an image including a plurality of pixels, each pixel associated with a different coordinate location of the image;

for each coordinate location of the image;

identifying a planar section corresponding to the brightest pixel value at the coordinate location;

identifying a position of one of the plurality of light sources corresponding to the identified planar section;

generating a virtual surface describing orientation of the portion of the face by aggregating the identified planar sections based at least in part on the identified positions;

mapping the virtual surface to the one or more landmarks of the face;

generating facial animation information based at least in part on the mapping and the virtual surface; and

providing the facial animation information to a display of the HMD for presentation to the user.

14. The method of claim 13, further comprising storing the calibration attributes in an online database including global calibration attributes received from a plurality of HMDs.

15. The method of claim 13, wherein the one or more landmarks of the face describe the location of one or more of the following:

an eye of the user, an eyebrow of the user, a nose of the user, a mouth of the user, and a cheek of the user.

16. The method of claim 13, wherein the facial animation information describes a three-dimensional virtual representation of the portion of the user'"'"'s face.

17. The method of claim 13, wherein the one or more facial sensors are selected from a group consisting of:

cameras, audio sensors, strain gauges, electromagnetic sensors, and proximity sensors.

18. The method of claim 13, further comprising:

receiving specular reflection information from the plurality of facial data; and

identifying a coordinate position of an eye of the user based at least in part on the specular reflection information.

19. (canceled)

20. A method comprising:

illuminating, via a plurality of light sources, portions of a face, inside a head mounted display (HMD), of a user wearing the HMD;

capturing a plurality of facial data of a portion of the face using one or more facial sensors located inside the HMD, wherein the plurality of facial data describes frames of an image including a plurality of pixels, each pixel associated with a different coordinate location of the image;

for each coordinate location of the image;

identifying a planar section corresponding to the brightest pixel value at the coordinate location;

identifying a position of one of the plurality of light sources corresponding to the identified planar section;

generating a virtual surface describing orientation of the portion of the face by aggregating the identified planar sections based at least in part on the identified positions;

mapping the virtual surface to one or more landmarks of the face;

providing the mapping and the virtual surface to a virtual reality (VR) console;

receiving, from the VR console, facial animation information based at least in part on the mapping and the virtual surface; and

providing the facial animation information to a display of the HMD for presentation to the user.