Facial animation using facial sensors within a head-mounted display

• US 10,430,988 B2
• Filed: 06/03/2016
• Issued: 10/01/2019
• Est. Priority Date: 06/03/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Katz, Dov, Toksvig, Michael John, Wang, Ziheng, Omernick, Timothy Paul, Herndon, Torin Ross
• Primary Examiner(s): Patel, Shivang I
• Application Number:
  US15/172,484
• Publication Number:
  US 20170352178A1
• Time in Patent Office: 1,215 Days
• Field of Search: 345473
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

A facial tracking system generates a virtual rendering of a portion of a face of a user wearing a head-mounted display (HMD). The facial tracking system illuminates portions of the face inside the HMD. The facial tracking system captures a plurality of facial data of the portion of the face using one or more facial sensors located inside the HMD. A plurality of planar sections of the portion of the face are identified based at least in part on the plurality of facial data. The plurality of planar sections are mapped to one or more landmarks of the face. Facial animation information is generated based at least in part on the mapping, the facial animation information describing a portion of a virtual face corresponding to the portion of the user'"'"'s face.

Claims

1. A method comprising:

causing, inside a head mounted display (HMD), a plurality of light sources to emit light by a single light source at a time in a particular sequence to illuminate a portion of a face of a user wearing the HMD, wherein the portion of the face includes the eyes of the user and portions of an eyebrow and a cheek of the user;

capturing a plurality of facial data of the portion of the face using one or more facial sensors located inside the HMD, wherein the plurality of facial data describes a plurality of frames of an image including a plurality of pixels, each pixel associated with a different coordinate location of the image, each of the plurality of frames captured when a different single light source of the plurality of light sources illuminated the portion of the face;

for each coordinate location of the image;

identifying a frame of the plurality of frames having the greatest intensity value based on reflected light from the portion of the face at the coordinate location;

identifying a position of a light source of the plurality of light sources illuminating the portion of the face when the frame was captured, the reflected light originating from the light source;

determining a planar section of the portion of the face based on the position of the light source and the coordinate location; and

determining a normal vector to the planar section based on the position of the light source illuminating the portion of the face when the frame of the plurality of frames having the greatest intensity value was captured;

generating a virtual surface describing orientation of the portion of the face by aggregating the normal vectors for the planar sections;

mapping the virtual surface to one or more landmarks of the face; and

generating facial animation information based at least in part on the mapping and the virtual surface, the facial animation information describing a portion of a virtual face corresponding to the portion of the face.

2. The method of claim 1, further comprising:

updating a virtual face of an avatar using the facial animation information; and

providing the virtual face to a display element of the HMD for presentation to the user.

3. The method of claim 1, further comprising:

providing instructions to the user to perform one or more facial expressions in a calibration process;

generating calibration attributes responsive to the user performing at least one of the one or more facial expressions; and

identifying the one or more landmarks of the face of the user based at least in part on the calibration attributes.

4. The method of claim 3, further comprising storing the calibration attributes in an online database including global calibration attributes received from a plurality of HMDs.

5. The method of claim 3, wherein the one or more landmarks of the face describe the location of one or more of the following:

an eye of the user, an eyebrow of the user, a nose of the user, a mouth of the user, and a cheek of the user.

6. The method of claim 1, wherein the facial animation information describes a three-dimensional virtual representation of the portion of the face.

7. The method of claim 1, wherein the one or more facial sensors are selected from a group consisting of:

cameras, audio sensors, strain gauges, electromagnetic sensors, and proximity sensors.

8. The method of claim 1, wherein illuminating the portion of the face comprises:

providing instructions to the plurality of light sources to illuminate the portion of the face; and

providing instructions to the one or more facial sensors, coordinated with the instructions to the plurality of light sources, to capture the plurality of facial data.

9. The method of claim 8, wherein the plurality of light sources are positioned in a ring arrangement, and wherein the instructions provided to the plurality of light sources are coordinated such that only one light source of the plurality of light sources illuminates the portion of the face at any given time.

10. The method of claim 1, further comprising:

receiving specular reflection information from the plurality of facial data; and

identifying a coordinate position of an eye of the user based at least in part on the specular reflection information.

11. The method of claim 10, wherein the facial animation information is generated further based on the coordinate position of the eye of the user.

12. The method of claim 1, wherein the virtual surface is generated further based on the normal vectors.

13. A method comprising:

receiving calibration attributes including one or more landmarks of a face, inside a head mounted display (HMD), of a user wearing the HMD;

capturing a plurality of facial data of a portion of the face using a plurality of light sources and one or more facial sensors located inside the HMD and off a line of sight of the user, wherein the portion of the face includes the eyes of the user and portions of an eyebrow and a cheek of the user, and the plurality of facial data describes a plurality of frames of an image including a plurality of pixels, each pixel associated with a different coordinate location of the image, each of the plurality of frames captured when a different single light source of the plurality of light sources illuminated the portion of the face, the plurality of light sources emitting light by a single light source at a time in a particular sequence to illuminate the portion of the face;

for each coordinate location of the image;

identifying a frame of the plurality of frames having the greatest intensity value based on reflected light from the portion of the face at the coordinate location;

identifying a position of a light source of the plurality of light sources illuminating the portion of the face when the frame was captured, the reflected light originating from the light source;

determining a planar section of the portion of the face based on the position of the light source and the coordinate location; and

determining a normal vector to the planar section based on the position of the light source illuminating the portion of the face when the frame of the plurality of frames having the greatest intensity value was captured;

generating a virtual surface describing orientation of the portion of the face by aggregating the normal vectors for the planar sections;

mapping the virtual surface to the one or more landmarks of the face;

generating facial animation information based at least in part on the mapping and the virtual surface; and

providing the facial animation information to a display of the HMD for presentation to the user.

14. The method of claim 13, further comprising storing the calibration attributes in an online database including global calibration attributes received from a plurality of HMDs.

15. The method of claim 13, wherein the one or more landmarks of the face describe the location of one or more of the following:

an eye of the user, an eyebrow of the user, a nose of the user, a mouth of the user, and a cheek of the user.

16. The method of claim 13, wherein the facial animation information describes a three-dimensional virtual representation of the portion of the user'"'"'s face.

17. The method of claim 13, wherein the one or more facial sensors are selected from a group consisting of:

cameras, audio sensors, strain gauges, electromagnetic sensors, and proximity sensors.

18. The method of claim 13, further comprising:

receiving specular reflection information from the plurality of facial data; and

identifying a coordinate position of an eye of the user based at least in part on the specular reflection information.

19. A method comprising:

causing, inside a head mounted display (HMD), a plurality of light sources to emit light by a single light source at a time in a particular sequence to illuminate a portion of a face of a user wearing the HMD, wherein the portion of the face includes the eyes of the user and portions of an eyebrow and a cheek of the user;

capturing a plurality of facial data of the portion of the face using one or more facial sensors located inside the HMD, wherein the plurality of facial data describes a plurality of frames of an image including a plurality of pixels, each pixel associated with a different coordinate location of the image, each of the plurality of frames captured when a different single light source of the plurality of light sources illuminated the portion of the face;

for each coordinate location of the image;

identifying a frame of the plurality of frames having the greatest intensity value based on reflected light from the portion of the face at the coordinate location;

identifying a position of a light source of the plurality of light sources illuminating the portion of the face when the frame was captured, the reflected light originating from the light source;

determining a planar section of the portion of the face based on the position of the light source and the coordinate location; and

determining a normal vector to the planar section based on the position of the light source illuminating the portion of the face when the frame of the plurality of frames having the greatest intensity value was captured;

generating a virtual surface describing orientation of the portion of the face by aggregating the normal vectors for the planar sections;

mapping the virtual surface to one or more landmarks of the face;

providing the mapping and the virtual surface to a virtual reality (VR) console;

receiving, from the VR console, facial animation information based at least in part on the mapping and the virtual surface; and

providing the facial animation information to a display of the HMD for presentation to the user.

20. The method of claim 19, further comprising:

updating a virtual face of an avatar using the facial animation information; and

providing the virtual face to the display element of the HMD for presentation to the user.