Face and eye tracking using facial sensors within a head-mounted display

• US 9,959,678 B2
• Filed: 06/03/2016
• Issued: 05/01/2018
• Est. Priority Date: 06/03/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Oculus VR, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Katz, Dov, Toksvig, Michael John, Wang, Ziheng, Omernick, Timothy Paul, Herndon, Torin Ross
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US15/172,473
• Publication Number:
  US 20170352183A1
• Time in Patent Office: 697 Days
• Field of Search: 345633, 345 8, 345473, 345474, 382103
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/56 provided with illuminating means

Abstract

A head mounted display (HMD) in a VR system includes sensors for tracking the eyes and face of a user wearing the HMD. The VR system records calibration attributes such as landmarks of the face of the user. Light sources illuminate portions of the user'"'"'s face covered by the HMD. In conjunction, facial sensors capture facial data. The VR system analyzes the facial data to determine the orientation of planar sections of the illuminated portions of face. The VR system aggregates planar sections of the face and maps the planar sections to landmarks of the face to generate a facial animation of the user, which can also include eye orientation information. The facial animation is represented as a virtual avatar and presented to the user.

Claims

1. A head mounted display (HMD) comprising:

a display element configured to display content to a user wearing the HMD;

an optics block configured to direct light from the display element to an exit pupil of the HMD;

a plurality of light sources positioned at discrete locations around the optics block, the plurality of light sources configured to illuminate portions of a face, inside the HMD, of the user;

a facial sensor configured to capture one or more facial data of a portion of the face illuminated by one or more of the plurality of light sources; and

a controller configured to;

receive a plurality of captured facial data from the facial sensor;

identify a plurality of surfaces of the portion of the face based at least in part on the plurality of captured facial data;

retrieve a plurality of landmarks each indicating a position of a different feature of the face, the plurality of landmarks determined based on calibration attributes corresponding to a plurality of different facial expressions performed by the user for a calibration process;

retrieve global calibration attributes, the global calibration attributes generated based on facial expressions performed by a population of users;

responsive to determining that a difference between the calibration attributes and the global calibration attributes is less than a threshold value;

map the plurality of surfaces to one of the plurality of landmarks; and

generate facial animation information describing the portion of the face of the user based at least in part on the plurality of captured facial data and the mapping.

2. The HMD of claim 1, wherein the controller is further configured to:

provide instructions to the user to perform the plurality of different facial expressions for the calibration process;

generate the calibration attributes responsive to the user performing at least one of the plurality of different facial expressions; and

identify the plurality of landmarks based at least in part on the calibration attributes.

3. The HMD of claim 1, wherein the display element is configured to display an avatar to the user, and the avatar'"'"'s face is based on the facial animation information.

4. The HMD of claim 1, wherein the controller is further configured to:

generate intensity curves for coordinates of frames of the plurality of captured facial data using pixel values of the plurality of captured facial data; and

wherein identifying the plurality of surfaces of the portion of the face comprises determining normal vectors using the intensity curves.

5. The HMD of claim 1, wherein the plurality of captured facial data describe frames of an image, the image comprising a plurality of pixels, each pixel associated with a coordinate (x, y) location of the image, and wherein the controller is further configured to:

identify, for each coordinate location of the image, a planar section corresponding to the brightest pixel value at the coordinate location;

identify a light source position corresponding to each of the identified planar sections;

generate a virtual surface of the portion of the face based at least in part on the identified planar sections and the identified light source positions; and

wherein generating the facial animation information is further based on the virtual surface.

6. The HMD of claim 1, wherein the controller is further configured to:

provide instructions to the plurality of light sources to illuminate the portions of the face; and

provide instructions to the facial sensor, coordinated with the instructions to the plurality of light sources, to capture the one or more facial data.

7. The HMD of claim 6, wherein the plurality of light sources are positioned in a ring arrangement around the optics block, and wherein the instructions provided to the plurality of light sources are coordinated such that only one light source of the plurality of light sources illuminates the portions of the face at any given time.

8. The HMD of claim 1, wherein the controller is further configured to:

receive specular reflection information from the plurality of facial data; and

identify a position of an eye of the user based at least in part on the specular reflection information.

9. The HMD of claim 1, wherein the global calibration attributes are retrieved from an online calibration server, and wherein the controller is further configured to upload the calibration attributes to the online calibration server.

10. A head mounted display (HMD) comprising:

a display element configured to display content to a user wearing the HMD;

a plurality of light sources positioned at discrete locations outside a line of sight of the user, the plurality of light sources configured to illuminate portions of a face, inside the HMD, of the user;

a facial sensor configured to capture one or more facial data of a portion of the face illuminated by one or more of the plurality of light sources; and

a controller configured to;

receive a plurality of captured facial data from the facial sensor;

identify a plurality of surfaces of the portion of the face based at least in part on the plurality of captured facial data;

retrieve a plurality of landmarks each indicating a position of a different feature of the face, the plurality of landmarks determined based on calibration attributes corresponding to a plurality of different facial expressions performed by the user for a calibration process;

retrieve global calibration attributes, the global calibration attributes generated based on facial expressions performed by a population of users;

responsive to determining that a difference between the calibration attributes and the global calibration attributes is less than a threshold value;

map the plurality of surfaces to one of the plurality of landmarks; and

generate facial animation information describing the portion of the face of the user based at least in part on the plurality of captured facial data and the mapping.

11. The HMD of claim 10, wherein the controller is further configured to:

provide instructions to the user to perform the plurality of different facial expressions for the calibration process;

generate the calibration attributes responsive to the user performing at least one of the plurality of different facial expressions; and

identify the plurality of landmarks based at least in part on the calibration attributes.

12. The HMD of claim 10, wherein the display element is configured to display an avatar to the user, and the avatar'"'"'s face is based on the facial animation information.

13. The HMD of claim 10, wherein the controller is further configured to:

generate intensity curves for coordinates of frames of the plurality of captured facial data using pixel values of the plurality of captured facial data; and

wherein identifying the plurality of surfaces of the portion of the face comprises determining normal vectors using the intensity curves.

14. The HMD of claim 10, wherein the plurality of captured facial data describe frames of an image, the image comprising a plurality of pixels, each pixel associated with a coordinate (x, y) location of the image, and wherein the controller is further configured to:

identify, for each coordinate location of the image, a planar section corresponding to the brightest pixel value at the coordinate location;

identify a light source position corresponding to each of the identified planar sections;

generate a virtual surface of the portion of the face based at least in part on the identified planar sections and the identified light source positions; and

wherein generating the facial animation information is further based on the virtual surface.

15. The HMD of claim 10, wherein the controller is further configured to:

provide instructions to the plurality of light sources to illuminate the portions of the face; and

provide instructions to the facial sensor, coordinated with the instructions to the plurality of light sources, to capture the one or more facial data.

16. The HMD of claim 15, wherein the plurality of light sources are positioned in a ring arrangement around the optics block, and wherein the instructions provided to the plurality of light sources are coordinated such that only one light source of the plurality of light sources illuminates the portion of the face at any given time.

17. The HMD of claim 10, wherein the controller is further configured to:

receive specular reflection information from the plurality of facial data; and

identify a position of an eye of the user based at least in part on the specular reflection information.

18. A head mounted display (HMD) comprising:

a display element configured to display content to a user wearing the HMD;

an optics block configured to direct light from the display element to an exit pupil of the HMD;

a plurality of light sources positioned at discrete locations around the optics block, the plurality of light sources configured to illuminate portions of a face, inside the HMD, of the user;

a facial sensor configured to capture one or more facial data of a portion of the face illuminated by one or more of the plurality of light sources; and

a controller configured to;

receive a plurality of captured facial data from the facial sensor;

identify a plurality of surfaces of the portion of the face based at least in part on the plurality of captured facial data;

retrieve global calibration attributes, the global calibration attributes generated based on facial expressions performed by a population of users;

responsive to determining that a difference between calibration attributes of the user and the global calibration attributes is less than a threshold value;

provide the plurality of surfaces to a virtual reality (VR) console; and

receive, from the VR console a virtual animation of the portion of the face of the user, the virtual animation generated by mapping the plurality of surfaces to one of a plurality of landmarks each indicating a position of a different feature of the face, the plurality of landmarks determined based on the calibration attributes of the user corresponding to a plurality of different facial expressions performed by the user for a calibration process.

19. The HMD of claim 18, wherein the controller is further configured to:

provide instructions to the user to perform the plurality of different facial expressions for the calibration process;

generate the calibration attributes responsive to the user performing at least one of the plurality of different facial expressions;

identify the plurality of landmarks based at least in part on the calibration attributes; and

provide the plurality of landmarks to the VR console.

20. The HMD of claim 18, wherein the facial sensor is selected from a group consisting of:

a camera, audio sensor, strain gauge, electromagnetic sensor, and proximity sensor.