Head-mounted display with facial expression detecting capability

• US 10,089,522 B2
• Filed: 09/26/2016
• Issued: 10/02/2018
• Est. Priority Date: 09/29/2015
• Status: Active Grant

• Current Assignee: Epic Games, Inc.
• Original Assignee: BinaryVR, Inc.
• Inventors: Yu, Jihun, Park, Jungwoon
• Primary Examiner(s): HON, MING Y
• Application Number:
  US15/276,674
• Publication Number:
  US 20170091535A1
• Time in Patent Office: 736 Days
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Embodiments relate to detecting a user'"'"'s facial expressions in real-time using a head-mounted display unit that includes a 2D camera (e.g., infrared camera) that capture a user'"'"'s eye region, and a depth camera or another 2D camera that captures the user'"'"'s lower facial features including lips, chin and cheek. The images captured by the first and second camera are processed to extract parameters associated with facial expressions. The parameters can be sent or processed so that the user'"'"'s digital representation including the facial expression can be obtained.

Claims

1. A method for detecting facial expressions, the method comprising:

capturing, by a first image capturing device on a main body of a head-mounted display, first images of an upper portion of a user'"'"'s face including the user'"'"'s eye region;

capturing, by a second image capturing device on an extension member of the head-mounted display extending downwards from the main body toward a lower portion of the user'"'"'s face, second images of the user including the lower portion of the user'"'"'s face; and

processing the first images and the second images to extract facial expression parameters representing a facial expression of the user, the processing of the first images and the second images comprising;

performing rigid stabilization to determine a relative pose of the user'"'"'s face relative to the first image capturing device and the second image capturing device.

2. The method of claim 1, wherein the first image capturing device comprises a pair of infrared cameras, and the second image capturing device comprises one of a depth camera, a color camera, an infrared camera or two stereoscopic cameras.

3. The method of claim 1, processing the first images and the second images comprises:

detecting landmark locations associated with the user'"'"'s eye and the user'"'"'s eyebrow from the first images; and

detecting landmark locations associated with a lower portion of the user'"'"'s face from the second images.

4. The method of claim 1, further comprising applying the extracted facial expression parameters to a digital representation of the user to generate a graphical representation of the user.

5. The method of claim 1, further comprising performing calibration by capturing and processing calibration images representing a neutral face of the user.

6. The method of claim 5, wherein performing calibration comprises:

generating a personalized neutral face mesh based on the calibration images, and

building a personalized tracking model by applying a deformation transfer technique to the personalized neutral face mesh, and

wherein the processing of the first images and the second images comprises fitting at least a blendshape model to landmark locations in the first images and the second images based on the personalized tracking model to obtain the facial expression parameters.

7. The method of claim 6, wherein the processing of the first images and the second images is performed in real-time.

8. A virtual or augmented reality system comprising:

a head-mounted display unit, comprising;

a first capturing device configured to capture first images of an upper portion of a user'"'"'s face including an eye region,

a second capturing device at a location below the first capturing device and configured to capture second images of a lower portion of the user'"'"'s face,

a display device configured to display images to the user, a body configured to mount the first capturing device and the display device, and

an extension member extending from the body toward the lower portion of the user'"'"'s face, the second capturing device mounted on the extension member; and

a computing device communicatively coupled to the head-mounted display unit and configured to;

receive the first and second images from the head-mounted display unit,

perform calibration by (i) generating a personalized neutral face mesh based on rigid stabilization, and (ii) building a personalized tracking model by applying a deformation transfer technique to the personalized neutral face mesh, and

process the first images and the second images to extract facial expression parameters representing a facial expression of the user by fitting at least a blendshape model to landmark locations in the first images and the second images based on the personalized tracking model to obtain the facial expression parameters.

9. The virtual or augmented reality system of claim 8, wherein the computing device is configured to:

detect landmark locations associated with the user'"'"'s eye and the user'"'"'s eyebrow from the first images; and

detect landmark locations associated with the lower portion of the user'"'"'s face from the second images.