Headset removal in virtual, augmented, and mixed reality using an eye gaze database

• US 10,269,177 B2
• Filed: 06/07/2017
• Issued: 04/23/2019
• Est. Priority Date: 10/06/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Frueh, Christian, Kwatra, Vivek, Sud, Avneesh
• Primary Examiner(s): Tswei, YuJang
• Application Number:
  US15/616,634
• Publication Number:
  US 20180101227A1
• Time in Patent Office: 685 Days
• Field of Search: None
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/19 : Sensors therefor

Abstract

A camera captures an image of a user wearing a head mounted device (HMD) that occludes a portion of the user'"'"'s face. A three-dimensional (3-D) pose that indicates an orientation and a location of the user'"'"'s face in a camera coordinate system is determined. A representation of the occluded portion of the user'"'"'s face is determined based on a 3-D model of the user'"'"'s face. The representation replaces a portion of the HMD in the image based on the 3-D pose of the user'"'"'s face in the camera coordinate system. In some cases, the 3-D model of the user'"'"'s face is selected from 3-D models of the user'"'"'s face stored in a database that is indexed by eye gaze direction. Mixed reality images can be generated by combining virtual reality images, unoccluded portions of the user'"'"'s face, and representations of an occluded portion of the user'"'"'s face.

Claims

1. A method, comprising:

capturing, using a camera, an image of a user that is wearing a head mounted device (HMD) that occludes a portion of the user'"'"'s face;

determining a three-dimensional (3-D) pose that indicates an orientation and a location of the user'"'"'s face in a camera coordinate system associated with the camera;

detecting an eye gaze direction of the user;

accessing texture samples from an eye gaze database based on the eye gaze direction; and

rendering a representation of the occluded portion of the user'"'"'s face based on a 3-D model of the user'"'"'s face and using the texture samples accessed from the eye gaze database, wherein the representation replaces a portion of the HMD in the image based on the 3-D pose of the user'"'"'s face in the camera coordinate system.

2. The method of claim 1, further comprising:

determining an initial 3-D pose of the user'"'"'s face in the camera coordinate system by matching a model of an unoccluded portion of the user'"'"'s face to an initial image.

3. The method of claim 2, wherein determining the initial 3-D pose of the user'"'"'s face in the camera coordinate system comprises determining the initial 3-D pose of the user'"'"'s face in the camera coordinate system in non-real-time using a background thread or pre-process.

4. The method of claim 3, further comprising:

determining an initial pose of the HMD by matching at least one feature of the HMD to the initial image; and

determining a first transform of an HMD coordinate system to a camera coordinate system based on an inverse of the initial 3-D pose of the user'"'"'s face in the camera coordinate system and the initial pose of the HMD.

5. The method of claim 4, further comprising:

determining a rotation matrix that represents a relative orientation of the user'"'"'s face and the HMD based on an inverse of the initial pose of the HMD and the initial 3-D pose of the user'"'"'s face relative to the camera.

6. The method of claim 5, wherein determining the 3-D pose of the user'"'"'s face in the camera coordinate system comprises determining the 3-D pose of the user'"'"'s face in the camera coordinate system based on the first transform, a pose of the HMD in the HMD coordinate system, and the rotation matrix.

7. The method of claim 1, wherein determining the 3-D pose of the user'"'"'s face in the camera coordinate system comprises determining the 3-D pose of the user'"'"'s face in the camera coordinate system in real time by matching a model of an unoccluded portion of the user'"'"'s face to the image.

8. The method of claim 7, wherein matching a model of the unoccluded portion of the user'"'"'s face to the image comprises matching a subset of points in the model to a subset of points in the image.

9. The method of claim 1, further comprising:

color correcting the texture samples using an affine transformation that maps colors of the texture samples to colors of an unoccluded portion of the user'"'"'s face in the image.

10. The method of claim 1, wherein rendering the 3-D model of the occluded portion of the user'"'"'s face comprises rendering the 3-D model of the occluded portion of the user'"'"'s face using a translucence that varies from a center to an edge of the occluded portion of the user'"'"'s face.

11. An apparatus, comprising:

a camera configured to capture an image of a user that is wearing a head mounted device (HMD) that occludes a portion of the user'"'"'s face;

an eye gaze tracker configured to detect an eye gaze direction of the user; and

a processor configured to determine a three-dimensional (3-D) pose that indicates an orientation and a location of the user'"'"'s face relative to the camera, access texture samples from an eye gaze database stored in a memory, and render a representation of the occluded portion of the user'"'"'s face based on a 3-D model of the user'"'"'s face and using the accessed texture samples, wherein the representation replaces a portion of the HMD in the image based on the 3-D pose.

12. The apparatus of claim 11, wherein the processor is configured to determine an initial 3-D pose of the user'"'"'s face in a camera coordinate system by matching a model of an unoccluded portion of the user'"'"'s face to an initial image.

13. The apparatus of claim 12, wherein the processor is configured to determine the initial 3-D pose of the user'"'"'s face in the camera coordinate system in non-real-time using a background thread or pre-process.

14. The apparatus of claim 13, wherein the processor is configured to:

determine an initial pose of the HMD by matching at least one feature of the HMD to the initial image; and

determine a first transform of an HMD coordinate system to a camera coordinate system based on an inverse of the initial 3-D pose of the user'"'"'s face in the camera coordinate system and the initial pose of the HMD.

15. The apparatus of claim 14, wherein the processor is configured to:

determine a rotation matrix that represents a relative orientation of the user'"'"'s face and the HMD based on an inverse of the initial pose of the HMD and the initial 3-D pose of the user'"'"'s face relative to the camera.

16. The apparatus of claim 15, wherein the processor is configured to determine the 3-D pose of the user'"'"'s face in the camera coordinate system based on the first transform, a pose of the HMD in the HMD coordinate system, and the rotation matrix.

17. The apparatus of claim 11, wherein the processor is configured to determine the 3-D pose of the user'"'"'s face in a camera coordinate system in real time by matching a model of an unoccluded portion of the user'"'"'s face to the image.

18. The apparatus of claim 17, wherein the processor is configured to match a subset of points in the model to a subset of points in the image.

19. The apparatus of claim 13, wherein the processor is configured to color correct the texture samples using an affine transformation that maps colors of the texture samples to colors of an unoccluded portion of the user'"'"'s face in the image.

20. The apparatus of claim 11, wherein the processor is configured to render the 3-D model of the occluded portion of the user'"'"'s face using a translucence that varies from a center to an edge of the occluded portion of the user'"'"'s face.

21. A method, comprising:

detecting an first portion of a user'"'"'s face in an image captured by a camera, wherein the user is wearing a head mounted device (HMD) that occludes a second portion of the user'"'"'s face in the image; and

rendering a 3-D model of the second portion of the user'"'"'s face into the image to replace a portion of the HMD in the image, wherein rendering the 3-D model of the second portion includes rendering the 3-D model of the second portion using texture samples accessed from an eye gaze database based on an eye gaze direction of the user and further includes color correcting the texture samples using an affine transformation that maps colors of the texture samples to colors of the first portion of the user'"'"'s face.

22. The method of claim 21, wherein detecting the first portion of the user'"'"'s face in the image comprises matching a model of the first portion of the user'"'"'s face to the image.

23. The method of claim 22, wherein rendering the 3-D model of the second portion of the user'"'"'s face comprises determining a 3-D pose of the user'"'"'s face based on a first transform of an HMD coordinate system to a camera coordinate system, a pose of the HMD in an HMD coordinate system, and a rotation matrix that represents a position or orientation of the user'"'"'s face relative to the HMD.

24. The method of claim 21, wherein rendering the 3-D model of the second portion of the user'"'"'s face comprises rendering the 3-D model of the second portion of the user'"'"'s face using a translucence that varies from a center to an edge of the second portion of the user'"'"'s face.