Method and system for representing a virtual object in a view of a real environment

• US 10,417,824 B2
• Filed: 03/25/2014
• Issued: 09/17/2019
• Est. Priority Date: 03/25/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Knorr, Sebastian, Meier, Peter
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Noh, Jae N
• Application Number:
  US15/128,418
• Publication Number:
  US 20170109931A1
• Time in Patent Office: 2,002 Days
• Field of Search: 348 47
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence

Abstract

The invention relates to a method of representing a virtual object in a view of a real environment which comprises the steps of providing image information of a first image of at least part of a human face captured by a first camera, providing at least one human face specific characteristic, determining at least part of an image area of the face in the first image as a face region of the first image, determining at least one first light falling on the face according to the face region of the first image and the at least one human face specific characteristic, and blending in the virtual object on a display device in the view of the real environment according to the at least one first light. The invention also relates to a system for representing a virtual object in a view of a real environment.

Claims

1. A method of representing a virtual object in a view of a real environment, comprising:

obtaining image information for a first image of at least part of a human face in a real environment captured by a first camera;

determining at least part of an image area of the face in the first image as a face region of the first image;

determining a first light source comprising;

obtaining at least part of a 3D face model,

determining a pose of the human face relative to the first camera when the first camera captures the first image, and

determining the first light source according to intensity information associated with the face region, the pose of the human face, and the at least part of the 3D face model;

generating a virtual object with an illumination according to the determined first light source; and

displaying the virtual object on a display device in the view of the real environment.

2. The method according to claim 1, wherein the first camera is associated with the display device such that the at least part of the human face is captured by the first camera when the human face is facing the display device.

3. The method according to claim 1, wherein determining the first light source is further based on at least one of the following:

a generic 3D face model, a geometry of at least part of a face, a material property of at least part of a face, a color of at least part of a face, a radiance transfer property of at least part of a face, a reflection property of at least part of a face, and a suitability rating of at least part of a face for usage in light estimation.

4. The method according to claim 1, wherein determining the first light source comprises determining at least one characteristic of the following:

type, intensity, wavelength, direction, distance, position, dimension, area, shape, spherical harmonics coefficient, wavelet basis coefficients, and coefficients of mixtures of von Mises-Fisher distributions, particularly at least one of number of modes, mean direction and concentration parameters.

5. The method according to claim 1, wherein:

the view of the real environment is captured as an image by a second camera, wherein the second camera has a known spatial relationship to the first camera, and

displaying the virtual object comprises illuminating at least part of the virtual object according to the determined first light source and the known spatial relationship between the first camera and second camera.

6. The method according to claim 1, wherein the determining the first light source comprises:

obtaining at least part of a 3D face model, wherein the 3D face model is a generic 3D face model or a 3D model of a face,

determining a pose of the face relative to the first camera when the first camera captures the first image, and

determining the first light source according to intensity information associated with the face region and the at least part of the 3D face model.

7. The method according to claim 1, wherein the determining the first light source comprises:

obtaining image information of a plurality of images of human faces in a plurality of different illuminations and poses, wherein the plurality of different illuminations and poses are known,

providing a plurality of face sample positions,

determining, for each of the plurality of images, image positions of at least part of the face sample positions,

determining, for each of the plurality of face sample positions, a radiance transfer function according to intensity information associated with the image positions in at least part of the plurality of images and the illuminations and poses associated with the at least part of the plurality of images,

determining image positions of at least part of the face sample positions in the first image, and

determining the first light source according to intensity information associated with the image positions of at least part of the face sample positions in the first image and the radiance transfer functions associated with the at least part of the face sample positions in the first image.

8. The method according to claim 1, further comprising:

determining at least part of a non-face image area in the first image as a background region,

determining a second light source according to the background region in a way different from determining the first light source, and

updating the illumination of the virtual object using the determined first light source and the determined second light source.

9. A system for representing a virtual object in a view of a real environment, comprising:

one or more cameras;

one or more processors; and

a memory coupled to the one or more processors and comprising instructions which, when executed by the one or more processors, cause the system to;

obtain image information for a first image of at least part of a human face in a real environment captured by a first camera of the one or more cameras,

determine at least part of an image area of the face in the first image as a face region of the first image,

determine a first light source, comprising;

obtaining at least part of a 3D face model,

determining a pose of the human face relative to the first camera when the first camera captures the first image, and

determining the first light source according to intensity information associated with the face region, the pose of the human face, and the at least part of the 3D face model,

generate a virtual object with an illumination according to the determined first light source, and

cause the virtual object to be displayed on a display device in the view of the real environment.

10. The system of claim 9, wherein the first camera is associated with the display device such that the at least part of the human face is captured by the first camera when the human face is facing the display device.

11. The system of claim 9, wherein determining the first light source is further based on at least one of the following:

a generic 3D face model, a geometry of at least part of a face, a material property of at least part of a face, a color of at least part of a face, a radiance transfer property of at least part of a face, a reflection property of at least part of a face, and a suitability rating of at least part of a face for usage in light estimation.

12. The system of claim 9, wherein:

the view of the real environment is captured as an image by a second camera, wherein the second camera has a known spatial relationship to the first camera, and

displaying the virtual object comprises illuminating at least part of the virtual object according to the determined first light source and the known spatial relationship between the first camera and second camera.

13. The system of claim 9, wherein determining the first light source comprises:

obtaining image information of a plurality of images of human faces in a plurality of different illuminations and poses, wherein the plurality of different illuminations and poses are known,

providing a plurality of face sample positions,

determining, for each of the plurality of images, image positions of at least part of the face sample positions,

determining, for each of the plurality of face sample positions, a radiance transfer function according to intensity information associated with the image positions in at least part of the plurality of images and the illuminations and poses associated with the at least part of the plurality of images,

determining image positions of at least part of the face sample positions in the first image, and

determining the first light source according to intensity information associated with the image positions of at least part of the face sample positions in the first image and the radiance transfer functions associated with the at least part of the face sample positions in the first image.

14. The system of claim 9, further comprising instructions to:

determine at least part of a non-face image area in the first image as a background region,

determine a second light source according to the background region in a way different from determining the first light source, and

updating the illumination of the virtual object using the determined first light source and the determined second light source.

15. A non-transitory computer readable medium comprising computer code executable by one or more processors to:

obtain image information for a first image of at least part of a human face in a real environment captured by a first camera;

determine at least part of an image area of the face in the first image as a face region of the first image;

determining a first light source according to intensity information associated with the face region, a pose of the human face, and at least part of a 3D face model; and

generate a virtual object with an illumination according to the determined first light source.

16. The non-transitory computer readable medium of claim 15, wherein the first camera is associated with the display device such that the at least part of the human face is captured by the first camera when the human face is facing the display device.

17. The non-transitory computer readable medium of claim 15, wherein determining the first light source is further based on at least one of the following:

a generic 3D face model, a geometry of at least part of a face, a material property of at least part of a face, a color of at least part of a face, a radiance transfer property of at least part of a face, a reflection property of at least part of a face, and a suitability rating of at least part of a face for usage in light estimation.

18. The non-transitory computer readable medium of claim 15, wherein:

the view of the real environment is captured as an image by a second camera, wherein the second camera has a known spatial relationship to the first camera, and

displaying the virtual object comprises illuminating at least part of the virtual object according to the determined first light source and the known spatial relationship between the first camera and second camera.

19. The non-transitory computer readable medium of claim 15, wherein determining the first light source comprises:

obtaining image information of a plurality of images of human faces in a plurality of different illuminations and poses, wherein the plurality of different illuminations and poses are known,

providing a plurality of face sample positions,

determining, for each of the plurality of images, image positions of at least part of the face sample positions,

determining, for each of the plurality of face sample positions, a radiance transfer function according to intensity information associated with the image positions in at least part of the plurality of images and the illuminations and poses associated with the at least part of the plurality of images,

determining image positions of at least part of the face sample positions in the first image, and

determining the first light source according to intensity information associated with the image positions of at least part of the face sample positions in the first image and the radiance transfer functions associated with the at least part of the face sample positions in the first image.

20. The non-transitory computer readable medium of claim 15, further comprising instructions to:

determine at least part of a non-face image area in the first image as a background region,

determine a second light source according to the background region in a way different from determining the first light source, and

updating the illumination of the virtual object using the determined first light source and the determined second light source.