Method and system for representing a virtual object in a view of a real environment

• US 10,109,107 B2
• Filed: 03/25/2015
• Issued: 10/23/2018
• Est. Priority Date: 03/25/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Knorr, Sebastian, Meier, Peter
• Primary Examiner(s): Johnson, M Good
• Application Number:
  US14/668,381
• Publication Number:
  US 20150279113A1
• Time in Patent Office: 1,308 Days
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence

Abstract

A method and system of representing a virtual object in a view of a real environment is provided which includes providing image information of a first image of at least part of a human face captured by a camera, providing at least one human face specific characteristic, determining an image area of the face in the first image as a face region, determining at least one first light falling on the face according to the face region and the at least one human face specific characteristic, and blending in the virtual object on a display device in the view of the real environment according to at least one first light. The method further includes providing a plurality of face sample positions, providing for each of the plurality of face sample positions a radiance transfer function, determining image positions of the face sample positions in the first image, and determining a first light according to intensity information associated with the image positions of the face sample positions in the first image and the radiance transfer functions associated with the face sample positions in the first image.

Claims

1. A method of representing a virtual object in a view of a real environment, comprising:

obtaining a first image of at least part of a human face in a real environment captured by a first camera;

determining one or more image positions in the first image corresponding to one or more face sample positions, wherein the one or more face sample positions are each associated with a known radiance transfer function;

determining at least one first light according to intensity information associated with the one or more image positions and the radiance transfer functions associated with the one or more face sample positions corresponding to the image positions in the first image; and

displaying the virtual object on a display device in the view of the real environment according to the at least one first light.

2. The method according to claim 1, wherein the first camera is associated with the display device such that the at least part of the human face is captured by the first camera when the human face is facing the display device, and the display device has a normal and the first camera has an optical axis, and the normal of the display device and the optical axis of the first camera are oriented in substantially the same direction.

3. The method according to claim 1, wherein determining the at least one first light comprises determining at least one characteristic of at least one light source emitting or reflecting or refracting at least part of the at least one first light, wherein the at least one characteristic of the at least one light source comprises spherical harmonics coefficient.

4. The method according to claim 1, wherein:

the view of the real environment is captured as an image by the first camera; and

displaying the virtual object comprises illuminating at least part of the virtual object according to the determined at least one first light.

5. The method according to claim 1, wherein the determining the at least one first light comprises obtaining depth information associated with a region of the first image comprising the human face, and determining the at least one first light according to intensity and the depth information associated with the region of the first image comprising the human face.

6. The method according to claim 1, further comprising:

determining at least part of a non-face image area in the first image as a background region;

determining at least one second light according to the background region in a way different from determining the at least one first light; and

updating the determined at least one first light by combining the determined at least one first light and the determined at least one second light.

7. The method according to claim 1, further comprising:

providing image information of a second image of at least part of the real environment captured by a second camera, wherein the second camera has a known spatial relationship to the first camera; and

determining at least one third light according to the second image, and

updating the determined at least one first light by combining the determined at least one first light and the determined at least one third light according to the known spatial relationship.

8. The method according to claim 1, further comprising:

determining a first pose of the face relative to the first camera according to the first image;

providing image information of a third image of at least part of the face captured by the first camera;

determining at least one fourth light according to the third image;

determining a third pose of the face relative to the first camera when the first camera captures the third image; and

updating the determined at least one first light by combining the determined at least one first light and the determined at least one fourth light according to the first pose and the third pose.

9. The method according to claim 8, wherein the combining the determined at least one first light and the determined at least one fourth light comprises determining a direction of the at least one first light and a direction of the at least one fourth light, and determining an intersection between lines along the two determined directions.

10. The method according to claim 8, wherein the combining the determined at least one first light and the determined at least one fourth light comprises computing a value of a characteristic of the updated at least one first light according to a value of the characteristic of the at least one first light and a value of the characteristic of the at least one fourth light based on a mathematic method, particularly at least one of averaging, least squares minimization, ransac, and probability maximization.

11. The method according to claim 1, wherein the method further comprises:

providing image information of a third image of at least part of the face captured by the first camera;

determining at least one fifth light according to the third image;

determining a motion between where the first camera is capturing the first image and the third image in a real environment coordinate system associated with at least part of the real environment; and

updating the determined at least one first light by combining the determined at least one first light and the determined at least one fifth light according to the motion.

12. The method according to claim 11, wherein the motion is determined according to at least one selected from a group consisting of non-face areas of the first and third images, at least one tracking sensor, and at least one image of at least part of the real environment captured by a second camera, wherein the second camera has a known spatial relationship to the first camera or is rigidly fixed relative to the first camera.

13. The method according to claim 1, wherein determining the at least one first light further comprises:

obtaining a lower bound constraint for a range of solutions of the at least one first light; and

solving at least one equation system for a solution of the at least one first light,

wherein the at least one equation system comprises unknown variables of the at least one first light, known parameters of the intensity information associated with the image positions of at least part of the face sample positions in the first image, and known parameters of the radiance transfer functions associated with the at least part of the face sample positions in the first image.

14. The method according to claim 1, wherein the method further comprises:

determining target intensity values for the one or more face sample positions according to the determined at least one first light and the radiance transfer functions associated with the one or more face sample positions;

determining intensity differences between the target intensity values and intensity information associated with the image positions of the one or more face sample positions in the first image; and

determining at least one sixth light falling on the face according to a fourth image of the human face and the determined intensity differences, wherein the fourth image and the first image are the same image or different images.

15. The method according to claim 14, wherein determining the at least one sixth light comprises;

determining weights for the at least part of the face sample positions according to the determined intensity differences, and wherein the determined weights are used to determine the at least one sixth light.

16. The method according to claim 1, wherein the radiance transfer functions are determined according to a training method comprising:

providing image information of a plurality of images of human faces in a plurality of different illuminations, wherein the plurality of different illuminations are known;

determining, for each of the plurality of images, image positions of at least part of the face sample positions; and

determining, for each of the plurality of face sample positions, the respective radiance transfer function according to intensity information associated with the image positions in at least part of the plurality of images and the illuminations and poses associated with the at least part of the plurality of images.

17. A non-transitory computer readable medium comprising software code sections which when executed causes one or more processors to:

obtain a first image of at least part of a human face in a real environment captured by a first camera;

determine one or more image positions in the first image corresponding to one or more face sample positions, wherein the one or more face sample positions are each associated with a known radiance transfer function;

determine at least one first light according to intensity information associated with the one or more image positions and the radiance transfer functions associated with the one or more face sample positions corresponding to the image positions in the first image; and

display the virtual object on a display device in the view of the real environment according to the at least one first light.

18. A system for representing a virtual object in a view of a real environment, comprising:

a first camera;

one or more processors; and

a memory coupled to the one or more processors and comprising computer readable instructions which, when executed by the one or more processors, cause the system to;

obtain an image of at least part of a human face in a real environment captured by the first camera;

determine one or more image positions in the first image corresponding to one or more face sample positions, wherein the one or more face sample positions are each associated with a known radiance transfer function;

determine at least one first light according to intensity information associated with the one or more image positions and the radiance transfer functions associated with the one or more face sample positions corresponding to the image positions in the first image; and

display the virtual object on a display device in the view of the real environment according to the at least one first light.

19. The system according to claim 18, comprising a mobile device which comprises the first camera and the display device, wherein the display device has a normal and the first camera has an optical axis, and the normal of the display device and the optical axis of the first camera are oriented in substantially the same direction.

20. The system according to claim 19, wherein the mobile device is a mobile phone, a tablet computer, or a mobile computer.

21. The system according to claim 19, wherein the one or more processors are comprised in the mobile device and/or in a server computer adapted to communicate with the mobile device.