Real-time compositing in mixed reality

• US 10,475,248 B1
• Filed: 06/08/2018
• Issued: 11/12/2019
• Est. Priority Date: 06/08/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Klein, Andrew, Caillaud, Cedric, Kersey, Jeremy, Lee, Cedric, Hanke, William Ben
• Primary Examiner(s): Wang, Jin Cheng
• Application Number:
  US16/004,250
• Time in Patent Office: 522 Days
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 15/55 Radiosity
• CPC Class Codes: G06T 15/80 Shading
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes

Abstract

A system may include a memory device to store instructions and data, and at least one processing device to execute the instructions stored in the memory device to: receive a background image and a digital object to be composited onto the background image in a mixed reality view, generate a 2D bounding region for the digital object, select a version of the background image at a pre-defined resolution, overlay the 2D bounding region on the selected version of the background image and obtain a set of samples of the colors of pixels of the selected version along a perimeter of the 2D bounding region, and determine a value for one or more digital lighting sources to illuminate the digital object in the mixed reality view, based, at least in part, on the set of samples.

Claims

1. A system, comprising:

a memory device to store instructions and data; and

at least one processing device to execute the instructions stored in the memory device to;

receive a background image and a digital object to be composited onto the background image in a mixed reality view;

generate a 2D bounding region for the digital object;

select a version of the background image at a pre-defined resolution;

overlay the 2D bounding region on the selected version of the background image and obtain a set of samples of the colors of pixels of the selected version along a perimeter of the 2D bounding region; and

determine a value for one or more digital lighting sources to illuminate the digital object in the mixed reality view, based, at least in part, on the set of samples.

2. The system of claim 1, further comprising a camera or image sensor coupled to the processing device, and wherein the background image is a frame of video captured by the camera or image sensor in real time.

3. The system of claim 1, wherein the background image is stored in the memory prior to being loaded by the processing device.

4. The system of claim 1, wherein the at least one processing device executes the instructions stored in the memory device further to calculate an ambient light value.

5. The system of claim 4, wherein the at least one processing device executes the instructions stored in the memory device further to, at least one of:

tint light reflected onto the digital object, or

adjust intensity of light reflected onto the digital object,

based, at least in part, on the ambient light value.

6. The system of claim 4, wherein the ambient light value is an average color of the set of samples.

7. The system of claim 1, wherein the at least one processing device executes the instructions stored in the memory device further to:

generate a set of versions of the background image at different resolutions.

8. The system of claim 1, wherein at least one of:

the selected version of the background image is based, at least in part, on the dimensions of the 2D bounding region.

9. The system of claim 1, wherein the at least one processing device executes the instructions stored in the memory device further to:

determine the value for the one or more digital lighting sources based, at least in part, on weighting the set of samples differently for each digital lighting source, and combining the weighted samples.

10. A method, comprising:

receiving a background image and a digital object to be composited onto the background image in a mixed reality view;

generating a 2D bounding region for the digital object;

selecting a version of the background image at a pre-defined resolution;

overlaying the 2D bounding region on the selected version of the background image and obtaining a set of samples of the colors of pixels of the selected version along a perimeter of the 2D bounding region; and

determining a value for one or more digital lighting sources to illuminate the digital object in the mixed reality view, based, at least in part, on the set of samples.

11. The method of claim 10, wherein the background image is one of:

a frame of video captured by a camera or image sensor coupled to the processing device in real time;

or

stored in memory prior to being received by the processing device.

12. The method of claim 10, further comprising calculating, by the processing device, an ambient light value at the location of the digital object.

13. The method of claim 12, further comprising:

tinting light reflected onto the digital object, or adjusting intensity of light reflected onto the digital object, based, at least in part, on the ambient light value.

14. The method of claim 12, wherein the ambient light value is an average color of the set of samples.

15. The method of claim 10, further comprising rendering, by the processing device, the digital object as illuminated by the digital lighting sources and displaying the digital object on a display device coupled to the processor.

16. The method of claim 15, further comprising rendering, by the processing device, the digital object with simulated shadows.

17. A computer-readable storage medium of a tangible article of manufacture comprising instructions that, when executed by one or more processing devices, causes the one or more processing devices to:

receive a background image and a digital object to be composited onto the background image in a mixed reality view;

generate a 2D bounding region for the digital object;

select a version of the background image at a pre-defined resolution;

overlay the 2D bounding region on the selected version of the background image and obtain a set of samples of the colors of pixels of the selected version along a perimeter of the 2D bounding region; and

determine a value for one or more digital lighting sources to illuminate the digital object in the mixed reality view, based, at least in part, on the set of samples.

18. The computer-readable storage medium of claim 17, wherein execution of the instructions by the one or more processing devices, causes the one or more processing devices further to:

calculate an ambient light value, the ambient light value being an average color of the set of samples.

19. The computer-readable storage medium of claim 18, wherein execution of the instructions by the one or more processing devices, causes the one or more processing devices further to:

tint light reflected onto the digital object, or

adjust intensity of light reflected onto the digital object,

based, at least in part, on the ambient light value.

20. The computer-readable storage medium of claim 17, wherein execution of the instructions by the one or more processing devices, cause the one or more processing devices further to:

determine a value for the one or more digital lighting sources to illuminate the digital object in the mixed reality view, based, at least in part, on weighting the set of samples differently for each digital lighting source, and combining the weighted samples.