Handling real-world light sources in virtual, augmented, and mixed reality (xR) applications

• US 10,540,812 B1
• Filed: 01/09/2019
• Issued: 01/21/2020
• Est. Priority Date: 01/09/2019
• Status: Active Grant

• Current Assignee: Dell Products LP (Dell Technologies Inc.)
• Original Assignee: Dell Products LP (Dell Technologies Inc.)
• Inventors: Yildiz, Yagiz Can, Gatson, Michael S., Comer, Ryan Nicholas
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US16/243,924
• Time in Patent Office: 377 Days
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/579 from motion

Abstract

Embodiments of systems and methods for handling real-world light sources in virtual, augmented, and mixed reality (xR) applications are described. In an illustrative, non-limiting implementation, an Information Handling System (IHS) may include a processor and a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the IHS to: detect a real-world light source using a sensor mounted on a Head-Mounted Device (HMD); identify a property of the real-world light source; and indicate the property to a rendering engine during execution of an xR application, where the rendering engine is configured to render an image for display by the HMD based on the property.

Claims

1. An Information Handling System (IHS), comprising:

a processor; and

a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the IHS to;

detect a real-world light source using a sensor mounted on a Head-Mounted Device (HMD);

identify a property of the real-world light source; and

indicate the property to a rendering engine during execution of a virtual, augmented, or mixed reality (xR) application, at least in part, by providing at least a portion of a look-up table (LUT) of virtual light sources comprising type, position, color, intensity, and size using an Application Programming Interface (API), wherein the rendering engine is configured to render an image for display by the HMD based on the property.

2. The IHS of claim 1, wherein the real-world light source comprises a natural or artificial light source.

3. The IHS of claim 1, wherein the sensor comprises an RGB camera.

4. The IHS of claim 1, wherein the property comprises at least one of:

location, size, intensity, or color.

5. The IHS of claim 4, wherein determining the location further comprises determining HMD pose derived from Simultaneous Localization and Mapping (SLAM) data.

6. The IHS of claim 4, wherein determining the location further comprises determining a distance between the real-world light source and the HMD using a depth camera.

7. The IHS of claim 1, wherein the program instructions, upon execution by the processor, further cause the IHS to transform a real-world location of the real-world light source to a virtual location in a coordinate system used by the xR application.

8. The IHS of claim 1, wherein the program instructions, upon execution by the processor, further cause the IHS to classify the real-world light source as:

point, spot, directional, or area.

9. The IHS of claim 1, wherein the program instructions, upon execution by the processor, further cause the IHS to:

detect movement of the real-world light source during execution of a virtual, augmented, or mixed reality (xR) application; and

update the property to reflect the movement.

10. The IHS of claim 1, wherein the program instructions, upon execution by the processor, further cause the IHS to:

detect a change in an attribute of the real-world light source during execution of a virtual, augmented, or mixed reality (xR) application; and

update the property to reflect the change in the attribute.

11. A method, comprising:

identifying a real-world light source using a camera mounted on a Head-Mounted Device (HMD);

modeling the real-world light source as a virtual light source having a type, position, color, intensity, and size;

providing at least a portion of a look-up table (LUT) of virtual light sources comprising type, position, color, intensity, and size to a rendering engine using an Application Programming Interface (API); and

rendering a three-dimensional overlay image for display by the HMD, at least in part, using the portion of the LUT.

12. The method of claim 11, further comprising classifying the real-world light source as:

point, spot, directional, or area.

13. The method of claim 11, further comprising:

detecting movement of the real-world light source;

simulating a corresponding movement of the virtual light source; and

updating an illumination of the overlay image according to the movement.

14. The method of claim 11, further comprising:

detecting a change in intensity of the real-world light source;

simulating a corresponding change of intensity of the virtual light source; and

updating an illumination of the overlay image according to the change in intensity.

15. The method of claim 11, further comprising:

detecting a change in color of the real-world light source;

simulating a corresponding change of color of the virtual light source; and

updating an illumination of the overlay image according to the change in color.

16. A hardware memory of a Head-Mounted Device (HMD), the hardware memory having program instructions stored thereon that, upon execution by a processor, cause the HMD to:

capture an image of a real-world light source during execution of an augmented reality (AR) application using a sensor mounted on a Head-Mounted Device (HMD);

provide the image to an Information Handling System (IHS) coupled to the HMD, wherein the IHS is configured to;

(a) identify a property of the real-world light source, at least in part, using at least a portion of a look-up table (LUT) of virtual light sources comprising type, position, color, intensity, and size using an Application Programming Interface (API), and (b) illuminate or shade an image rendered using the property;

receive the rendered image from the IHS; and

display the rendered image.

17. The hardware memory of claim 16, wherein to illuminate or shade the image rendered using the property, the IHS is further configured to transform a real-world location of the real-world light source to another location in a coordinate system used by the AR application.