Augmented reality lighting effects

• US 10,354,449 B2
• Filed: 06/07/2016
• Issued: 07/16/2019
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Hand Held Products Incorporated (Honeywell International Inc.)
• Original Assignee: Hand Held Products Incorporated (Honeywell International Inc.)
• Inventors: Todeschini, Erik
• Primary Examiner(s): Tswei, YuJang
• Application Number:
  US15/175,626
• Publication Number:
  US 20160364914A1
• Time in Patent Office: 1,134 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 15/55 Radiosity
• CPC Class Codes: G06T 15/80 Shading
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G09G 2320/0626 for control of overall brightness
• CPC Class Codes: G09G 2320/0686 with two or more screen areas displaying information with different brightness or colours
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 3/001 using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background slide projectors per se G03B23/00 = 42 HP
• CPC Class Codes: G09G 3/36 using liquid crystals

Abstract

The present invention embraces a system, device, and method for adding lighting effects to augmented reality (AR) content (i.e., virtual objects). Light sensors in an augmented reality (AR) system monitor an environment'"'"'s lighting conditions to acquire lighting data that can be used to create (or update) virtual light sources. Depth sensors in the AR system sense the environment to acquire mapping data that can be used to create a 3D model of the environment while tracking the system'"'"'s location within the environment. Algorithms running on a processor may then add the virtual light sources to the 3D model of the environment so that, when AR content is created, lighting effects corresponding to the virtual light sources can be added. The resulting AR content with virtual lighting effects appear more realistic to a user.

Claims

1. An augmented reality (AR) system, comprising:

a display for displaying AR content to a user, the AR content overlapping the AR system'"'"'s perspective view of an environment;

one or more light sensors for gathering light data from light sources in the environment;

one or more depth sensors for gathering mapping data of (i) physical objects in the environment and (ii) the light sources in the environment; and

a computing device communicatively coupled to the display, the one or more light sensors, and the one or more depth sensors, the computing device comprising a processor that is configured by software to;

construct a three-dimensional (3D) model of the environment using the mapping data,

detect and characterize the light sources in the environment using the light data,

add virtual light sources to the 3D model of the environment, the virtual light sources corresponding to the light sources in the environment,

create AR content corresponding to the AR system'"'"'s perspective view of the environment, wherein the AR content is configured to be proximate a physical object in the AR system'"'"'s perspective view of the environment,

add lighting effects to the AR content, the lighting effects corresponding to the virtual light sources, wherein the lighting effects added to the AR content, by the computing device and without user intervention, match with physical lighting effects of the physical object, and

transmit the AR content with lighting effects to the display,

wherein the physical object is a package and the AR content is configured to indicate the package.

2. The AR system according to claim 1, wherein the lighting effects correspond to the environment'"'"'s ambient light level.

3. The AR system according to claim 1, wherein the lighting effects correspond to (i) the characteristics of the virtual light sources and (ii) the positions and orientations of the virtual light sources with respect to AR content.

4. The AR system according to claim 3, wherein the characteristics of the virtual light sources comprise direction of radiation.

5. The AR system according to claim 3, wherein the characteristics of the virtual light sources comprise light color.

6. The AR system according to claim 3, wherein the characteristics of the virtual light sources comprise light intensity.

7. The AR system according to claim 1, wherein the AR content comprises a graphical 3D object.

8. The AR system according to claim 1, wherein the at least one light sensor comprises a charge-coupled device (CCD).

9. The AR system according to claim 8, wherein the detection of light sources comprises comparing the CCD'"'"'s pixel values to a threshold level.

10. The AR system according to claim 1, wherein the at least one depth sensor is an optical 3D scanner.

11. The AR system according to claim 1, wherein the display comprises a transparent plate that is (i) positioned in front of the user'"'"'s eye or eyes, allowing the user to view the environment through the transparent plate and (ii) arranged to display AR content to the user'"'"'s eye or eyes so that the AR content appears superimposed on the user'"'"'s view of the environment.

12. The AR system according to claim 1, wherein the display comprises a liquid crystal display (LCD).

13. The AR system according to claim 1, the adding virtual light sources to the 3D model of the environment is facilitated by simultaneous location and mapping (SLAM) techniques.

14. A method for applying lighting effects to virtual objects for an augmented reality (AR) system, wherein a virtual object of the virtual objects is configured to be proximate a physical object in the AR system'"'"'s perspective view of an environment, the method comprising:

receiving lighting information from at least one light sensor;

receiving position information from at least one depth sensor;

determining lighting effects from the lighting information and the position information; and

applying lighting effects to the virtual objects, comprising matching, without user intervention, lighting effects applied to the virtual object with physical lighting effects of the physical object;

updating the lighting effects if (i) the position information changes or (ii) the lighting information changes,

wherein the physical object is a package and the virtual object is configured to indicate the package.

15. The method according to claim 14, wherein the AR system comprises a head mounted display (HMD).

16. The method according to claim 14, wherein the at least one light sensor is a charge-coupled device (CCD) camera.

17. The method according to claim 14, wherein the lighting effects comprise coloring, shading, and/or lightening at least a portion of a virtual object.

18. An augmented reality (AR) device, comprising:

a light sensor to gather light data for constructing virtual light sources;

a depth sensor to gather mapping data for constructing a three-dimensional (3D) model of an environment; and

a processor communicatively coupled to the light sensor and the depth sensor, the processor configured by software to;

construct the 3D model of the environment including virtual light sources,

create AR content, wherein the AR content is configured to be proximate a physical object in the AR system'"'"'s perspective view of the environment, and

add lighting effects to the AR content, the lighting effects corresponding to the virtual light sources, wherein the lighting effects added to the AR content, without user intervention, match with physical lighting effects of the physical object,

wherein the physical object is a package and the AR content is configured to indicate the package.

19. The AR device according to claim 18, wherein the virtual light sources comprise a diffuse light source, a point light source, a directional light source, a distributed light source, and/or an ambient light source.

20. The AR device according to claim 18, wherein adding lighting effects to the AR content comprises changing the AR content'"'"'s color, intensity, and/or shading so as to appear illuminated by a light source.