Systems and methods for using image based light solutions for augmented or virtual reality

• US 9,911,233 B2
• Filed: 05/07/2015
• Issued: 03/06/2018
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: O'Connor, Philip, Miller, Samuel A.
• Primary Examiner(s): Wang, Jin-Cheng
• Application Number:
  US14/706,809
• Publication Number:
  US 20150302660A1
• Time in Patent Office: 1,034 Days
• Field of Search: 345633, 382103, 382190, 715852, 707609, 707722
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. A method for displaying virtual or augmented reality, comprising:

capturing at least one parameter associated with ambient light in an environment;

identifying a first light map corresponding to one or more first light map parameters and a second light map corresponding to one or more second light map parameters based at least in part on at least one parameter, wherein a light map includes a geometric space and models light information of virtual content and the environment in which a user is located;

identifying one or more errors in one or more light map parameters of the one or more first light map parameters or the one or more second light map parameters based in part or in whole upon the at least one parameter associated with ambient light and;

constructing and storing a new light map in a library comprising the first light map and the second light map at least by combining the one or more first light map parameters respectively with the one or more second light map parameters into one or more corresponding combined parameters and by modifying at least one image generation characteristic of a virtual or augmented reality system to reduce the one or more errors in the one or more light map parameters;

modifying the virtual content to be presented to a user based at least in part on the new light map; and

projecting light associated with the virtual content that has been modified.

2. The method of claim 1, wherein a light associated with the virtual content resembles that associated with real objects in an ambient environment of the user.

3. The method of claim 1, further comprising storing a library of light maps, wherein each light map of the library of light maps corresponds to a plurality of light parameters, and the first light map or a second map is identified further based in part or in whole upon an input of a user.

4. The method of claim 1, wherein selecting the light map is based at least in part on a closest approximation light map that comprises one or more characteristics that are closest to the at least one parameter.

5. The method of claim 1, wherein the at least one parameter corresponds to frequency data of the light.

6. The method of claim 1, wherein the at least one parameter corresponds to a color palette of the light.

7. The method of claim 1, wherein the at least one parameter corresponds to a dynamic range of the light.

8. The method of claim 1, wherein selecting the light map is based at least in part on a comparison of the at least one parameter against one or more other parameters associated with a plurality of light maps.

9. The method of claim 1, consulting with a neural network to select the first or the second light map.

10. The method of claim 1, further comprising modifying the first or the second light map based at least in part on the at least one parameter pertaining to the environment.

11. The method of claim 1, further comprising combining data from a plurality of light maps based at least in part on the at least one parameter pertaining to the environment.

12. The method of claim 11, further comprising creating the new light map based at least in part on the data that has been combined.

13. The method of claim 1, wherein

capturing the at least one parameter associated with ambient light comprises capturing one or more images of a 360 degree view of the environment, and

selecting the first or the second light map comprises creating a light map based at least in part on the one or more images of the 360 degree view of the environment.

14. The method of claim 13, wherein the light map is user-centric.

15. The method of claim 14, further comprising applying a transformation to the light map, wherein the transformation reduces an error corresponding to a distance between the user and a virtual object to be presented to the user.

16. The method of claim 14, further comprising:

modeling the light map as a user-centric sphere centered on the user;

modeling an object-centric sphere around a virtual object to be lit; and

projecting data from the user-centric sphere onto the object-centric sphere from a point of view of the virtual object, thereby creating a different light map.

17. The method of claim 13, further comprising attenuating a color intensity of the first or the second light map based at least in part on a distance between the user and a virtual object to be presented to the user.

18. The method of claim 13, further comprising determining a depth value of a plurality of axes of the first or the second light map.

19. The method of claim 18, further comprising determining respective coordinates of the plurality of axes, and wherein a color intensity of the first or the second light map is attenuated based at least in part on the respective coordinates of the plurality of axes, thereby creating a different light map.

20. The method of claim 1, further comprising:

storing a map of a real world, wherein the map comprises coordinates of one or more real objects of the real world; and

storing a plurality of light maps in a grid based at least in part on the map of the real world.

21. The method of claim 20, further comprising selecting the first or the second light map based at least in part on a detected location of the user of a virtual or augmented reality device and a stored grid of light maps.

22. The method of claim 1, further comprising updating the first or the second light map based at least in part on the parameters.

23. The method of claim 22, wherein the first or the second light map is updated such that it is not perceived by the user of an augmented reality device.

24. The method of claim 22, wherein the first or the second light map is updated based at least in part on a detected circumstance.

25. The method of claim 24, wherein the detected circumstance comprises an eye movement of the user.

26. The method of claim 24, wherein the detected circumstance comprises a presence of a shadow over a virtual object.

27. The method of claim 24, wherein the detected circumstance comprises a dimming of a light of the environment.

28. The method of claim 24, wherein the detected circumstance comprises another virtual object that is to keep a focus of the user.

29. The method of claim 22, further comprising updating the first or the second light map when a virtual object is out of a field of view of the user.

30. The method of claim 22, further comprising updating the first or the second light map when a virtual object is at a periphery of a field of view of the user.