Virtual and augmented reality systems and methods

• US 9,874,749 B2
• Filed: 03/07/2015
• Issued: 01/23/2018
• Est. Priority Date: 11/27/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Bradski, Gary R., Schowengerdt, Brian, Miller, Samuel A.
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US14/641,376
• Publication Number:
  US 20150178939A1
• Time in Patent Office: 1,053 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 30/34 Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details

Abstract

Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise a vision system having one or more optical elements to project one or more images to a user, a mapping database comprising map data corresponding to one or more real objects of the world, wherein the mapping database receives inputs from at least a component of one or more wearable augmented reality display systems, and a processor communicatively coupled to the mapping database for retrieving the map data and processing the retrieved map data to determine one or more output parameters, wherein the processor further controls the vision system in a manner such that the one or more images are projected to the user based at least in part on the determined output parameters.

Claims

1. A wearable augmented reality display system, comprising:

a vision system having one or more optical elements to project one or more images at respective one or more depth planes to a user;

a processor communicatively coupled to a mapping database for retrieving a map data corresponding to one or more real objects of the world, the mapping database receiving inputs from at least one component of one or more wearable augmented reality display systems;

a sensor to acquire sensor data; and

a pose module to utilize the retrieved map data and the sensor data to determine a head pose of the user,

wherein the processor processes the retrieved map data and the head pose of the user to determine one or more output parameters, and wherein the processor instructs the vision system to project the one or more images at the one or more depth planes to the user based at least in part on the determined output parameters.

2. The wearable augmented reality display system of claim 1, wherein the mapping database at least partly resides at a wearable component of the augmented reality display system, wherein the wearable component houses the processor and the vision system.

3. The wearable augmented reality display system of claim 1, wherein the mapping database at least partly resides at a cloud storage location.

4. The wearable augmented reality display system of claim 1, wherein the mapping database is accessed by multiple wearable augmented reality display systems.

5. The wearable augmented reality display system of claim 1, wherein the head pose comprises a position of the wearable augmented reality display system relative to the one or more real objects of the world.

6. The wearable augmented reality display system of claim 1, wherein the head pose comprises an orientation of the wearable augmented reality display system relative to the one or more real objects of the world.

7. The wearable augmented reality display system of claim 1, wherein the head pose is computed based at least in part on data collected through one or more input devices operatively coupled to the wearable augmented reality display system.

8. The wearable augmented reality display system of claim 1, wherein the sensor comprises an inertial measurement device (IMU).

9. The wearable augmented reality display system of claim 1, wherein the sensor comprises an accelerometer.

10. The wearable augmented reality display system of claim 1, wherein the sensor comprises gyro components.

11. The wearable augmented reality display system of claim 1, wherein the pose module receives surface information pertinent to one or more real or virtual objects.

12. The wearable augmented reality display system of claim 1, wherein the pose module outputs a sparse point representation of one or more objects based at least in part on the retrieved map data.

13. The wearable augmented reality display system of claim 12, wherein the sparse point representation are added to the mapping database to populate the map of the world.

14. The wearable augmented reality display system of claim 1, further comprising a depth map module communicatively coupled to the mapping database, wherein the depth map module is configured to receive information corresponding to one or more objects of the real world from one or more components of the wearable augmented reality display system, and determine depth information based at least in part on the received information.

15. The wearable augmented reality display system of claim 14, wherein the depth information added to the mapping database to populate the map of the world.

16. The wearable augmented reality display system of claim 14, wherein the one or more components of the wearable augmented reality display system are configured to obtain information corresponding to hand gestures of the user.

17. The wearable augmented reality display system of claim 14, wherein the one or more components of the wearable augmented reality display system comprises at least one infrared (IR) active projector.

18. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system comprises a gesture tracking module.

19. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system comprises an eye tracking module configured to track a vergence of the user'"'"'s eyes.

20. The wearable augmented reality display system of claim 19, wherein the vision system is configured to display at least the one or more images at the one or more depth planes to the user based at least in part on the tracked vergence.

21. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system from which input is received comprises at least one camera.

22. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system comprises a textured light projector to inject texture into a virtual scene projected by the vision system.

23. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system is calibrated.

24. The wearable augmented reality display system of claim 23, wherein the at least one component of the wearable augmented reality display system is calibrated intrinsically through one or more intrinsic parameters.

25. The wearable augmented reality display system of claim 23, wherein at least a portion of the calibration is performed in situ, based at least in part on anatomic variations of the user.

26. The wearable augmented reality display system of claim 23, wherein the at least one component of the wearable augmented reality display system is calibrated relative to another component of the wearable augmented reality display system.

41. The wearable augmented reality display system of claim 1, further comprising a fusion module to fuse a set of depth information.

42. The wearable augmented reality display system of claim 41, wherein the fusion module receives head pose information corresponding to one or more real objects of the world, wherein the head pose information is utilized to produce fused depth information correspond to one or more objects of the real world.

43. The wearable augmented reality display system of claim 41, wherein an output of the fusion module is added to the mapping database to populate the mapping database with surface information.

44. The wearable augmented reality display system of claim 24, wherein an intrinsic parameter of the one or more intrinsic parameters corresponding to a camera module comprises a focal length in pixels.

45. The wearable augmented reality display system of claim 24, wherein an intrinsic parameter of the one or more intrinsic parameters is a photogrammetric parameter.

46. The augmented reality display system of claim 24, wherein at least a portion of the calibration is performed in situ, based at least in part on anatomic variations of the user.

27. A method of displaying augmented reality using a wearable augmented reality display system having a vision system, a processor, a sensor, and a pose module, the method comprising:

the processor retrieving a map data from a mapping database, wherein the map data corresponds to one or more real objects of the world, and wherein the mapping database receives input from at least one component of the wearable augmented reality display systems;

the sensor acquiring sensor data;

the pose module processing the retrieved map data and the sensor data to determine a head pose of a user;

the processor processing the retrieved map data and the head pose of the user to determine one or more output parameters; and

the vision system projecting one or more images at respective one or more depth planes to the user based at least in part on the determined output parameters.

28. The method of claim 27, wherein the mapping database at least partly resides at a cloud storage location.

29. The method of claim 28, wherein the mapping database is accessed by multiple wearable augmented reality display systems.

30. The method of claim 27, wherein the head pose comprises a position of the wearable augmented reality display system relative to the one or more real objects of the world.

31. The method of claim 27, wherein the head pose comprises an orientation of the wearable augmented reality display system relative to the one or more real objects of the world.

32. The method of claim 27, further comprising the sensor acquiring data, wherein the pose module comprises the data acquired through the sensor.

33. The method of claim 27, further comprising the pose module outputting sparse point representations of one or more objects based at least in part on the retrieved map data.

34. The method of claim 33, wherein the sparse point representations are added to the mapping database to populate the map of the world.

35. The method of claim 27, wherein the wearable augmented reality display system also has a depth map module, the method further comprising:

the depth map module receiving information corresponding to one or more objects of the real world from one or more components of the wearable augmented reality display system; and

the depth map module determining depth information based at least in part on the received information.

36. The method of claim 27, wherein the wearable augmented reality display system also has a fusion module, the method further comprising the fusion module fusing a set of depth information to produce a surface representation of one or more objects of the real world.

37. The method of claim 27, wherein the at least one component of the wearable augmented reality display system comprises an eye tracking module, the method further comprising the eye tracking module tracking a vergence of the user'"'"'s eyes, wherein the tracked eye vergence is used as the at received input.

38. The method of claim 37, further comprising the vision system projecting the one or more images at the one or more depth planes to the user based at least in part on the tracked vergence.

39. The method of claim 27, further comprising calibrating the at least one component of the augmented reality display system.

40. The method of claim 39, wherein one component of the augmented reality display system is calibrated in relation to another component.