VIRTUAL AND AUGMENTED REALITY SYSTEMS AND METHODS

• US 20150178939A1
• Filed: 03/07/2015
• Published: 06/25/2015
• Est. Priority Date: 11/27/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Bradski, Gary R., Miller, Samuel A., Schowengerdt, Brian
• Granted Patent: US 9,874,749 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 30/34 Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details

Abstract

Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise a vision system having one or more optical elements to project one or more images to a user, a mapping database comprising map data corresponding to one or more real objects of the world, wherein the mapping database receives inputs from at least a component of one or more wearable augmented reality display systems, and a processor communicatively coupled to the mapping database for retrieving the map data and processing the retrieved map data to determine one or more output parameters, wherein the processor further controls the vision system in a manner such that the one or more images are projected to the user based at least in part on the determined output parameters.

Claims

1. A wearable augmented reality display system, comprising:

a vision system having one or more optical elements to project one or more images to a user; and

a processor communicatively coupled to a mapping database for retrieving a map data corresponding to one or more real objects of the world, the mapping database receiving inputs from at least a component of one or more wearable augmented reality display systems, wherein the processor processes the retrieved map data to determine one or more output parameters, and wherein the processor further controls the vision system in a manner such that the one or more images projected to the user are based at least in part on the determined output parameters.

2. The wearable augmented reality display system of claim 1, wherein the mapping database at least partly resides at a wearable component of the augmented reality display system, wherein the wearable component houses the processor and the vision system.

3. The wearable augmented reality display system of claim 1, wherein the mapping database at least partly resides at a cloud storage location.

4. The wearable augmented reality display system of claim 1, wherein the mapping database is accessed by multiple wearable augmented reality display systems.

5. The wearable augmented reality display system of claim 1, further comprising a pose module, wherein the pose module is configured to utilize the retrieved map data and determine a pose of the user.

7. The wearable augmented reality display system of claim 5, wherein the pose comprises a position of the wearable augmented reality display system relative to the one or more real objects of the world.

8. The wearable augmented reality display system of claim 5, wherein the pose comprises an orientation of the wearable augmented reality display system relative to the one or more real objects of the world.

9. The wearable augmented reality display system of claim 5, wherein the pose module comprises data acquired through one or more sensors corresponding to the wearable augmented reality display system.

10. The wearable augmented reality display system of claim 5, wherein the pose is computed based at least in part on data collected through one or more input devices operatively coupled to the wearable augmented reality display system.

11. The wearable augmented reality display system of claim 9, wherein the sensor comprises an inertial measurement device (IMU).

12. The wearable augmented reality display system of claim 9, wherein the sensor comprises an accelerometer.

13. The wearable augmented reality display system of claim 9, wherein the sensor comprises gyro components.

14. The wearable augmented reality display system of claim 5, wherein the pose module receives surface information pertinent to one or more real or virtual objects.

15. The wearable augmented reality display system of claim 5, wherein the pose module outputs a sparse point representation of one or more objects based at least in part on the retrieved map data.

16. The wearable augmented reality display system of claim 15, wherein the sparse point representation are added to the mapping database to populate the map of the world.

17. The wearable augmented reality display system of claim 1, further comprising a depth map module communicatively coupled to the mapping database, wherein the depth map module is configured to receive information corresponding to one or more objects of the real world from one or more components of the wearable augmented reality display system, and determine depth information based at least in part on the received information.

18. The wearable augmented reality display system of claim 17, wherein the depth information added to the mapping database to populate the map of the world.

19. The wearable augmented reality display system of claim 17, wherein the one or more components of the wearable augmented reality display system comprises information corresponding to hand gestures of the user.

20. The wearable augmented reality display system of claim 17, wherein the one or more components of the wearable augmented reality display system comprises at least one infrared (IR) active projector.

21. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system comprises a gesture tracking module.

22. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system comprises an eye tracking module configured to track a vergence of the user'"'"'s eyes.

23. The wearable augmented reality display system of claim 22, wherein the vision system is configured to display at least the one or more images on at least one depth plane to the user based at least in part on the tracked vergence.

24. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system from which input is received comprises at least one camera.

25. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system comprises a textured light projector to inject texture into a virtual scene projected by the vision system.

26. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system is calibrated.

27. The wearable augmented reality display system of claim 26, wherein the at least one component of the wearable augmented reality display system is calibrated intrinsically through one or more intrinsic parameters.

28. The wearable augmented reality display system of claim 26, wherein at least a portion of the calibration is performed in situ, based at least in part on anatomic variations of the user.

29. The wearable augmented reality display system of claim 26. The wearable augmented reality display system of claim 1, wherein the at least one component of the wearable augmented reality display system is calibrated relative to another component of the wearable augmented reality display system.

35. The method of claim 3, wherein the pose comprises an orientation of the wearable augmented reality display system relative to the one or more real objects of the world.

30. A method of displaying augmented reality, comprising:

retrieving a map data from a mapping database, wherein the map data corresponds to one or more real objects of the world, and wherein the mapping database receives inputs from at least a component of one or more wearable augmented reality display systems;

processing the retrieved map data to determine one or more output parameters; and

projecting one or more images to a user, wherein the one or more images are projected based at least in part on the determined output parameters.

31. The method of claim 30, wherein the mapping database at least partly resides at a cloud storage location.

32. The method of claim 31, wherein the mapping database is accessed by multiple wearable augmented reality display systems.

33. The method of claim 31, further comprising utilizing the retrieved map data to determine a pose of the user.

34. The method of claim 33, wherein the pose comprises a position of the wearable augmented reality display system relative to the one or more real objects of the world.

40. The method of claim 33, further comprising acquiring data wherein the pose module comprises data acquired through one or more sensors.

41. The method of claim 33, further comprising outputting sparse point representations of one or more objects based at least in part on the retrieved map data.

42. The method of claim 41, wherein the sparse point representations are added to the mapping database to populate the map of the world.

43. The method of claim 30, further comprising receiving information corresponding to one or more objects of the real world from one or more components of the wearable augmented reality display system, and determining depth information based at least in part on the received information.

44. The method of claim 30, further comprising fusing a set of depth information to produce a surface representation of one or more objects of the real world.

45. The method of claim 30, further comprising tracking a vergence of the user'"'"'s eyes, wherein the tracked eye vergence is used as the at least one received input.

46. The method of claim 45, further comprising projecting the one or more images on at least one depth plane to the user based at least in part on the tracked vergence.

47. The method of claim 30, further comprising calibrating the at least one component of the augmented reality display system.

48. The method of claim 47, wherein one component of the augmented reality display system is calibrated in relation to another component.

49. An augmented reality display system, comprising:

a vision system having one or more optical elements to project one or more images to a user;

a mapping database comprising map data corresponding to one or more real objects of the world, wherein the mapping database receives inputs from at least a component of one or more wearable augmented reality display systems; and

a processor communicatively coupled to the mapping database for retrieving the map data and processing the retrieved map data to determine one or more output parameters, wherein the processor further controls the vision system in a manner such that the one or more images are projected to the user based at least in part on the determined output parameters.

50. The augmented reality display system of claim 49, wherein the mapping database at least partly resides at a wearable component of the augmented reality display system, wherein the wearable component houses the processor and the vision system.

51. The augmented reality display system of claim 49, wherein the mapping database at least partly resides at a cloud storage location.

52. The augmented reality display system of claim 51, further comprising a pose module, wherein the pose module is configured to utilize the retrieved map data and determine a pose of the user.

53. The augmented reality display system of claim 52, wherein the pose module comprises data acquired through one or more sensors corresponding to the wearable augmented reality display system.

54. The augmented reality display system of claim 52, wherein the pose is computed based at least in part on data collected through one or more input devices operatively coupled to the wearable augmented reality display system.

55. The augmented reality display system of claim 52, wherein the pose module outputs a sparse point representation of one or more objects based at least in part on the retrieved map data.

56. The augmented reality display system of claim 55, wherein the sparse point representation are added to the mapping database to populate the map of the world.

57. The augmented reality display system of claim 49, further comprising a depth map module communicatively coupled to the mapping database, wherein the depth map module is configured to receive information corresponding to one or more objects of the real world from one or more components of the wearable augmented reality display system, and determine depth information based at least in part on the received information.

58. The augmented reality display system of claim 57, further comprising a fusion module to fuse a set of depth information.

59. The augmented reality display system of claim 57, wherein the fusion module receives pose information corresponding to one or more real objects of the world, wherein the pose information is utilized to produce fused depth information correspond to one or more objects of the real world.

60. The augmented reality display system of claim 58, wherein an output of the fusion module is added to the mapping database to populate the mapping database with surface information.

61. The augmented reality display system of claim 49, wherein the at least one component of the wearable augmented reality display system is calibrated.

62. The augmented reality display system of claim 61, wherein the at least one component of the wearable augmented reality display system is calibrated intrinsically through one or more intrinsic parameters.

63. The augmented reality display system of claim 62, wherein the intrinsic parameter of a camera module comprises a focal length in pixels.

64. The augmented reality display system of claim 62, wherein the intrinsic parameter is a photogrammetric parameter.

65. The augmented reality display system of claim 62, wherein at least a portion of the calibration is performed in situ, based at least in part on anatomic variations of the user.

66. The augmented reality display system of claim 62, wherein the calibration is performed at run-time to calibrate the display system for a particular user.

67. The augmented reality display system, further comprising an eye tracking module to track a vergence of the user'"'"'s eyes.

68. The augmented reality display system of claim 67, wherein the vision system is configured to display at least the one or more images on at least one depth plane to the user based at least in part on the tracked vergence.