SYSTEMS AND METHODS FOR AUGMENTED REALITY

• US 20170205903A1
• Filed: 02/06/2017
• Published: 07/20/2017
• Est. Priority Date: 03/05/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Woods, Michael J., Miller, Samuel A., Lundmark, David C.
• Granted Patent: US 10,180,734 B2
• Time in Patent Office: Days
• CPC Class Codes: G01B 7/004 for measuring coordinates of points
• CPC Class Codes: G01R 33/02 Measuring direction or magnitude of magnetic fields or magnetic flux G01R33/20 takes precedence
• CPC Class Codes: G01S 13/02 Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0125 Field-of-view increase by wavefront division
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0132 comprising binocular systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/174 : involving the use of two or more images
• CPC Class Codes: G06T 7/521 : from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 : from multiple images

Abstract

An augmented reality display system includes an electromagnetic field emitter to emit a known magnetic field in a known coordinate system. The system also includes an electromagnetic sensor to measure a parameter related to a magnetic flux at the electromagnetic sensor resulting from the known magnetic field. The system further includes a depth sensor to measure a distance in the known coordinate system. Moreover, the system includes a controller to determine pose information of the electromagnetic sensor relative to the electromagnetic field emitter in the known coordinate system based at least in part on the parameter related to the magnetic flux measured by the electromagnetic sensor and the distance measured by the depth sensor. In addition, the system includes a display system to display virtual content to a user based at least in part on the pose information of the electromagnetic sensor relative to the electromagnetic field emitter.

Claims

1. An augmented reality (AR) display system, comprising:

an electromagnetic field emitter to emit a known magnetic field in a known coordinate system;

an electromagnetic sensor to measure a parameter related to a magnetic flux at the electromagnetic sensor resulting from the known magnetic field;

a depth sensor to measure a distance in the known coordinate system;

a controller to determine pose information of the electromagnetic sensor relative to the electromagnetic field emitter in the known coordinate system based at least in part on the parameter related to the magnetic flux measured by the electromagnetic sensor and the distance measured by the depth sensor; and

a display system to display virtual content to a user based at least in part on the pose information of the electromagnetic sensor relative to the electromagnetic field emitter.

9. The AR display system of claim 1, further comprising a world capture camera and a picture camera,

wherein the depth sensor comprises a depth camera having a first field of view (FOV),

wherein the world capture camera has a second FOV at least partially overlapping with the first FOV,

wherein the picture camera has a third FOV at least partially overlapping with the first FOV and the second FOV, and

wherein the depth camera, the world capture camera, and the picture camera are configured to capture respective first, second, and third images.

15. The AR display system of claim 9, wherein the controller is programmed to segment the second and third images.

16. The AR display system of claim 15, wherein the controller is programmed to fuse the second and third images after segmenting the second and third images to generate a fused image.

17. The AR display system of claim 16, wherein measuring a distance in the known coordinate system comprises:

generating a hypothetical distance by analyzing the first image from the depth camera; and

generating the distance by analyzing the hypothetical distance and the fused image.

19. The AR display system of claim 1, further comprising an additional localization resource to provide additional information, wherein the pose information of the electromagnetic sensor relative to the electromagnetic field emitter in the known coordinate system is determined based at least in part on the parameter related to the magnetic flux measured by the electromagnetic sensor, the distance measured by the depth sensor, and the additional information provided by the additional localization resource.

38. The AR display system of claim 1, wherein the electromagnetic field emitter is coupled to a mobile component of the AR display system.

44. The AR display system of claim 1, wherein the electromagnetic field emitter is coupled to an object in the known coordinate system, such that the electromagnetic field emitter has a known position and a known orientation.

45. The AR display system of claim 44, wherein the electromagnetic sensor is coupled to a mobile component of the AR display system.

2-8. -8. (canceled)

10-14. -14. (canceled)

18. (canceled)

20-37. -37. (canceled)

39-43. -43. (canceled)

46-52. -52. (canceled)

53. A method for displaying augmented reality, the method comprising:

emitting, using an electromagnetic field emitter, a known magnetic field in a known coordinate system;

measuring, using an electromagnetic sensor, a parameter related to a magnetic flux at the electromagnetic sensor resulting from the known magnetic field;

measuring, using a depth sensor, a distance in the known coordinate system;

determining pose information of the electromagnetic sensor relative to the electromagnetic field emitter in the known coordinate system based at least in part on the parameter related to the magnetic flux measured using the electromagnetic sensor and the distance measured using the depth sensor; and

displaying virtual content to a user based at least in part on the pose information of the electromagnetic sensor relative to the electromagnetic field emitter.

61. The method of claim 53,

wherein the depth sensor comprises a depth camera having a first field of view (FOV), the method further comprising:

capturing a first image using the depth camera;

capture a second image using a world camera having a second FOV at least partially overlapping with the first FOV; and

capturing a third image using a picture camera having a third FOV at least partially overlapping with the first FOV and the second FOV.

67. The method of claim 61, further comprising segmenting the second and third images.

68. The method of claim 67, further comprising fusing the second and third images after segmenting the second and third images to generate a fused image.

69. The method of claim 68, wherein measuring a distance in the known coordinate system comprises:

generating a hypothetical distance by analyzing the first image from the depth camera; and

generating the distance by analyzing the hypothetical distance and the fused image.

71. The method of claim 53, further comprising, determining the pose information of the electromagnetic sensor relative to the electromagnetic field emitter in the known coordinate system based at least in part on the parameter related to the magnetic flux measured using the electromagnetic sensor, the distance measured using the depth sensor, and additional information provided by an additional localization resource.

90. The method of claim 53, wherein the electromagnetic field emitter is coupled to a mobile component of an AR display system.

96. The method of claim 53, wherein the electromagnetic field emitter is coupled to an object in the known coordinate system, such that the electromagnetic field emitter has a known position and a known orientation.

97. The method of claim 96, wherein the electromagnetic sensor is coupled to a mobile component of an AR display system.

104. The method of claim 53, further comprising analyzing the pose information to determine a position and an orientation of the electromagnetic sensor in the known coordinate system.

54-60. -60. (canceled)

62-66. -66. (canceled)

70. (canceled)

72-89. -89. (canceled)

91-95. -95. (canceled)

98-103. -103. (canceled)

105. An augmented reality display system, comprising:

a hand-held component coupled to an electromagnetic field emitter, the electromagnetic field emitter emitting a magnetic field;

a head-mounted component having a display system that displays virtual content to a user, the head mounted component coupled to an electromagnetic sensor measuring a parameter related to a magnetic flux at the electromagnetic sensor resulting from the magnetic field, wherein a head pose of the head-mounted component in a known coordinate system is known;

a depth sensor measuring a distance in the known coordinate system; and

a controller communicatively coupled to the hand-held component, the head-mounted component, and the depth sensor, the controller receiving the parameter related to the magnetic flux at the electromagnetic sensor from the head mounted component and receiving the distance from the depth sensor,

wherein the controller determines a hand pose of the hand-held component based at least in part on the parameter related to the magnetic flux measured by the electromagnetic sensor and the distance measured by the depth sensor,

wherein the system modifies the virtual content displayed to the user based at least in part on the hand pose.

106-143. -143. (canceled)