Methods and systems for mapping virtual objects in an augmented or virtual reality system

• US 10,115,233 B2
• Filed: 05/06/2015
• Issued: 10/30/2018
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A., Hand, Randall E.
• Primary Examiner(s): Patel, Jitesh
• Application Number:
  US14/705,869
• Publication Number:
  US 20150302656A1
• Time in Patent Office: 1,273 Days
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. A method of displaying virtual or augmented reality, comprising:

capturing a first set of data at one or more first sensors in a first virtual or augmented reality display system corresponding to a first location;

capturing a second set of data at one or more second sensors in a second virtual or augmented reality display system corresponding to a second location;

receiving the first set of data and the second set of data from the first and second virtual or augmented reality display systems via one or more computer networks;

providing, by at least one processor, one or more timing or quality targets to at least a mapping module for allocating computational resources;

constructing or updating, by the at least one processor of a computing system, a map of a real world at least by stitching, into the map of the real world, a smaller world model including a first node representing the first location and a second node representing the second location in the map with an edge that is emphasized with a first emphasis and is associated with a connectivity strength that represents an extent of sharing between the first node and the second node;

determining, at the computing system, the first virtual or augmented reality system, or the second virtual or augmented reality system, one or more map points for the smaller world model at least by viewing one or more existing features with one or more new, virtual keyframes that are positioned in relation to the first location and the second location;

identifying a map point from the map, a maximum residual stress value for the map, and a bundle adjust process;

determining whether a stress value associated with the map point exceeds the maximum residual stress value;

adjusting the stress value associated with the map point by applying the bundle adjust process to the map point; and

identifying a topological map that corresponds to the map.

2. The method of claim 1, wherein the first virtual or augmented reality display system is a head-mounted virtual or augmented reality display system, the first set of data comprises images that are captured by the first augmented reality system and are pose tagged according to the first location.

3. The method of claim 1, wherein the first virtual or augmented reality display system is a room-based virtual or augmented reality display system, and the one or more new, virtual keyframes that are positioned in a most orthogonal direction in relation to the first location and the second location.

4. The method of claim 1, wherein the constructed map is transmitted to at least one of the first and second virtual or augmented reality display systems.

5. The method of claim 4, wherein a virtual object is projected to at least one of the first and second virtual or augmented reality display systems based at least in part on the constructed map of the real world.

6. The method of claim 1, wherein the captured data is at least an image captured at the first or second location.

7. The method of claim 1, wherein the captured data corresponds to sensor data.

8. The method of claim 1, further comprising extracting a set of map points from the data captured from the first and second virtual or augmented reality display systems, wherein the set of map points are used to construct the map of the real world.

9. The method of claim 1, wherein a part of the map corresponding to the first virtual or augmented reality display system is transmitted to the second virtual or augmented reality display system.

10. The method of claim 1, wherein the captured data comprises pose information of the first and second virtual or augmented reality display systems, and the map is constructed based at least in part on the pose information.

11. The method of claim 1, further comprising:

representing the first location as the first node;

representing the second location as the second node; and

connecting the first node and the second node with the edge in the map.

12. The method of claim 11, further comprising:

representing the edge with the first emphasis in the map; and

generating or updating the map using the first node, the second node, and the edge with the first emphasis.

13. The method of claim 12, further comprising representing different edges having different connectivity strengths among a plurality of nodes in the map with different emphases, wherein connectivity strength indicates proximity between the first node in the first location and the second node in the second location or between a first key frame corresponding to the first node or the second key frame corresponding to the second node in the map.

14. The method of claim 1, further comprising:

layering the topological map on the map that includes a plurality of map points comprising the map point associated with the stress value;

identifying the map point in the map that is misaligned with a corresponding point in the topological map;

retrieving one or more key frames or a set of additional points based in part or in whole upon the map point from the map or the corresponding point in the topological map; and

when the set of points is retrieved, applying the bundle adjust process to the set of additional points but not to all points in the map.

15. The method of claim 14 further comprising:

identifying a wave propagation bundle adjust process for the bundle adjust process;

identifying a first degree of separation for the wave propagation bundle adjust process;

radially pushing the stress value from the map point to at least one point in the set of additional points based in part or in whole upon the first degree of separation;

determining whether a stress level associated with the map point is to be further adjusted and generating determining results; and

radially pushing the stress level from the map point or at least one point in the set of additional points to one or more other points in the set of additional points based in part or in whole upon a second degree of separation.

16. The method of claim 15, further comprising:

determining whether to adjust the stress level associated with the point one or more map points of the map or one or more points in the set of additional points or in the one or more other points between the first degree of separation and the second degree of separation;

identifying an incremental bundle adjust process;

controlling wave increments based in part or in whole upon results of determining whether to adjust the stress level between the first degree of separation and the second degree of separation; and

applying the incremental bundle adjust process to at least a portion of the map including the point one or more map points of the map or one or more points in the set of additional points or in the one or more other points to reduce the errors.

17. A system for displaying augmented reality, comprising:

one or more image capturing devices configured to capture a first set of data at one or more first sensors in a first virtual or augmented reality display system corresponding to a first location;

the one or more image capturing devices further configured to capture a second set of data at one or more second sensors in a second virtual or augmented reality display system corresponding to a second location;

a processor operatively connected to a network component and configured to receive the first set of data and the second set of data from the first and second virtual or augmented reality display systems;

the processor further configured to provide one or more timing or quality targets to at least a mapping module for allocating computational resources;

the processor further configured to construct or update a map of a real world at least by stitching, into the map of the real world, a smaller world model including a first node representing the first location and a second node representing the second location in the map with an edge that is associated with connectivity strength that represents an extent of sharing between the first node and the second node;

the processor further configured to determine, at the computing system, the first virtual or augmented reality system, or the second virtual or augmented reality system, one or more map points for the smaller world model at least by viewing one or more existing features with one or more new, virtual keyframes that are positioned in relation to the first location and the second location;

the processor further configured to identify a map point from the map, a maximum residual stress value for the map, and a bundle adjust process;

the processor further configured to determine whether a stress value associated with the map point exceeds the maximum residual stress value;

the processor further configured to adjust the stress value associated with the map point by applying the bundle adjust process to the map point; and

the processor further configured to identify a topological map that corresponds to the map.