Stress reduction in geometric maps of passable world model in augmented or virtual reality systems

• US 9,984,506 B2
• Filed: 05/07/2015
• Issued: 05/29/2018
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Wang, Jin-Cheng
• Application Number:
  US14/705,989
• Publication Number:
  US 20150302643A1
• Time in Patent Office: 1,118 Days
• Field of Search: 345633, 345419, 345440, 348 50, 382103, 382153
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. An augmented or virtual reality system, comprising:

a set of map points captured in a plurality of existing keyframes from a real world by a set of augmented or virtual reality display systems;

a data structure to receive or store the set of map points captured from the real world;

a processor communicatively coupled to the data structure to construct or update a geometric map of the real world based at least in part on the set of map points, wherein

a node in the geometric map comprises a keyframe that captures at least a subset of map points of the set of map points, and

a strength of a connection between two nodes in the geometric map corresponds to a number of map points shared between the two nodes;

the processor further configured to generate or update a simplified map corresponding to the geometric map at least by representing the real world with a plurality of point nodes corresponding to the plurality of existing keyframes and connecting the plurality of point nodes with a set of edges;

the processor further configured to determine and position a virtual keyframe in relation to a normal direction of at least one existing keyframe of the plurality of existing keyframes, and

a head worn or mounted display coupled to the processor, comprising a waveguide, and configured to project light beams to at least one eye of a user based at least in part upon the geometric map, the simplified map, and the virtual keyframe that is determined and positioned in relation to a normal direction of at least one existing keyframe of the plurality of existing keyframes.

2. The augmented or virtual reality system of claim 1, wherein the processor is configured to determine one or more new map points at least by rendering rays or cones from the virtual keyframe to at least some map points in the set of map points based in part or in whole upon one or more points of intersections formed by the rays or cones.

3. The augmented or virtual reality system of claim 2, wherein the processor is further configured to identify a point of stress in the geometric map, and the point of stress is identified based at least in part on information retrieved from a topological map.

4. The augmented or virtual reality system of claim 2, wherein the point of stress is identified based at least in part on a discrepancy in a location of a particular keyframe in relation to the geometric map.

5. The augmented or virtual reality system of claim 2, wherein the point of stress is identified based on a maximum residual error of the geometric map.

6. The augmented or virtual reality system of claim 2, wherein a stress associated with the point of stress is distributed through a bundle adjust process.

7. The augmented or virtual reality system of claim 2, wherein a stress associated with the identified point of stress is radially distributed to a first wave of nodes outside the node closest to the point of stress.

8. The augmented or virtual reality system of claim 7, wherein the first wave of nodes outside of the node comprises a network or nodes that have a single degree of separation from the node closest to the point of stress.

9. The augmented or virtual reality system of claim 7, wherein a stress associated with the point of stress is further radially distributed to second wave of nodes outside the first wave of nodes.

10. The augmented or virtual reality system of claim 7, wherein the first wave of nodes are marked when the stress is radially distributed to the first wave of nodes.

11. The augmented or virtual reality system of claim 1, further comprising:

a first positioning system that receives first positioning data of a first augmented or virtual reality system, the first positioning data indicative of a first location of the first augmented or virtual reality system in the real world; and

a set of pose tagged images comprising pose information of the set of augmented or virtual reality system and corresponding to the plurality of existing keyframes.

12. The augmented or virtual reality system of claim 11, further comprising:

the processor further configured to generate or update first finer granular positioning data corresponding to the first location for the simplified map at least by presenting only respective fingerprints of two physical spaces represented as two corresponding point nodes in the simplified map and a relationship between the two physical spaces.

13. The augmented or virtual reality system of claim 12, wherein the first positioning data is less granular or precise than the first finer granular positioning data, and the first finer granular positioning data is not achievable through the first positioning data that is provided by the first positioning system and includes first coordinate information of the first location in the real world.

14. The augmented or virtual reality system of claim 13, further comprising:

the processor further configured to identify parametric geometry data comprising raster imagery, point clouds, descriptor clouds, or geometric definitions of features in the real world; and

a passable world model that comprises the parametric geometry data, wherein a smaller portion of the passable world model is transmitted from a remote computing system through a computer network to the first augmented or virtual reality system by using the simplified map based in part or in whole upon information captured by the first augmented or virtual reality system and transmitted to the remote computing system.

15. The augmented or virtual reality system of claim 14, further comprising:

a second augmented or virtual reality system in the set of augmented or virtual reality systems;

a second positioning system that receives second positioning data of the second augmented or virtual reality system, the second positioning data indicative of a second location of the second augmented or virtual reality system in the real world;

the processor further configured to generate or update second finer granular positioning data corresponding to the second location for the simplified map;

the first augmented or virtual reality system configured to transmit at least a part of the passable world model to the second augmented or virtual reality system; and

the second augmented or virtual reality system configured to receive the at least the part of the passable world model from the first augmented or virtual reality system.

16. The augmented or virtual reality system of claim 15, wherein the second augmented or virtual reality system is first positioned in the real world by using the simplified map before the parametric geometry data including the geometric definitions associated with the at least the part of the passable world model is transmitted to the second augmented or virtual reality system.

17. The augmented or virtual reality system of claim 1, wherein an edge connecting two point nodes in the simplified map represents a shared augmented or virtual reality system or a shared user between the two point nodes.

18. The augmented or virtual reality system of claim 17, wherein an edge connecting two point nodes in the simplified map does not represent geographical connectivity in the real world between the two point nodes.

19. The augmented or virtual reality system of claim 1, further comprising:

the processor further configured to identify loop-closure stresses in the geometric map or geometric configurations of one or more physical spaces in the real world based in part or in whole upon the simplified map.

20. The augmented or virtual reality system of claim 19, further comprising:

the processor further configured to layer the simplified map on the geometric map;

the processor further configured to identify a first set of one or more map points for the geometric map from a first keyframe, which is captured by the augmented or virtual reality system at a first time instant, of the plurality of existing keyframes;

the processor further configured to determine first positioning information of the first keyframe;

the processor further configured to identify a second keyframe from the plurality of existing keyframes;

the processor further configured to determine that the second keyframe and the first keyframe represent a same physical space based at least in part upon the simplified map;

the processor further configured to identify a second set of one or more map points for the geometric map from the second keyframe, which is captured by the augmented or virtual reality system at a second time instant, of the plurality of existing keyframes;

the processor further configured to identify the loop-closure stresses when the first set of one or more map points are misaligned with the second set of one or more map points.