Reducing stresses in the passable world model in augmented or virtual reality systems

• US 10,665,018 B2
• Filed: 05/07/2015
• Issued: 05/26/2020
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Mathews, Crystal
• Application Number:
  US14/705,986
• Publication Number:
  US 20170061688A1
• Time in Patent Office: 1,846 Days
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. A method of displaying augmented reality, comprising:

capturing one or more sets of map points from the real world through a plurality of augmented reality systems, wherein the one or more sets of map points include a map point that corresponds to a real world object;

determining parameterized geometry of the real world object at least by deriving, via a least two-dimensional and three-dimensional data fusion on at least a set of sparse points, a structure of and one or more properties about a parametric primitive that represents the real world object with an adjustable size and at least by parameterizing geometry of the real world object into the parameterized geometry that includes parametric geometry, wherein the parametric geometry includes geometric definitions and polygonal definitions of the real world object;

constructing a geometric map of the real world based at least in part on the one or more sets of map points and the parameterized geometry of the real world object;

estimating, at one or more object recognizers of a second augmented reality system, a future position, a dynamic movement, or a dynamic orientation of the real world object with at least the parameterized geometry and semantic information attached to the parametric primitive, rather than via repeatedly captures of movements and orientations of the real world object; and

displaying, at the second augmented reality system, a virtual object that represents the real world object at the future position, the dynamic movement, or the dynamic orientation estimated by the one or more object recognizers.

2. The method of claim 1, further comprising identifying a point of stress in the geometric map, wherein a strength of a connection between two nodes of the geometric map corresponds to a number of shared map points between the two nodes.

3. The method of claim 2, wherein the point of stress is identified based at least in part on information retrieved from a topological map.

4. The method of claim 3, further comprising:

determining loop closure stress at one or more map points using at least the topological map.

5. The method of claim 4, further comprising:

layering the topological map on the geometric map by layering topological map nodes of the topological map on top of geometric map nodes in the geometric map;

determining whether two geometric map nodes in the geometric map are to be a same geometric map node at least by examining one or more corresponding topological map nodes; and

identifying the loop closure stress in the geometric map when a single topological map node corresponds to the two geometric map nodes.

6. The method of claim 4, further comprising:

identifying a first physical space in real world;

creating a first fingerprint or signature of a first physical space in real world at least by processing first data captured by a first augmented reality system of the plurality of augmented reality systems, rather than using positioning system location data or pose tagged images;

creating the topological map at least by representing the first physical space in the real world as a first point node in the topological map and associating the first fingerprint or signature with the first point node; and

updating the topological map at least by adding a second point node to the topological map and connecting the first point node and the second point node with an edge in the topological map, wherein the edge represents connectivity between the first point node and the second point node.

7. The method of claim 6, further comprising:

determining whether a second augmented reality system is in the first physical space at least by processing second data captured by the second augmented reality system into a second fingerprint or signature and by comparing the second fingerprint or signature to the first fingerprint or signature associated with the first point node in the topological map; and

when the second augmented reality system is determined to be in the first physical space, retrieving one or more keyframes associated with the first point node and transmitting the one or more keyframes to the second augmented reality system via a network.

8. The method of claim 2, wherein the point of stress is identified based at least in part on a discrepancy in a location of a particular keyframe in relation to the geometric map.

9. The method of claim 2, wherein the point of stress is identified based at least in part on a maximum residual error of the geometric map.

10. The method of claim 2, wherein the point of stress is distributed through a bundle adjust process.

11. The method of claim 10, further comprising:

distributing a stress at the point of stress to an area between two degrees of separation and three degrees of separation in the geometric map with respect to the point of stress.

12. The method of claim 2, wherein a stress associated with the point of stress is radially distributed to a first wave of nodes outside a closest node that is closest to the point of stress.

13. The method of claim 12, wherein the first wave of nodes outside of the closest node comprises a network or nodes that has a single degree of separation from the closest node that is closest to the point of stress.

14. The method of claim 12, wherein the stress associated with point of stress is further radially distributed to second wave of nodes outside the first wave of nodes.

15. The method of claim 12, wherein the first wave of nodes are marked when the stress is radially distributed to the first wave of nodes.

16. The method of claim 1, further comprising:

identifying a plurality of keyframes including the keyframe, wherein the semantic information comprises one or more taxonomical descriptors for the real world object;

representing the plurality of keyframes as a plurality of nodes that includes the node; and

determining whether two nodes in the plurality of nodes are connected based in part or in whole upon whether there is at least one shared augmented reality device in common.

17. The method of claim 16, further comprising:

determining an edge between the two nodes based in part or in whole upon a total number of shared map points between the two nodes that are determined to be connected; and

constructing the geometric map at least by inserting the plurality of keyframes as the plurality of nodes in the geometric map and connecting the plurality of nodes with the at least the edge between the two nodes.

18. The method of claim 17,

capturing a plurality of sets of two-dimensional map points from the real world through the plurality of augmented reality systems;

and

determining a set of three-dimensional (3D) map points based in part or in whole upon the plurality of sets of two-dimensional map points, wherein

a first keyframe corresponding to a first node in the geometric map represents a first room in a real world, and a second keyframe corresponding to a second node in the geometric map represents a second room in the real world.

19. The method of claim 17, further comprising:

representing a first edge connecting a first pair of nodes in the geometric map with a first graphical emphasis; and

representing a second edge connecting a second pair of nodes in the geometric map with a second graphical emphasis.

20. The method of claim 17, further comprising:

determining a strength of the edge based in part or in whole upon geometric proximity between two real world locations represented by the two nodes in the geometric map.