Systems and methods for augmented and virtual reality

• US 10,262,462 B2
• Filed: 04/18/2015
• Issued: 04/16/2019
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A., Hand, Randall E., O'Connor, Philip, Chan, Richmond B., Greco, Paul M.
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Mathews, Crystal
• Application Number:
  US14/690,401
• Publication Number:
  US 20150302652A1
• Time in Patent Office: 1,459 Days
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. An augmented reality system, comprising:

at least one server comprising a processor and configured to;

receive first data corresponding to a first location and second data corresponding to a second location respectively from the first virtual or augmented reality display system and the second virtual or augmented reality display system through one or more networks communicably connecting a first and second virtual or augmented reality display systems to the at least one server, wherein

the first data corresponds to the first location at which the first virtual or augmented reality display system is located,

the second data corresponds to the second location at which the second virtual or augmented reality display system is located, and

the first virtual or augmented reality display system is different from the second virtual or augmented reality display system,

reduce the first location to a first node and the second location to a second node using at least the first data and the second data,

create a plurality of virtual keyframes based at least in part upon a plurality of existing keyframes, which have been captured by the first and second virtual or augmented reality display systems, at least by performing a plurality of analyses on corresponding normals of the plurality of existing keyframes, wherein a virtual keyframe is created to provide a new position to view a plurality of existing points or features;

determine a set of shared points from a first set of points and a second set of points that are respectively identified from the first data and the second data based at least in part upon performing one or more analyses using at least one or more corresponding virtual keyframes of the plurality of virtual keyframes,

construct at least a portion of a map of a real world at least by connecting the first node for the first location and the second node for the second location with an edge that is determined based in part or in whole upon a total number of shared points in the set of shared points;

identify, at the at least one server from a storage device, detailed environment data pertaining to the first location and virtual contents from the at least one server to the second virtual or augmented reality display system based at least in part upon an anticipated position of the second virtual or augmented reality display system.

2. The augmented reality system of claim 1, wherein the first augmented reality display system is a head-mounted augmented reality display system or a room-based sensor system, the first data comprises one or more pose tagged images corresponding to the first location, and the map is transmitted to at least one of the first and second augmented reality display systems.

3. The augmented reality display system of claim 2, wherein a virtual object is rendered in at least the portion of the real world by at least one of the first and second augmented reality display systems based at least in part on the at least the portion of the map of the real world.

4. The augmented reality display system of claim 1, further comprising determining the anticipated position based at least in part upon a current position and an intended direction or orientation pertaining to the second virtual or augmented reality display system, by localizing the second virtual or augmented reality display system using the at least the portion of the map, instead of using detailed parametric or geometric information about the anticipated location.

5. The augmented reality display system of claim 4, further comprising:

establishing a correspondence between the detailed environment data pertaining to a plurality of locations and a plurality of nodes comprising the first node and the second node by using the map and a world model.

6. The augmented reality display system of claim 5, further comprising:

identifying the detailed environment data from a storage device accessible by the at least one server based in part or in whole upon the correspondence and results of localizing the second virtual or augmented reality display system using the at least the portion of the map, rather than sorting through detailed data in a model representing the real world.

7. The augmented reality display system of claim 1, wherein

the first data or the second data includes at least an image captured at the first or second location, or the first data or the second data corresponds to sensor data,

the processor extracts a set of map points from the first and second data captured from the first and second augmented reality display systems, and the set of map points is used to construct the at least the portion of the map of the real world, or

a part of the map corresponding to the first augmented reality display system is transmitted to the second augmented reality display system.

8. The augmented reality display system of claim 1, wherein the at least one server configured to create the plurality of keyframes is further configured to:

perform a first analysis of the plurality of analyses, wherein performance of the first analysis determines a first direction relative to the plurality of existing keyframes for the plurality of virtual keyframes;

identifying a first virtual keyframe from the one or more corresponding virtual keyframes;

position the first virtual keyframe based at least in part on the first direction;

render a plurality of geometric entities from the first virtual keyframe to the plurality of existing points or features including the first set of shared points and the second set of points;

determine one or more new points based at least in part upon one or more characteristics of the geometric entities; and

add the one or more new points to the map of the real world.

9. The augmented reality display system of claim 8, wherein the one or more characteristics of the geometric entities comprise a plurality of points of intersection of the plurality of geometric entities or a pitch pixel that corresponds to the plurality of points of intersection of the plurality of geometric entities.

10. The augmented reality display system of claim 1, wherein the first and second data comprises pose information of the first and second augmented reality display systems, wherein the at least the portion of the map is constructed based at least in part on the pose information.

11. The augmented reality display system of claim 1, the server further configured to:

receive the first data captured by the first virtual or augmented reality display system;

associate first parametric geometry with the first node in the map, wherein the first parametric geometry is stored separately from the map;

receive the second data captured by the second virtual or augmented reality display system; and

associate second parametric geometry with the second node in the map, wherein the second parametric geometry is stored separately from the map.

12. The augmented reality display system of claim 11, the server further configured to:

generate a first signature for the first node using at least the first data; and

generate a second signature for the second node using at least the second data.

13. The augmented reality display system of claim 12, the server further configured to:

receive third data from a third virtual or augmented reality display system;

generate a third signature based in part or in whole upon the third data;

identify a repository of a plurality of signatures; and

determining whether the third signature matches at least one signature in the repository.

14. The augmented reality display system of claim 13, the server further configured to:

generate a third node and add the third node to the map when the third signature is determined not to match the at least one signature in the repository; and

identify a corresponding location associated with the at least one signature when the third signature is determined to match the at least one signature in the repository; and

determine a third location for the third virtual or augmented reality display system to be the corresponding location associated with the at least one signature when the third signature is determined to match the at least one signature in the repository.

15. The augmented reality display system of claim 12, further comprising:

one or more first sensors of the first virtual or augmented reality display system configured to capture first additional information pertaining to one or more first real world locations of the first virtual or augmented reality display system at the first location; and

one or more sensor sensors of the second virtual or augmented reality display system configured to capture second additional information pertaining to one or more second real world locations of the second virtual or augmented reality display system at the second location.

16. The augmented reality display system of claim 15, the server further configured to:

hashing the first additional information with the first data into first hashed data;

hashing the second additional information with the second data into second hashed data,

wherein

the first or second additional information comprises at least one of global positioning system (GPS) location information, Wi-Fi location information, received signal strength information, one or more histograms generated from one or more images captured by the first or second virtual or augmented reality system at the first or the second location, or object information of one or more objects around the first or second virtual or augmented reality display system.

17. The augmented reality display system of claim 16, the server further configured to:

identifying a first real world location of a user at the first location using the first hashed data, without understanding or having to understand or processing all geometries associated with the one or more first real world location.

18. The augmented reality display system of claim 16, the first virtual or augmented reality display system further configured to:

present one or more virtual objects to the user in relation to one or more real world objects at the actual real world location by using at least the map.

19. The augmented reality display system of claim 11, the server further configured to:

identify a relation between the first node and the second node; and

represent the relation at least by generating a first edge that connects the first node and the second node in the map.

20. The augmented reality display system of claim 11, the server further configured to:

identify a geometric map or geometric configuration of the first location;

overlaying the map on the geometric map or the geometric configuration; and

identifying loop-closure stresses in the geometric map or the geometric configuration based in part or in whole upon alignment information between the map and the geometric map or the geometric configuration.