Using passable world model for augmented or virtual reality

• US 10,846,930 B2
• Filed: 05/07/2015
• Issued: 11/24/2020
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Wang, Jin Cheng
• Application Number:
  US14/705,980
• Publication Number:
  US 20150302657A1
• Time in Patent Office: 2,028 Days
• Field of Search: 345633
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

A method of displaying augmented reality comprises storing a passable world model data, wherein the passable world model data comprises sets of points respectively pertaining to real objects of the physical world, processing the passable world model data using each of one or more object recognizers dedicated to recognize a predetermined object based at least in part on a known geometry of the object corresponding to a set of points of the passable world model data, and displaying virtual content to a user of a particular augmented reality display system based at least in part on the recognized object.

Claims

1. A method of displaying augmented reality, comprising:

storing, into a passable world model, data that comprises (1) a set of points identifying a position and orientation of a real object, (2) a descriptor for the real object, and (3) a digital representation of a real space in a physical world in three-dimensional space, wherein

the digital representation of the real space is used to place virtual content in relation to physical coordinates in the physical world;

running a first object recognizer of a plurality of object recognizers on at least the set of points of the passable world model, wherein

the first object recognizer recognizes a type of objects for the real object in the physical world at least by processing the set of points based at least in part on a parametric geometry of the type of objects, without regard to a specific feature about the type of objects;

recognizing a specific object for the real object at least by processing, by the second object recognizer, the set of points that has been processed by the first object recognizer based at least in part upon the specific feature about the type of objects and a different object that pertains to the specific feature, wherein

the different object is of a different type that is different from the type of objects, and

the type of objects comprises multiple different specific objects including the specific object; and

displaying the virtual content to a user of an augmented reality display system based at least in part the specific object that has been recognized for the real object and the digital representation of the real space in the passable world model.

2. The method of claim 1, wherein the parametric geometry in the passable world model comprises a geometric definition of the type of objects corresponding to the physical world.

3. The method of claim 1, wherein the set of points is captured by a plurality of augmented reality display systems, and the plurality of augmented reality display systems capture data pertaining to a plurality of locations in the physical world.

4. The method of claim 1, wherein the set of points includes a 2D point.

5. The method of claim 1, wherein an object recognizer utilizes depth information to recognize a real-world object.

6. The method of claim 1, wherein the set of points corresponds to one or more images captured by one or more augmented reality display systems, and an object recognizer of the plurality of object recognizers identifies geometry of the real object relative to a known position of the augmented reality display system at least by scaling the parametric geometry of the type of objects.

7. The method of claim 1, wherein a new set of points identifying the position and the orientation of a new real object in the real space and a new descriptor of the new real object are synchronized to the passable world model stored on a remote system, wherein the new real object is recognized in the real space by the augmented reality display system based at least in part upon a different parametric geometry of a different type of objects.

8. The method of claim 1, wherein semantic information is attached to the parametric geometry of the type of objects, and at least one object recognizer of the plurality of object recognizers parameterizes geometry of the type of object into the parametric geometry so that the parametric geometry of the type of objects is characterized by being resizable.

9. The method of claim 8, wherein the semantic information comprises a taxonomical descriptor.

10. The method of claim 8, further comprising utilizing the semantic information to estimate a future position of the real object that has been recognized.

11. The method of claim 1, wherein the first or the second object recognizer receives sparse points collected from one or more images of the physical world.

12. The method of claim 1, wherein at least one object recognizer of the plurality of object recognizers outputs the parametric geometry of the type of objects at least by parameterizing a first object of the type of objects.

13. The method of claim 1, wherein each object recognizer of the plurality of object recognizers is dedicated to recognize a predetermined type of real objects.

14. The method of claim 1, wherein the augmented reality display system is autonomic, autonomous, and/or asynchronous to receive existing information of the real space from the passable world model stored on a remote system and to update the passable world model on the remote system with new information recognized by the augmented reality display system.

15. The method of claim 1, wherein the plurality of object recognizers simultaneously run on the passable world model data independent of each other to recognize different real objects in the real world.

16. The method of claim 1, wherein the passable world model includes the first object recognizer and the second object recognizer.

17. The method of claim 1, further comprising:

identifying a location of the user;

drawing a radius that communicates a position and an anticipated direction of the user;

retrieving a piece of the passable world model based at least in part upon the anticipated direction of the user; and

rendering a virtual world to the augmented reality display system of the user based at least in part upon the piece of the passable world.

18. The method of claim 1, further comprising:

disposing, by a first user wearing a first augmented reality display system, a virtual object in relation to same physical coordinates in the physical world; and

displaying the virtual object in relation to the same physical coordinates in the physical world in the augmented reality display system of the user based at least in part upon the digital representation of the real space.

19. The method of claim 1, further comprising:

dynamically estimating, by the first object recognizer or the second object recognizer on the augmented reality display system, a movement or a relative position of the real object in relation to another real object in the real space based at least in part upon the parametric geometry, rather than capturing images and determining the movement or the relative position of the real object by processing the images.

20. The method of claim 1, further comprising:

recognizing, by the second object recognizer, the specific object for the real object based at least in part upon an ontological relationship between the type of object and the different object.