Recognizing objects in a passable world model in an augmented or virtual reality system

• US 9,767,616 B2
• Filed: 05/08/2015
• Issued: 09/19/2017
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US14/706,994
• Publication Number:
  US 20150302663A1
• Time in Patent Office: 865 Days
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. A method for recognizing objects with a virtual or augmented reality display system, comprising:

identifying map points in a world model that is projected to at least one eye of a user by a projector in a virtual or augmented reality display system, wherein the map points corresponds to one or more real objects and is used to determine a geometric structure of a location of the user;

recognizing, using at least one microprocessor, a real object of the one or more real objects in the world model presented by the virtual or augmented reality display system to the user at least by deriving a geometric structure of the real object using an image segmentation technique based in part or in whole upon the map points; and

displaying one or more virtual objects to the user relative to the real object, based at least in part on a location of the user.

2. The method of claim 1, further comprising:

determining a set of parameters corresponding to a movement of the user of the augmented reality system relative to the detected location;

calculating, based on the determined movement of the user, an anticipated position of the user; and

retrieving another data pertaining to the anticipated position of the user, wherein the other data pertaining to the anticipated position comprises map points corresponding to one or more real objects of the anticipated position.

3. The method of claim 1, wherein the map points corresponding to one or more real objects are used to construct a map of the real world.

4. The method of claim 1, further comprising recognizing one or more objects of the real world based in part or in whole on the map points.

5. The method of claim 1, wherein the map points are used to create a coordinate space of the real world, and wherein the one or more virtual objects are displayed based on the created coordinate space of the real world.

6. The method of claim 1, further comprising:

recognizing one or more objects of the real world based in part or in whole on the map points; and

displaying the virtual object based at least in part on a property of the one or more objects that have been recognized.

7. The method of claim 1, wherein the map points pertain to a geometry of the location.

8. The method of claim 1, further comprising downloading a set of map points pertaining to the location to the augmented reality display system.

9. The method of claim 1, further comprising uploading another set of map points from the augmented reality display system to a cloud resource.

10. The method of claim 1, recognizing the real object further comprising:

for

capturing, at one or more sensors of the virtual or augmented reality display system, a plurality of pose tagged images of surroundings of the user;

extracting a set of points from the plurality of pose tagged images;

segmenting, using at least one microprocessor, image data of a real object of the one or more real objects into segmented object features; and

correlating the segmented object features with the set of points.

11. The method of claim 10, further comprising:

deriving the geometric structure of using at least the segmented object features correlated with the set of points; and

recognizing a real object of the or more real objects using the geometric structure and an object recognizer of a plurality of object recognizers.

12. The method of claim 11, further comprising:

parameterizing the geometric structure of the real object into parametric primitive for the real object; and

attaching semantic information to the parametric primitive.

13. The method of claim 12, further comprising:

synchronizing the parametric primitive of the real object to the world model; and

estimating a future position of the real object in the world model using at least the semantic information, wherein the semantic information includes a taxonomical descriptor.

14. The method of claim 1, further comprising:

identifying a first real object of a first type;

recognizing the real object of the first type with a first object recognizer that recognizes the first type of object; and

recognizing a smaller subset of the real object of the first type with a second object recognizer.

15. The method of claim 1, identifying the map points comprising:

identifying two-dimensional (2D) map points captured with one or more respective sensors of multiple virtual or augmented reality display systems;

generating a geometric map comprising a keyframe that includes the map points, wherein the geometric map includes at least two nodes connected with an edge that corresponds to a number of shared map points between the at least two nodes; and

identifying a point of stress in the geometric map based in part or in whole upon information in a topological map, a discrepancy in a location of the keyframe, in relation to the geometric map, or a residual error of the geometric map.

16. The method of claim 15, identifying the map points comprising:

identifying a first node based in part or in whole upon the point of stress;

identifying a first wave of nodes having a single degree of separation from the first node; and

distributing the point of stress to the first wave of nodes.

17. The method of claim 1, further comprising:

identifying the map points pertaining to a real world environment;

determining a position of a plurality of keyframes that are used to determine the map points; and

determining at least one new map point based in part or in whole upon the map points and the position of the plurality of keyframes that are used to determine the map points.

18. The method of claim 17, further comprising:

rendering respective triangular cones from the position of the plurality of keyframes to at least some of the map points at least by placing a respective map point along a bisector of a corresponding triangular cone;

selectively shading the respective triangular cones such each bisector of a corresponding triangular cone is associated with a graphical emphasis in relation to a remaining portion of the corresponding triangular cone;

identifying a plurality of points of intersection between two intersecting triangular cones; and

identifying the at least one new map point based in part or in whole upon the graphical emphases corresponding to the plurality of points of intersection or upon a pixel pitch corresponding to the plurality of points of intersection.

19. The method of claim 17, further comprising:

placing a virtual keyframe in relation to the plurality of keyframes;

determining a relatively more orthogonal direction to the plurality of keyframes;

positioning the virtual keyframe along the relatively more orthogonal direction;

rendering respective lines from the virtual keyframe to the map points;

determining a plurality of points of intersections among the respective lines; and

determining the at least one new map point based in part or in whole upon the plurality of points of intersection rendered from the virtual keyframe to the map points.

20. A system for displaying augmented reality, comprising:

a virtual or augmented reality display system configured to identify map points in a world model that is presented to a user, wherein the map points corresponds to one or more real objects and is used to determine a geometric structure of a location of the user;

at least one microprocessor configured to recognize a real object of the one or more real objects in the world model presented by the virtual or augmented reality display system to the user at least by deriving a geometric structure of the real object using an image segmentation technique based in part or in whole upon the map points; and

one or more projectors configured to display one or more virtual objects to the user relative to the real object by projecting light beams pertaining to the one or more virtual objects to at least one eye of the user, based at least in part on a location of the user.