Finding new points by render rather than search in augmented or virtual reality systems

• US 10,109,108 B2
• Filed: 05/07/2015
• Issued: 10/23/2018
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A., Hand, Randall E.
• Primary Examiner(s): Merouan, Abderrahim
• Application Number:
  US14/705,990
• Publication Number:
  US 20150356784A1
• Time in Patent Office: 1,265 Days
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

A method of displaying augmented reality comprises capturing a set of map points pertaining to the real world, wherein the set of map points are captured through a plurality of augmented reality systems, determining a position of a plurality of keyframes that captured the set of map points, and determining a set of new map points based at least in part on the captured set of map points and the determined position of the plurality of keyframes.

Claims

1. A method of displaying augmented reality, comprising:

capturing a set of map points pertaining to the real world, wherein the set of map points are captured through a plurality of augmented reality systems;

determining positions of a plurality of keyframes that captured the set of map points;

rendering lines from the determined positions of the plurality of keyframes to respective map points captured from the plurality of keyframes;

identifying points of intersection between the rendered lines; and

determining a set of new map points based at least in part on the identified points of intersection.

2. The method of claim 1, wherein the set of new map points are determined based at least in part on a pixel pitch corresponding to the identified points of intersection.

3. The method of claim 1, wherein the set of new map points are added to a map of the real world.

4. The method of claim 3, further comprising delivering virtual content to one or more augmented reality display systems based at least in part on the map of the real world.

5. The method of claim 3, further comprising delivering virtual content to one or more augmented reality display systems based at least in part on the map of the real world.

6. A method of displaying augmented reality, comprising:

capturing a set of map points pertaining to the real world, wherein the set of map points are captured through a plurality of augmented reality systems;

determining positions of a plurality of keyframes that captured the set of map points;

rendering triangular cones from the determined positions of the plurality of keyframes to respective map points captured from the plurality of keyframes, wherein the captured map points lie on bisectors of the triangular cones; and

determining a set of new map points based at least in part on the rendered triangular cones.

7. The method of claim 6, further comprising selectively shading the triangular cones such that the bisectors of the triangular cones are the brightest portions of the triangular cones.

8. The method of claim 7, further comprising identifying points of intersection between at least two rendered triangular cones, wherein the set of new map points are based at least in part on the identified points of intersection.

9. The method of claim 8, wherein the set of new map points are determined based at least in part on the brightness of the identified points of intersection.

10. The method of claim 8, wherein the set of new map points are determined based at least in part on a pixel pitch corresponding to the identified points of intersection.

11. The method of claim 6, wherein the set of new map points are added to a map of the real world.

12. A method of displaying augmented reality, comprising:

capturing a set of map points pertaining to the real world, wherein the set of map points are captured through a plurality of augmented reality systems;

determining positions of a plurality of keyframes that captured the set of map points;

placing a virtual keyframe in relation to an existing set of keyframes;

determining a set of new map points based at least in part on the virtual keyframe; and

determining a most orthogonal direction to the existing set of keyframes, and positioning the virtual keyframe at the determined orthogonal direction.

13. The method of claim 12, wherein the most orthogonal direction is determined along one of an x coordinate, a y coordinate, and a z coordinate.

14. The method of claim 12, further comprising rendering lines from the virtual keyframe to the set of map points, and determining the new map points based at least in part on one or more points of intersection of the rendered lines.

15. The method of claim 14, further comprising applying a summing buffer to determine the points of intersection.

16. The method of claim 12, further comprising rendering triangular cones from the virtual keyframe to the set of map points, and determining the new map points based at least in part on one or more points of intersection.

17. The method of claim 16, further comprising performing a bundle adjust to correct a location of a new map point of the set of new map points.

18. The method of claim 12, wherein the set of new map points are added to a map of the real world.

19. The method of claim 18, further comprising delivering virtual content to one or more augmented reality display systems based at least in part on the map of the real world.