Triangulation of points using known points in augmented or virtual reality systems

• US 9,766,703 B2
• Filed: 05/08/2015
• Issued: 09/19/2017
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US14/706,996
• Publication Number:
  US 20150302665A1
• Time in Patent Office: 865 Days
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. A method of displaying augmented reality, comprising:

capturing a set of 2D points of a real-world environment at one or more image capturing devices of one or more augmented reality systems;

determining respective positions of the one or more image capturing devices for the set of 2D points;

determining a three-dimensional (3D) position of a 2D point of the set of 2D points based at least in part on the set of 2D points and the respective positions; and

generating, at an augmented reality system, a virtual content for the real-world environment by using at least the three-dimensional position that is shared via a computer network.

2. The method of claim 1, further comprising determining one or more respective poses of the one or more augmented reality systems.

3. The method of claim 2, wherein a pose of the one or more respective poses comprises a position or an orientation of an augmented reality system in relation to at least one 2D point in the set of 2D points.

4. The method of claim 1, further comprising:

segmenting one or more 2D images of the real-world environment into 2D segmented image features;

identifying one or more 3D nodes represented by respective 3D positions determined from the set of 2D points; and

determining a correlation between the segmented 2D segmented image features with the one or more 3D nodes.

5. The method of claim 1, further comprising determining respective depth information of one or more objects through at least one of the one or more augmented reality systems.

6. The method of claim 1, determining a pose of a separate augmented reality system based at least in part on the 3D position of the point.

7. The method of claim 4, further comprising;

deriving one or more object structures or one or more properties for one or more objects in the real-world environment based in part or in whole upon the correlation; and

constructing respective geometries of the one or more objects for the virtual content.

8. The method of claim 1, determining the three-dimensional position of the 2D point further comprising:

identifying a set of keyframes comprising the set of 2D points; and

identifying a set of keyframe positions respectively corresponding to the set of keyframes.

9. The method of claim 8, further comprising:

projecting a plurality of geometric entities from the set of keyframe positions of the set of keyframes to the set of 2D points; and

identifying an intersection between two geometric entities of the plurality of geometric entities as the three-dimensional position of the 2D point.

10. The method of claim 9, further comprising:

determining a relative likelihood of having a physical object at the 3D position based in part or in whole upon pixel coordinates of the 2D point.

11. The method of claim 9, projecting the plurality of geometric entities further comprising:

projecting a triangular cone from a keyframe position of the set of keyframe positions to the 2D point of the set of 2D points at least by bisecting the triangular cone with a line segment connecting the keyframe position and the 2D point.

12. The method of claim 11, further comprising:

shading an interior of triangular cone so that the line segment bisecting the triangular cone is rendered brighter.

13. The method of claim 12, further comprising:

identifying a resolution characteristic and lens calibration characteristic of an image capturing device of the one or more image capturing devices; and

configuring a buffer an image capturing device into a summing buffer so that a brighter spot represents a candidate location of a candidate 2D point.

14. The method of claim 9, further comprising:

reducing computational resource utilization at least by performing a process, the process comprising;

projecting the plurality of geometric entities from a number of unmatched 2D points in a number of prior keyframes into a first texture of a new keyframe; and

encoding a keyframe identifier and a feature identifier for the new keyframe.

15. The method of claim 14, further comprising:

identifying one or more 2D points from the new keyframe;

generating a second texture from the one or more 2D points from the new keyframe; and

masking the first texture with the second texture.

16. An apparatus for rendering augmented reality virtual contents, comprising:

one or more augmented reality systems comprising one or more image capturing devices, the one or more image capturing devices configured to capture a set of 2D points of a real-world environment;

a remote computing system configured to receive the set of 2D points, to determine respective positions of the one or more image capturing devices for the set of 2D points, to determine a three-dimensional (3D) position of a 2D point of the set of 2D points based at least in part on the set of 2D points and the respective positions, and to generate, at an augmented reality system, a virtual content for the real-world environment by using at least the three-dimensional position that is shared via a computer network.

17. The apparatus of claim 16, further comprising:

the remote computing system further configured to segment one or more 2D images of the real-world environment into 2D segmented image features, to identify one or more 3D nodes represented by respective 3D positions determined from the set of 2D points, and to determine a correlation between the segmented 2D segmented image features with the one or more 3D nodes.

18. The apparatus of claim 17, comprising:

the remote computing system further configured to derive one or more object structures or one or more properties for one or more objects in the real-world environment based in part or in whole upon the correlation, and to construct respective geometries of the one or more objects for the virtual content.

19. The apparatus of claim 16, further comprising:

the remote computing system further configured to identify a set of keyframes comprising the set of 2D points, to identify a set of keyframe positions respectively corresponding to the set of keyframes, to project a plurality of geometric entities from the set of keyframe positions of the set of keyframes to the set of 2D points, and to identify an intersection between two geometric entities of the plurality of geometric entities as the three-dimensional position of the 2D point.

20. The apparatus of claim 19, further comprising:

the remote computing system further configured to reduce computational resource utilization at least by projecting the plurality of geometric entities from a number of unmatched 2D points in a number of prior keyframes into a first texture of a new keyframe, identifying one or more 2D points from the new keyframe, generating a second texture from the one or more 2D points from the new keyframe, and masking the first texture with the second texture.