Estimating pose in 3D space

• US 10,163,011 B2
• Filed: 05/17/2017
• Issued: 12/25/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Kaehler, Adrian, Bradski, Gary
• Primary Examiner(s): Sheng, Xin
• Application Number:
  US15/597,694
• Publication Number:
  US 20180005034A1
• Time in Patent Office: 587 Days
• Field of Search: None
• CPC Class Codes: G01S 3/00 Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

Methods and devices for estimating position of a device within a 3D environment are described. Embodiments of the methods include sequentially receiving multiple image segments forming an image representing a field of view (FOV) comprising a portion of the environment. The image includes multiple sparse points that are identifiable based in part on a corresponding subset of image segments of the multiple image segments. The method also includes sequentially identifying one or more sparse points of the multiple sparse points when each subset of image segments corresponding to the one or more sparse points is received and estimating a position of the device in the environment based on the identified the one or more sparse points.

Claims

1. An imaging system comprising:

an image capture device comprising a lens and an image sensor, the lens configured to direct light from an environment surrounding the image capture device to the image sensor, the image sensor configured to;

sequentially capture a first plurality of image segments of an image based on the light from the environment, the image representing a field of view (FOV) of the image capture device, the FOV comprising a portion of the environment and including a plurality of sparse points, and

sequentially capture a second plurality of image segments, the second plurality of image segments captured after the first plurality of image segments and forming at least another portion of the image;

non-transitory data storage configured to sequentially receive the first and second plurality of image segments from the image sensor and store instructions for estimating at least one of a position and orientation of the image capture device within the environment; and

at least one hardware processor operably coupled to the non-transitory data storage and configured by the instructions to;

identify a first group of sparse points based in part on a corresponding subset of the first plurality of image segments, the first group of sparse points identified as the first plurality of image segments are received at the non-transitory data storage,

determine at least one of the position and orientation of the imaging device within the environment based on the first group of sparse points,

identify a second group of sparse points based in part on a corresponding subset of the second plurality of image segments, the second group of sparse points identified as the second plurality of image segments are received at the non-transitory data storage, and

update the at least one of the position and orientation of the imaging device within the environment based on the first and second group of sparse points.

2. The imaging system of claim 1, wherein the image sensor is a rolling shutter image sensor.

3. The imaging system of claim 1, wherein the non-transitory data storage comprises non-transitory buffer storage configured to sequentially receive the first and second plurality of image segments as the image segments are captured by the image sensor, the non-transitory buffer storage having a storage capacity based at least partly on a number of image segments included in each subset of image segments.

4. The imaging system of claim 1, wherein the first group of sparse points or the second group of sparse points comprises a number of sparse points between 10 and 20 sparse points.

5. The imaging system of claim 1, wherein the hardware processor is configured to update the at least one of the position and orientation of the image capture device based on a number of the most recently identified sparse points, wherein the most recently identified sparse points comprises at least one of the first group of sparse points, the second group of sparse points, and one or more of the first and the second group of sparse points.

6. The imaging system of claim 5, wherein the number of the most recently identified sparse points is equal to a number of sparse points in the first group of sparse points.

7. The imaging system of claim 1, wherein the hardware processor is configured to perform a visual simultaneous localization and mapping (V-SLAM) algorithm.

8. The imaging system of claim 1, wherein the plurality of sparse points are identified based on at least one of a real-world object, a virtual image element, and a non-visible indicator projected into the environment.

9. A head mounted display (HMD) configured to be worn on the head of a user, the HMD comprising:

a frame;

a display supported by the frame and disposed forward of an eye of the user;

an outward facing image capture device disposed on the frame and comprising a lens and an image sensor, the lens configured to direct light from an environment surrounding the HMD to the image sensor and the image sensor configured to sequentially capture a plurality of image segments of an image based on the light from the environment, the image representing a field of view (FOV) of the outward facing image capture device, the FOV comprising a portion of an environment and including a plurality of sparse points, wherein each sparse point is identifiable in part based on a corresponding subset of the plurality of image segments;

non-transitory data storage configured to sequentially receive the plurality of image segments from the image sensor and store instructions for estimating at least one of a position and orientation of the HMD within the environment; and

at least one hardware processor operably coupled to the non-transitory data storage and configured by the instructions to;

sequentially identify one or more sparse points of the plurality of sparse points when each subset of image segments corresponding to the one or more sparse points is received at the non-transitory data storage, and

estimate the at least one of the position and orientation of the HMD within the environment based on the identified the one or more sparse points.

10. The HMD of claim 9, wherein the non-transitory data storage comprises a circular or rolling buffer.

11. The HMD of claim 9, wherein the plurality of image segments comprises at least a first plurality of image segments and a second plurality of image segments, and wherein the image sensor is configured to sequentially transmit the first and second image segments to the non-transitory data storage.

12. The HMD of claim 11, wherein the hardware processor is configured to:

sequentially identify a first group of one or more sparse points when the first plurality of image segments corresponding to the one or more sparse points of the first group is received; and

sequentially identify a second group of one or more sparse points when the second plurality of image segments corresponding to the one or more sparse points of the second group is received,

wherein the second plurality of image segments is received after the first plurality of image segments.

13. The HMD of claim 12, wherein the hardware processor is configured to estimate the at least one of the position and orientation of the HMD based on the identified first group of one or more sparse points.

14. The HMD of claim 13, wherein the first group of sparse points or the second group of sparse points comprises a number of sparse points between 2 and 20.

15. The HMD of claim 13, wherein the first group of sparse points or the second group of comprises a number of sparse points between 10 and 20.

16. The HMD of claim 13, wherein the hardware processor is further configured to update the at least one of the position and orientation of the HMD based on the identified second group of one or more sparse points.

17. The HMD of claim 9, wherein the hardware processor is further configured to update the at least one of the position and orientation of the HMD when a number of the sequentially identified one or more sparse points are identified.

18. The HMD of claim 17, wherein the number of sequentially identified one or more sparse points comprises at least one of the sparse points of the first group of one or more sparse points.

19. The HMD of claim 9, wherein the plurality of sparse points are identified based on at least one of a real-world object, a virtual image element, and a non-visible indicator projected into the environment.

20. The HMD of claim 9, wherein the hardware processor is further configured to:

extract the sequentially identified one or more sparse points from the corresponding subset of the plurality of image segments; and

perform a visual simultaneous location mapping (VSLAM) algorithm on the sequentially identified one or more sparse points to estimate the at least one of the position and orientation of the image capture device.