Systems and methods for augmented reality preparation, processing, and application

• US 9,754,419 B2
• Filed: 11/13/2015
• Issued: 09/05/2017
• Est. Priority Date: 11/16/2014
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Eonite Perception, Inc. (Intel Corporation)
• Inventors: Petrovskaya, Anna, Varvak, Peter
• Primary Examiner(s): Hoang, Peter
• Application Number:
  US14/941,448
• Publication Number:
  US 20160148433A1
• Time in Patent Office: 662 Days
• Field of Search: None
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 7/36 using image sharpness techniques , e.g. image processing techniques for generating autofocus signals image data processing per se G06T; in cameras having a solid state image sensor H04N23/67
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Various of the disclosed embodiments provide systems and methods for acquiring and applying a depth determination of an environment in e.g., various augmented reality applications. A user may passively or actively scan a device (e.g., a tablet device, a mobile phone device, etc.) about the environment acquiring depth data for various regions. The system may integrate these scans into an internal three-dimensional model. This model may then be used in conjunction with subsequent data acquisitions to determine a device'"'"'s location and orientation within the environment with high fidelity. In some embodiments, these determinations may be accomplished in real-time or near-real-time. Using the high-fidelity orientation and position determination, various augmented reality applications may then be possible using the same device used to acquire the depth data or a new device.

Claims

1. A device to facilitate an augmented reality (AR) experience, the device comprising:

a depth sensor;

at least one processor;

at least one memory comprising instructions configured to cause the at least one processor to perform a method comprising;

receiving depth information, the depth information comprising distances from the depth sensor to surfaces in a real-world environment;

determining a final pose, wherein determining the final pose comprises;

determining a first pose;

determining at least one neighborhood based upon the first pose; and

iteratively, through two or more iterations;

selecting a new pose from a plurality of poses within a neighborhood of a plurality of neighborhoods, each neighborhood of the plurality of neighborhoods covering a portion of a search space associated with a virtual model of the real-world environment;

pruning a neighborhood from the plurality of neighborhoods, at least in part, by determining correspondences between the depth information oriented using a pose from the pruned neighborhood and at least a portion of the virtual model of the real-world environment; and

setting a size of at least one neighborhood in a subsequent iteration to be smaller than a size of at least one neighborhood in a preceding iteration, wherein

the final pose is the new pose so iteratively determined to more closely correspond to the virtual model than a pose of a previous iteration; and

making the final pose available to an AR application.

2. The device of claim 1, wherein the first pose is a predicted pose associated with a belief determined based, at least in part, upon the depth information.

3. The device of claim 2, further comprising an inertial measurement unit (IMU), and wherein the predicted pose is determined, at least in part, based upon IMU data acquired in conjunction with the acquisition of the depth information, wherein the device applies one or more offsets to relate a clock of the at least one processor, a clock of the depth sensor, and a clock of the IMU, to a common reference at substantially the time when the IMU data is acquired.

4. The device of claim 1, wherein the iterations cease when successively determined poses resemble one another within a threshold, the threshold measured in meters-and-radians (MARs).

5. The device of claim 1, wherein the virtual model comprises a Likelihood Field-Float (LFF) model or a Likelihood Field-Integer (LFI) model.

6. The device of claim 1, wherein determining a final pose based upon the correspondences comprises applying a Pose Search Algorithm (PSA).

7. The device of claim 6, wherein the PSA comprises a Scaling Series algorithm.

8. The device of claim 7, further comprising an inertial measurement unit (IMU), wherein the first pose is a predicted pose, the predicted pose based upon inertial measurement unit (IMU) data acquired in conjunction with the acquisition of the depth information.

9. The device of claim 2, further comprising:

an inertial measurement unit (IMU), and wherein

the two or more iterations comprise operations in a Scaling Series algorithm, the Scaling Series Algorithm further comprising a division of a state space corresponding to the virtual model into a plurality of cells in accordance with either a grid hash or a grid tree,

selecting a new pose from a plurality of poses comprises selecting a new pose based upon the grid hash or the grid tree,

the iterations cease when successively determined poses resemble one another within a threshold, the threshold measured in meters-and-radians (MARs),

the virtual model comprises a Likelihood Field-Float (LFF) model or a Likelihood Field- Integer (LFI) model,

the predicted pose is determined, at least in part, based upon IMU data acquired from the IMU in conjunction with the acquisition of the depth information,

the device is further configured to apply one or more offsets to relate a clock of the at least one processor, a clock of the depth sensor, and a clock of the IMU, to a common reference at substantially the time when the IMU data is acquired, and wherein

the belief comprises a distribution of probabilities across a corpus of candidate pose transforms determined based, at least in part, upon the depth information.

10. The device of claim 5, wherein the virtual model is a LFF Model when performing a global localization and an LFI Model when performing a standard tracking operation.

11. A device to determine a final pose, the device comprising:

an interface configured to receive data from an augmented reality (AR) experience device;

at least one processor;

at least one memory comprising instructions configured to cause the at least one processor to perform a method comprising;

receiving depth information from the AR experience device via the interface, the depth information comprising distances from a depth sensor to surfaces in a real-world environment;

determining a final pose, wherein determining the final pose comprises;

determining a first pose;

determining at least one neighborhood based upon the first pose; and

iteratively, through two or more iterations;

selecting a new pose from a plurality of poses within a neighborhood of a plurality of neighborhoods, each neighborhood of the plurality of neighborhoods covering a portion of a search space associated with a virtual model of the real-world environment;

pruning a neighborhood from the plurality of neighborhoods, at least in part, by determining correspondences between the depth information oriented using a pose from the pruned neighborhood and at least a portion of the virtual model of the real-world environment; and

setting a size of at least one neighborhood in a subsequent iteration to be smaller than a size of at least one neighborhood in a preceding iteration, wherein

the final pose is the new pose so iteratively determined to more closely correspond to the virtual model than a pose of a previous iteration; and

making the final pose available to the AR experience device.

12. The device of claim 11, wherein the first pose is a predicted pose associated with a belief determined based, at least in part, upon the depth information.

13. The device of claim 11, wherein the iterations cease when successively determined poses resemble one another within a threshold, the threshold measured in meters-and-radians (MARs).

14. The device of claim 11, wherein determining a final pose based upon the correspondences comprises applying a Pose Search Algorithm (PSA).

15. The device of claim 14, wherein the PSA comprises a Scaling Series algorithm.

16. The non-transitory computer-readable medium of claim 11, wherein the interface is a network interface.

17. A computer-implemented method for determining a pose of a device relative to a real-world environment, comprising:

receiving depth information associated with the device, the depth information comprising distances from the device to surfaces in a real-world environment; and

determining a final pose, wherein determining the final pose comprises applying a Pose Search Algorithm (PSA) to a virtual model of the real-world environment and a first pose determined from the depth information, wherein the PSA comprises;

determining at least one neighborhood based upon the first pose; and

iteratively, through two or more iterations;

selecting a new pose from a plurality of poses within a neighborhood of a plurality of neighborhoods, each neighborhood of the plurality of neighborhoods covering a portion of a search space associated with the virtual model of the real-world environment;

pruning a neighborhood from the plurality of neighborhoods, at least in part, by determining correspondences between the depth information oriented using a pose from the pruned neighborhood and at least a portion of the virtual model of the real-world environment; and

setting a size of at least one neighborhood in a subsequent iteration to be smaller than a size of at least one neighborhood in a preceding iteration, and

wherein

the final pose is the new pose so iteratively determined to more closely correspond to the virtual model than a pose of a previous iteration.

18. The computer-implemented method of claim 17, wherein the first pose is a predicted pose associated with a belief determined based, at least in part, upon the depth information and upon inertial measurement unit (IMU) data acquired in conjunction with the acquisition of the depth information.

19. The computer-implemented method of claim 17, wherein the PSA comprises a Scaling Series algorithm.

20. A non-transitory computer-readable medium comprising instructions configured to cause a device to perform a method, the method comprising:

receiving depth information associated with the device, the depth information comprising distances from the device to surfaces in a real-world environment; and

determining a final pose, wherein determining the final pose comprises applying a Pose Search Algorithm (PSA) to a virtual model of the real-world environment and a first pose determined from the depth information, wherein the PSA comprises;

determining at least one neighborhood based upon the first pose; and

iteratively, through two or more iterations;

selecting a new pose from a plurality of poses within a neighborhood of a plurality of neighborhoods, each neighborhood of the plurality of neighborhoods covering a portion of a search space associated with the virtual model of the real-world environment;

pruning a neighborhood from the plurality of neighborhoods, at least in part, by determining correspondences between the depth information oriented using a pose from the pruned neighborhood and at least a portion of the virtual model of the real-world environment; and

setting a size of at least one neighborhood in a subsequent iteration to be smaller than a size of at least one neighborhood in a preceding iteration, wherein

the final pose is the new pose so iteratively determined to more closely correspond to the virtual model than a pose of a previous iteration.

21. The non-transitory computer-readable medium of claim 20, wherein the first pose is a predicted pose associated with a belief determined based, at least in part, upon the depth information and upon inertial measurement unit (IMU) data acquired in conjunction with the acquisition of the depth information.

22. The non-transitory computer-readable medium of claim 20, wherein the PSA comprises a Scaling Series algorithm.