SYSTEMS AND METHODS FOR AUGMENTED REALITY PREPARATION, PROCESSING, AND APPLICATION

• US 20170352192A1
• Filed: 07/14/2017
• Published: 12/07/2017
• Est. Priority Date: 11/16/2014
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Eonite Perception, Inc. (Intel Corporation)
• Inventors: Petrovskaya, Anna, Varvak, Peter
• Granted Patent: US 9,972,137 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 7/36 using image sharpness techniques , e.g. image processing techniques for generating autofocus signals image data processing per se G06T; in cameras having a solid state image sensor H04N23/67
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Various of the disclosed embodiments provide systems and methods for acquiring and applying a depth determination of an environment in e.g., various augmented reality applications. A user may passively or actively scan a device (e.g., a tablet device, a mobile phone device, etc.) about the environment acquiring depth data for various regions. The system may integrate these scans into an internal three-dimensional model. This model may then be used in conjunction with subsequent data acquisitions to determine a device'"'"'s location and orientation within the environment with high fidelity. In some embodiments, these determinations may be accomplished in real-time or near-real-time. Using the high-fidelity orientation and position determination, various augmented reality applications may then be possible using the same device used to acquire the depth data or a new device.

Claims

1. A device to facilitate an augmented reality (AR)experience, the device comprising:

a depth sensor;

at least one processor;

at least one memory comprising instructions configured to cause the at least one processor to cause the device to perform a method comprising;

receiving depth frame data acquired using the depth sensor, the depth frame data comprising distances from the depth sensor to surfaces in a real-world environment;

determining a final pose, wherein determining the final pose comprises iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process, wherein

the correspondence identification process comprises;

determining a plurality of correspondences between the frame data transformed using a pose and model data of the real-world environment, and wherein

the pose determination process comprises;

iteratively refining an intermediate pose determination, at least in part, by pruning candidate poses within a search space and considering poses from progressively smaller neighborhoods; and

making the final pose available to an augmented reality application.

20. The device of claim 1, wherein the pose determination process comprises a Pose Search Algorithm (PSA) and iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process comprises performing an Expectation Maximization (EM) process.

21. The device of claim 20, wherein the pose determination process generates:

a Bayesian belief probability distribution associated with a given time; and

a pose corresponding to a transform.

22. The device of claim 20, wherein the PSA comprises a Scaling Series algorithm.

23. The device of claim 1, wherein the correspondence identification process comprises a likelihood field model with KD-trees.

24. The device of claim 1, wherein the correspondence identification process comprises an independent beam model.

25. The device of claim 1, wherein determining the final pose further comprises:

determining a first pose from a first iteration of the correspondence identification process and the pose determination process;

determining a second pose from a second successive iteration of the correspondence identification process and the pose determination process;

determining a difference between the first pose and the second pose in meters-and-radians;

determining that the difference is beneath a threshold; and

outputting the second pose as the final pose.

2-19. -19. (canceled)

26. A computer-implemented method comprising:

receiving depth frame data acquired using a depth sensor associated with an augmented reality device, the depth frame data comprising distances from the depth sensor to surfaces in a real-world environment;

determining a final pose, wherein determining the final pose comprises iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process, wherein

the correspondence identification process comprises;

determining a plurality of correspondences between the frame data transformed using a pose and model data of the real-world environment, and wherein

the pose determination process comprises;

iteratively refining an intermediate pose determination, at least in part, by pruning candidate poses within a search space and considering poses from progressively smaller neighborhoods; and

making the final pose available to an augmented reality application.

27. The computer-implemented method of claim 26, wherein the pose determination process comprises a Pose Search Algorithm (PSA) and iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process comprises performing an Expectation Maximization (EM) process.

28. The computer-implemented method of claim 27, wherein the pose determination process generates:

a Bayesian belief probability distribution associated with a given time; and

a pose corresponding to a transform.

29. The computer-implemented method of claim 27, wherein the PSA comprises a Scaling Series algorithm.

30. The computer-implemented method of claim 26, wherein the correspondence identification process comprises a likelihood field model with KD-trees.

31. The computer-implemented method of claim 26, wherein the correspondence identification process comprises an independent beam model.

32. The computer-implemented method of claim 26, wherein determining the final pose further comprises:

determining a first pose from a first iteration of the correspondence identification process and the pose determination process;

determining a second pose from a second successive iteration of the correspondence identification process and the pose determination process;

determining a difference between the first pose and the second pose in meters-and-radians;

determining that the difference is beneath a threshold; and

outputting the second pose as the final pose.

33. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method, comprising:

receiving depth frame data acquired using a depth sensor associated with an augmented reality device, the depth frame data comprising distances from the depth sensor to surfaces in a real-world environment;

determining a final pose based, wherein determining the final pose comprises iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process, wherein

the correspondence identification process comprises;

determining a plurality of correspondences between the frame data transformed using a pose and model data of the real-world environment, and wherein

the pose determination process comprises;

iteratively refining an intermediate pose determination, at least in part, by pruning candidate poses within a search space and considering poses from progressively smaller neighborhoods; and

making the final pose available to an augmented reality application.

34. The non-transitory computer-readable medium of claim 33, wherein the pose determination process comprises a Pose Search Algorithm (PSA) and iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process comprises performing an Expectation Maximization (EM) process.

35. The non-transitory computer-readable medium of claim 34, wherein the pose determination process generates:

a Bayesian belief probability distribution associated with a given time; and

a pose corresponding to a transform.

36. The non-transitory computer-readable medium of claim 34, wherein the PSA comprises a Scaling Series algorithm.

37. The non-transitory computer-readable medium of claim 33, wherein the correspondence identification process comprises a likelihood field model with KD-trees.

38. The non-transitory computer-readable medium of claim 33, wherein the correspondence identification process comprises an independent beam model.

39. The non-transitory computer-readable medium of claim 33, wherein determining the final pose further comprises:

determining a first pose from a first iteration of the correspondence identification process and the pose determination process;

determining a second pose from a second successive iteration of the correspondence identification process and the pose determination process;

determining a difference between the first pose and the second pose in meters-and-radians;

determining that the difference is beneath a threshold; and

outputting the second pose as the final pose.