Mobile Device Localization In Complex, Three-Dimensional Scenes

• US 20190012806A1
• Filed: 07/06/2017
• Published: 01/10/2019
• Est. Priority Date: 07/06/2017
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Lehmann, Oliver, Kluckner, Stefan, Chen, Terrence
• Granted Patent: US 10,699,438 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models

Abstract

The present embodiments relate to localizing a mobile device in a complex, three-dimensional scene. By way of introduction, the present embodiments described below include apparatuses and methods for using multiple, independent pose estimations to increase the accuracy of a single, resulting pose estimation. The present embodiments increase the amount of input data by windowing a single depth image, using multiple depth images from the same sensor, and/or using multiple depth image from different sensors. The resulting pose estimation uses the input data with a multi-window model, a multi-shot model, a multi-sensor model, or a combination thereof to accurately estimate the pose of a mobile device.

Claims

1. A method of estimating the pose of a mobile device in a three-dimensional scene, the method comprising:

receiving, by a processor of the mobile device, a plurality of depth data measurements, the depth data measurements indicative of a depth from the mobile device to the three-dimensional scene;

estimating, based on a first depth measurement of the plurality of depth data measurements, a first pose of the mobile device with respect to the three-dimensional scene;

estimating, based on a second depth measurement of the plurality of depth data measurements, a second pose of the mobile device with respect to the three-dimensional scene;

estimating, with a pose model based on the first pose and the second pose, a third pose of the mobile device with respect to the three-dimensional scene; and

providing, based on the third pose, an output to a user.

2. The method of claim 1, wherein the plurality of depth data measurements comprise measurements of a same sensor, and wherein the pose model is a motion model based on movement of the same sensor between measurements.

3. The method of claim 2, wherein the motion model is a linear motion model based on sensor velocity measurements and corresponding time measurements.

4. The method of claim 2, wherein the motion model is a learned motion model, wherein a plurality of possible trajectories of the sensor are determined by the plurality of depth data measurements, the first pose and the second pose.

5. The method of claim 1, wherein the plurality of depth data measurements comprise measurements of different sensors, and wherein the pose model is a spatial model based on known spatial relationships between the different sensors.

6. The method of claim 1, wherein the plurality of depth data measurements comprise a plurality of measurements of each of a plurality of different sensors, and wherein the pose model includes a motion model based on movement of each sensor between measurements by the sensor and a spatial model based on known spatial relationships between the different sensors.

7. The method of claim 1, wherein the plurality of depth data measurements are partitioned from a single sensor measurement, and wherein the pose model comprises combining the first pose and the second pose into the third pose.

8. The method of claim 7, wherein the combining comprises a weighted average, mode finding or semantic mapping.

9. The method of claim 1, wherein the plurality of depth data measurements comprise measurements of a same sensor, and wherein a location of the mobile device for the second depth measurement is specified based on the first pose of the mobile device.

10. The method of claim 1, wherein the three-dimensional scene comprises a three-dimensional apparatus, and wherein the output to the user is a component of the three-dimensional apparatus.

11. The method of claim 1, wherein the output to the user comprises initialization or tracking in an augmented reality scenario.

12. A system for determining the pose of a mobile device in a three-dimensional scene, the system comprising:

a memory configured to store a plurality of depth data measurements, the depth data measurements indicative of a depth from the mobile device to the three-dimensional scene; and

a processor configured to;

receive the plurality of depth data measurements; and

determine a pose of the mobile device with respect to the three-dimensional scene, the pose based on fusing estimated poses for each of the plurality of depth data measurements with a dynamic model, the pose comprising a location and viewing angle of the mobile device.

13. The system of claim 12, wherein the dynamic model comprises one or more of:

a multi-window fusion model based on partitioning sensor measurements into a plurality of windows for each sensor measurement;

a multi-shot fusion model based on sensor movement between sensor measurements; and

a multi-sensor fusion model based on known spatial relationships between different sensors.

14. The system of claim 12, further comprising:

a sensor configured to capture the plurality of depth data measurements.

15. The system of claim 14, wherein the sensor is a 2.5D sensor or a depth sensor.

16. The system of claim 12, further comprising:

a display configured to provide an output to a user based on the estimated pose.

17. A method of localizing a mobile device in a three-dimensional scene, the method comprising:

capturing, by a sensor of the mobile device, a plurality of depth image data sets of the three-dimensional scene;

generating, by a processor of the mobile device, a plurality of initial pose estimations of the mobile device with respect to the three-dimensional scene, each of the initial pose estimations based on a different depth image data set;

generating, by the processor of the mobile device, a fused pose estimation of the mobile device with respect to the three-dimensional scene, the fused pose estimation determined using a trained machine-learning model based on the initial pose estimations; and

displaying, by a display of the mobile device, an output based on the fused pose estimation.

18. The system of claim 17, wherein the trained machine-learning model is a neural network.

19. The system of claim 17, wherein pose estimations of the plurality of initial pose estimations are generated in parallel operations by the processor of the mobile device.

20. The system of claim 17, wherein the output is a scene-based decision.