Hybrid photo navigation and mapping

• US 9,400,930 B2
• Filed: 09/25/2014
• Issued: 07/26/2016
• Est. Priority Date: 09/27/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Moeglein, Mark Leo, Brunner, Christopher, Chao, Hui, Chari, Murali Ramaswamy, Ramanandan, Arvind, Ramachandran, Mahesh, Tyagi, Abhishek
• Primary Examiner(s): Brandt, Christopher M
• Application Number:
  US14/497,219
• Publication Number:
  US 20150094089A1
• Time in Patent Office: 670 Days
• Field of Search: 455/456.1, 455/456.2, 455/456.3, 455/456.5, 455/456.6
• US Class Current: 1/1
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 25/005 initial alignment, calibration or starting-up of inertial devices
• CPC Class Codes: G01S 19/14 specially adapted for specific applications
• CPC Class Codes: G01S 5/0264 at least one of the systems being a non-radio wave positioning system
• CPC Class Codes: G01S 5/0294 Trajectory determination or predictive filtering, e.g. target tracking or Kalman filtering
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings

Abstract

Embodiments disclosed obtain a plurality of measurement sets from a plurality of sensors in conjunction with the capture of a sequence of exterior and interior images of a structure while traversing locations in and around the structure. Each measurement set may be associated with at least one image. An external structural envelope of the structure is determined from exterior images of the structure and the corresponding outdoor trajectory of a UE. The position and orientation of the structure and the structural envelope is determined in absolute coordinates. Further, an indoor map of the structure in absolute coordinates may be obtained based on interior images of the structure, a structural envelope in absolute coordinates, and measurements associated with the indoor trajectory of the UE during traversal of the indoor area to capture the interior images.

Claims

1. A method on a user equipment (UE) comprising:

capturing a plurality of images of an interior of a structure when traversing a plurality of locations within the structure;

capturing a plurality of measurement sets, wherein each measurement set corresponds to at least one image in the plurality of images, and each measurement set comprises at least one of Inertial Measurement Unit (IMU) measurements or available wireless measurements;

estimating a trajectory of the UE based, in part, on the plurality of images and the corresponding plurality of measurement sets;

sending the estimated trajectory, the plurality of images and corresponding measurement sets to a server wirelessly coupled to the UE; and

receiving, based on the estimated trajectory, the plurality of images and corresponding plurality of measurement sets, an indoor map of the structure from the server, wherein the indoor map is registered to absolute coordinates and comprises information pertaining to a corrected trajectory of the UE.

2. The method of claim 1, wherein estimating the trajectory of the UE comprises:

obtaining a position of the UE in absolute coordinates in a vicinity of the structure; and

estimating the trajectory based, in part, on the position of the UE obtained in absolute coordinates.

3. The method of claim 2, wherein estimating the trajectory of the UE comprises:

applying Visual Simultaneous Localization and Mapping (VSLAM) techniques to a subset of the plurality of images to determine a corresponding 6 Degrees Of Freedom (6DOF) pose of the UE relative to each image in the subset, the 6DOF pose being determined based on keypoints in the subset of images; and

determining, for each image in the subset, a corresponding pose of the UE in absolute coordinates, based in part, on the corresponding 6DOF pose, and a corresponding measurement set.

4. The method of claim 3, wherein determining the corresponding pose of the UE in absolute coordinates comprises:

providing, for each image in the subset, the corresponding 6DOF pose and the corresponding measurement set to an Extended Kalman Filter (EKF), wherein the EKF determines the corresponding pose of the UE in absolute coordinates.

5. The method of claim 1, wherein each measurement set further comprises one or more of:

barometric measurements;

or

altimeter measurements;

or

magnetometer measurements.

6. The method of claim 5, wherein the indoor map includes a plurality of layers aligned to the absolute coordinates.

7. The method of claim 6, wherein the plurality of aligned layers includes at least two of:

a plan view map, or

a 3-dimensional structure model, or

annotations for any variations to barometric pressure not predicted by a standard adiabatic lapse rate model, or

annotations for any magnetic field variations not predicted by a standard Earth magnetic field model, or

attenuation or delay of WWAN signals associated with the structure, or

annotations for any variation in WWAN signal strength within the structure, or

attenuation or delay of WLAN signals associated with the structure, or

an indication of WLAN AP locations within the structure, or

an indication of WLAN field strengths within the structure, or

an indication of WLAN RTT delay calibrations within the structure.

8. The method of claim 1, wherein the available wireless measurements comprise one or more of:

Wireless Wide Area Network (WWAN) measurements comprising one of;

Observed Time Difference of Arrival (OTDOA) measurements, or

Reference Signal Time Difference (RSTD) measurements, or

Advanced Forward Link Trilateralation (AFLT) measurements, or

hybrid-AFLT measurements;

or

Wireless Local Area Network (WLAN) measurements comprising Radio Frequency (RF) measurements of Access Points (APs) signals.

9. The method of claim 1, wherein the corrected trajectory of the UE corresponds to a closed-loop trajectory of the UE.

10. The method of claim 1, wherein the method is performed by a mapping application on the UE, wherein the mapping application provides instructions to the capture the plurality of images.

11. A User Equipment (UE) comprising:

a camera, the camera configured to capture a plurality of images of an interior of a structure when traversing a plurality of locations within the structure;

a plurality of sensors, the plurality of sensors comprising an Inertial Measurement Unit (IMU),

a wireless module, the wireless module configured to take measurements of available wireless signals, and

a processor, the processor coupled to the camera, the plurality of sensors and the wireless module, wherein the processor is configured to;

obtain the plurality of images of the interior of the structure;

obtain a plurality of measurement sets, wherein each measurement set corresponds to at least one image in the plurality of images, and each measurement set comprises at least one of IMU measurements and available wireless measurements;

estimate a trajectory of the UE based, in part, on the plurality of images and the corresponding plurality of measurement sets;

send the estimated trajectory, plurality of images and corresponding plurality of measurement sets to a server wirelessly coupled to the UE; and

receive, based on the estimated trajectory, plurality of images and corresponding plurality of measurement sets, an indoor map of the structure from the server, wherein the indoor map is registered to absolute coordinates and comprises information pertaining to a corrected trajectory of the UE.

12. The UE of claim 11, wherein, to estimate the trajectory of the UE, the processor is configured to:

obtain a position of the UE in absolute coordinates in a vicinity of the structure; and

estimate the trajectory of the UE based, in part, on the position of the UE obtained in absolute coordinates.

13. The UE of claim 12, wherein, to estimate the trajectory of the UE, the processor is configured to:

apply Visual Simultaneous Localization and Mapping (VSLAM) techniques to a subset of the plurality of images to determine a corresponding 6 Degrees Of Freedom (6DOF) pose of the UE relative to each image in the subset, the 6DOF pose being determined based on keypoints in the subset of images; and

determine, for each image in the subset, a corresponding pose of the UE in absolute coordinates, based in part, on the corresponding 6DOF pose, and a corresponding measurement set.

14. The UE of claim 13, wherein to determine a pose of the UE in absolute coordinates corresponding to each image in the subset, the processor is configured to:

provide, for each image in the subset, the corresponding 6DOF pose and the corresponding measurement set to an Extended Kalman Filter (EKF), wherein the EKF determines the corresponding pose of the UE in absolute coordinates.

15. The UE of claim 11, wherein each measurement set further comprises one or more of:

barometric measurements;

or

altimeter measurements;

or

magnetometer measurements.

16. The UE of claim 15, wherein the indoor map includes a plurality of layers aligned to the absolute coordinates.

17. The UE of claim 16, wherein the plurality of aligned layers includes at least two of:

a plan view map, or

a 3-dimensional structure model, or

annotations for any variations to barometric pressure not predicted by a standard adiabatic lapse rate model, or

annotations for any magnetic field variations not predicted by a standard Earth magnetic field model, or

attenuation or delay of WWAN signals associated with the structure, or

annotations for any variation in WWAN signal strength within the structure, or

attenuation or delay of WLAN signals associated with the structure, or

an indication of WLAN AP locations within the structure, or

an indication of WLAN field strengths within the structure, or

an indication of WLAN RTT delay calibrations within the structure.

18. The UE of claim 11, wherein the available wireless measurements comprise one or more of:

Wireless Wide Area Network (WWAN) measurements comprising one of;

Observed Time Difference of Arrival (OTDOA) measurements, or

Reference Signal Time Difference (RSTD) measurements, or

Advanced Forward Link Trilateralation (AFLT) measurements, or

hybrid-AFLT measurements;

or

Wireless Local Area Network (WLAN) measurements comprising Radio Frequency (RF) measurements of Access Points (APs) signals.

19. The UE of claim 11, wherein the corrected trajectory of the UE corresponds to a closed-loop trajectory of the UE.

20. The UE of claim 11, wherein the processor is configured to:

provide user-instructions to capture the plurality of images.

21. A User Equipment (UE) comprising:

imaging means, the imaging means configured to capture a plurality of images of an interior of a structure when traversing a plurality of locations within the structure;

sensing means, the sensing means comprising an Inertial Measurement Unit (IMU) means;

wireless measurement means, the wireless measurement means configured to take measurements of available wireless signals;

means for obtaining the plurality of images of the interior of the structure;

means for obtaining a plurality of measurement sets, wherein each measurement set corresponds to at least one image in the plurality of images, and each measurement set comprises at least one of IMU measurements and available wireless measurements;

means for estimating a trajectory of the UE based, in part, on the plurality of images and the plurality of measurement sets;

means for sending the estimated trajectory, plurality of images and corresponding plurality of measurement sets to a server wirelessly coupled to the UE; and

means for receiving, based on the estimated trajectory, plurality of images and corresponding plurality of measurement sets, an indoor map of the structure from the server, wherein the indoor map is registered to absolute coordinates and comprises information pertaining to a corrected trajectory of the UE.

22. The UE of claim 21, wherein means for estimating the trajectory of the UE comprises:

means for obtaining a position of the UE in absolute coordinates in a vicinity of the structure, and wherein the means for estimating the trajectory of the UE estimates the trajectory of the UE based, in part, on the position of the UE obtained in absolute coordinates.

23. The UE of claim 22, wherein, means for estimating the trajectory of the UE comprises:

means for applying Visual Simultaneous Localization and Mapping (VSLAM) techniques to a subset of the plurality of images to determine a corresponding 6 Degrees Of Freedom (6DOF) pose of the UE relative to each image in the subset, the 6DOF pose being determined based on keypoints in the subset of images; and

means for determining, for each image in the subset, a corresponding pose of the UE in absolute coordinates, based in part, on the corresponding 6DOF pose, and a corresponding measurement set.

24. The UE of claim 23, wherein means for determining a pose of the UE in absolute coordinates corresponding to each image in the subset, comprises:

means for providing, for each image in the subset, the corresponding 6DOF pose and the corresponding measurement set to an Extended Kalman Filter (EKF) means, wherein the EKF means determines the corresponding pose of the UE in absolute coordinates.

25. The UE of claim 21, wherein each measurement set further comprises one or more of:

barometric measurements;

or

altimeter measurements;

or

magnetometer measurements.

26. A non-transitory computer-readable medium comprising instructions, which, when executed by a processor coupled to a User Equipment (UE), cause the processor to be configured to:

capture a plurality of images of an interior of a structure when traversing a plurality of locations within the structure;

capture a plurality of measurement sets, wherein each measurement set corresponds to at least one image, and each measurement set comprises at least one of Inertial Measurement Unit (IMU) measurements or available wireless measurements;

estimate a trajectory of the UE based, in part, on the plurality of images and the plurality of measurement sets;

send the estimated trajectory, plurality of images and corresponding plurality of measurement sets to a server wirelessly coupled to the UE; and

receive, based on the estimated trajectory, plurality of images and corresponding plurality of measurement sets, an indoor map of the structure from the server, wherein the indoor map is registered to absolute coordinates and comprises information pertaining to a corrected trajectory of the UE.

27. The computer-readable medium of claim 26, wherein the instructions to estimate a trajectory of the UE cause the processor to:

obtain a position of the UE in absolute coordinates in a vicinity of the structure; and

estimate the trajectory based, in part, on the position of the UE obtained in absolute coordinates.

28. The computer-readable medium of claim 27, wherein the instructions to estimate a trajectory of the UE cause the processor to:

apply Visual Simultaneous Localization and Mapping (VSLAM) techniques to a subset of the plurality of images to determine a corresponding 6 Degrees Of Freedom (6DOF) pose of the UE relative to each image in the subset, the 6DOF pose being determined based on keypoints in the subset of images; and

determine, for each image in the subset, a corresponding pose of the UE in absolute coordinates, based in part, on the corresponding 6DOF pose, and a corresponding measurement set.

29. The computer-readable medium of claim 28, wherein the instructions to determine the corresponding pose of the UE in absolute coordinates cause the processor to:

provide, for each image in the subset, the corresponding 6DOF pose and the corresponding measurement set to an Extended Kalman Filter (EKF), wherein the EKF determines the corresponding pose of the UE in absolute coordinates.

30. The computer-readable medium of claim 26, wherein each measurement set further comprises one or more of:

barometric measurements;

or

altimeter measurements;

or

magnetometer measurements.