Exterior hybrid photo mapping

• US 9,947,100 B2
• Filed: 06/28/2016
• Issued: 04/17/2018
• Est. Priority Date: 09/27/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Moeglein, Mark Leo, Brunner, Christopher, Ramanandan, Arvind, Ramachandran, Mahesh, Tyagi, Abhishek, Chari, Murali Ramaswamy
• Primary Examiner(s): LANG, MICHAEL DEAN
• Application Number:
  US15/195,921
• Publication Number:
  US 20160307328A1
• Time in Patent Office: 658 Days
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 25/005 initial alignment, calibration or starting-up of inertial devices
• CPC Class Codes: G01S 19/14 specially adapted for specific applications
• CPC Class Codes: G01S 5/0264 at least one of the systems being a non-radio wave positioning system
• CPC Class Codes: G01S 5/0294 Trajectory determination or predictive filtering, e.g. target tracking or Kalman filtering
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings

Abstract

Embodiments disclosed pertain to the use of user equipment (UE) for the generation of a 3D exterior envelope of a structure based on captured images and a measurement set associated with each captured image. In some embodiments, a sequence of exterior images of a structure is captured and a corresponding measurement set comprising Inertial Measurement Unit (IMU) measurements, wireless measurements (including Global Navigation Satellite (GNSS) measurements) and/or other non-wireless sensor measurements may be obtained concurrently. A closed-loop trajectory of the UE in global coordinates may be determined and a 3D structural envelope of the structure may be obtained based on the closed loop trajectory and feature points in a subset of images selected from the sequence of exterior images of the structure.

Claims

1. A method on a device, the method comprising:

capturing a plurality of images of a structure when traversing a plurality of locations in a vicinity of the structure;

capturing a plurality of measurement sets, wherein each measurement set corresponds to at least one image in the plurality of images;

estimating, based, in part, on the plurality of images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the device;

transmitting information comprising the estimated trajectory of the device, the 3D structural envelope, the plurality of images, and the corresponding measurement sets, to a first server wirelessly coupled to the device; and

receiving, based, in part, on the transmitted information, a corrected trajectory of the device registered to absolute coordinates and a 3D structural envelope of the structure registered to the absolute coordinates.

2. The method of claim 1, wherein each measurement set comprises inertial Measurement Unit (IMU) measurements.

3. The method of claim 2, wherein each measurement set comprises available wireless measurements with correction information for the available wireless measurements.

4. The method of claim 3, wherein the available wireless measurements comprise:

Global Navigation Satellite System (GNSS) measurements comprising differentially corrected code and carrier phase observables and wherein the correction information for the wireless measurements comprises one or more of;

GNSS code and carrier differential corrections, GNSS precise orbital and clock information, or GNSS atmospheric corrections.

5. The method of claim 4, wherein estimating the trajectory of the device comprises:

applying, for each location in a subset of the plurality of locations, one or more of;

the GNSS code and carrier differential corrections, the GNSS precise orbital and clock information, or the GNSS atmospheric corrections to the GNSS measurements corresponding to the location to obtain a corresponding corrected GNSS measurement;

determining a set of corrected locations, wherein, for each location in the subset of the plurality of images, a corresponding corrected location in the set of corrected locations is obtained based on the corrected GNSS measurements; and

determining the estimated trajectory based, in part, on the set of corrected locations.

6. The method of claim 3, wherein the available wireless measurements further comprise Wireless Wide Area Network (WWAN) measurements comprising one or more of:

Observed Time Difference of Arrival (OTDoA) measurements, or

Reference Signal Time Difference (RSTD) measurements, or

Advanced Forward Link Trilateralation (AFLT) measurements, or

hybrid-AFLT measurements.

7. The method of claim 3, wherein the correction information for the available wireless measurements comprises antenna pattern information for the device.

8. The method of claim 1, wherein the corrected trajectory of the device registered to the absolute coordinates is determined based, in part, on a closed-loop trajectory of the device.

9. The method of claim 1, wherein receiving the corrected trajectory of the device registered to the absolute coordinates and the 3D structural envelope of the structure registered to the absolute coordinates comprises:

receiving an outdoor map comprising the corrected trajectory and the 3D structural envelope, wherein the outdoor map comprises a plurality of map layers registered to the absolute coordinates.

10. The method of claim 9, wherein the plurality of map layers includes at least two of:

a plan view map layer;

or

a 3D structural map layer, with external 3D structural envelope information for neighboring structures;

a map layer with attenuation or delay of WWAN signals associated with the structure, or

a map layer with annotations for any variation WWAN signal strength in the vicinity of the structure;

or

a Line Of Sight (LOS) map layer, indicating, for each absolute coordinate in a first plurality of the absolute coordinates on the LOS map layer, corresponding antennas in a line of sight relative to the absolute coordinate.

11. The method of claim 1, wherein estimating the trajectory of the device and the 3D structural envelope of the structure comprises:

applying Visual Simultaneous Localization and Mapping (VSLAM) techniques to a subset of the plurality of images of the structure to determine a scene geometry and a set of first 6 Degrees Of Freedom (6DOF) poses of the device, wherein each first 6DOF pose of the device in the set of first 6DOF poses corresponds to a distinct image in the subset of the plurality of images each first 6DOF pose being determined based on keypoints in the subset of the plurality of images; and

determining the external 3D structural envelope of the structure based, in part, on the scene geometry; and

determining the estimated trajectory based, in part, on the set of first 6DOF poses.

12. The method of claim 11, wherein determining the estimated trajectory based on the set of first 6DOF poses comprises:

providing each first 6DOF pose in the set of first 6DOF poses to an Extended Kalman Filter (EKF), wherein for each first 6DOF pose, the EKF determines a corresponding second 6DOF pose of the device in the absolute coordinates, based in part, on the first 6DOF pose, Inertial Measurement Unit (IMU) measurements, and available wireless measurements, wherein each measurement set comprises IMU measurements and available wireless measurements with correction information for the available wireless measurements.

13. A device comprising:

a camera configured to capture a plurality of images of a structure when traversing a plurality of locations in a vicinity of the structure;

one or more sensors;

a processor, coupled to the camera and the one or more sensors wherein the processor is configured to;

determine a plurality of measurement sets, wherein each measurement set comprises measurements by the one or more sensors and each measurement set corresponds to at least one image in the plurality of images;

estimate, based, in part, on the plurality of images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the device;

transmit information comprising the estimated trajectory of the device, and the 3D structural envelope, the plurality of images, and the corresponding measurement sets, to a first server wirelessly coupled to the device; and

receive, based, in part, on the transmitted information, a corrected trajectory of the device registered to absolute coordinates and a 3D structural envelope of the structure registered to the absolute coordinates.

14. The device of claim 13, wherein:

the one or more sensors comprise an Inertial Measurement Unit (IMU); and

each measurement set comprises IMU measurements.

15. The device of claim 14, further comprising:

a wireless measurement circuit coupled to the processor, wherein the wireless measurement circuit is configured to take wireless measurements of available wireless signals, and acquire correction information for the wireless measurements; and

each measurement set further comprises the wireless measurements with the correction information for the wireless measurements.

16. The device of claim 15, wherein the wireless measurements comprise:

Global Navigation Satellite System (GNSS) measurements comprising differentially corrected code and carrier phase observables and wherein the correction information for the wireless measurements comprises one or more of;

GNSS code and carrier differential corrections, GNSS precise orbital and clock information, or GNSS atmospheric corrections.

17. The device of claim 16, wherein, to estimate the trajectory of the device, the processor is configured to:

apply, for each location in a subset of plurality of locations, one or more of the GNSS code and carrier differential corrections, the GNSS precise orbital and clock information, or the GNSS atmospheric corrections to the GNSS measurements corresponding to the location to obtain a corresponding corrected GNSS measurement;

determine a set of corrected locations, wherein, for each location in the subset of the plurality of images, a corresponding corrected location in the set of corrected locations is obtained based on the corrected GNSS measurements; and

determine the estimated trajectory based, in part, on the set of corrected locations.

18. The device of claim 15, wherein the wireless measurements further comprise:

Wireless Wide Area Network (WWAN) measurements comprising one or more of;

Observed Time Difference of Arrival (OTDOA) measurements, or

Reference Signal Time Difference (RSTD) measurements, or

Advanced Forward Link Trilateralation (AFLT) measurements, or

hybrid-AFLT measurements.

19. The device of claim 15, wherein correction information fir the wireless measurements comprises antenna pattern information for the device.

20. The device of claim 13, wherein the corrected trajectory of the device corresponds to a closed-loop trajectory of the device.

21. The device of claim 13, wherein to receive the corrected trajectory of the device registered to the absolute coordinates and 3D structural envelope of the structure registered to the absolute coordinates, the processor is configured to:

receive an outdoor map comprising the corrected trajectory of the device and the 3D structural envelope, wherein the outdoor map comprises a plurality of map layers registered to the absolute coordinates.

22. The device of claim 21, wherein the plurality of map layers includes at least two of:

a plan view map layer;

or

a 3D structural map layer, with external 3D structural envelope information for neighboring structures;

a map layer with attenuation or delay of WWAN signals associated with the structure;

or

a map layer with annotations for any variation in WWAN signal strength in the vicinity of the structure;

or

a Line Of Sight (LOS) map layer, indicating, for each absolute coordinate in the absolute coordinates on the LOS map layer, corresponding antennas in a line of sight relative to the absolute coordinate.

23. The device of claim 13, wherein, to estimate the trajectory of the device and 3D structural envelope of the structure, the processor is configured to:

apply Visual Simultaneous Localization and Mapping (VSLAM) techniques to a subset of the plurality of images of the structure to determine a scene geometry and a set of first 6 Degrees Of Freedom (6DOF) poses of the device, wherein each first 6DOF pose of the device in the set of first 6DOF poses corresponds to a distinct image in the subset of the plurality of images each 6DOF pose being determined based on keypoints in the subset of the plurality of images; and

determine the external 3D structural envelope of the structure based, in part, on the scene geometry; and

determine the estimated trajectory based, in part, on the set of first 6DOF poses.

24. The device of claim 23, wherein, to determine the estimated trajectory of the device, the processor is configured to:

provide each first 6DOF pose in the set of first 6DOF poses to an Extended Kalman Filter (EKF), wherein, for each first 6DOF pose, the EKF determines a corresponding second 6DOF pose of the device in the absolute coordinates, based in part, on the first 6DOF pose, the IMU measurements and the wireless measurements.

25. A device comprising:

imagine means configured to capture a plurality of images of a structure when traversing a plurality of locations in a vicinity of the structure;

sensing means;

means for determining a plurality of measurement sets based, in part, on measurements by the sensing means, wherein each measurement set corresponds to at least one image in the plurality of images;

means for estimating, based, in part, on the plurality of images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the device;

means for transmitting information comprising the estimated trajectory of the device, the 3D structural envelope, the plurality of images, and the corresponding measurement sets, to a server wirelessly coupled to the device; and

means for receiving, based, in part, on the transmitted information, a corrected trajectory of the device registered to absolute coordinates and a 3D structural envelope of the structure registered to the absolute coordinates.

26. The device of claim 25, wherein the sensing means comprises an Inertial Measurement Unit (IMU) means.

27. The device of claim 25, further comprising wireless measurement means configured to take wireless measurements of available wireless signals, and acquire correction information for the wireless measurements, wherein each measurement set further comprises the wireless signal measurements.

28. A non-transitory computer-readable medium comprising instructions, which, when executed by a processor, cause the processor to be configured to:

capture a plurality of images of a structure when traversing a plurality of locations in a vicinity of the structure;

capture a plurality of measurement sets, wherein each measurement set corresponds to at least one image in the plurality of images;

estimate, based, in part, on the plurality of images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the device;

transmit information comprising the estimated trajectory of the device, and the 3D structural envelope, the plurality of images, and the corresponding measurement sets, to a first server wirelessly coupled to the device; and

receive, based, in part, on the transmitted information, a corrected trajectory of the device registered to absolute coordinates and a 3D structural envelope of the structure registered to the absolute coordinates.

29. The computer-readable medium of claim 28, wherein each measurement set comprises Inertial Measurement Unit (IMU) measurements.

30. The computer-readable medium of claim 29, wherein each measurement set comprises available wireless measurements with correction information for the available wireless measurements.