EXTERIOR HYBRID PHOTO MAPPING

• US 20150094952A1
• Filed: 09/25/2014
• Published: 04/02/2015
• Est. Priority Date: 09/27/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: RAMANANDAN, Arvind, MOEGLEIN, Mark Leo, BRUNNER, Christopher, RAMACHANDRAN, Mahesh, TYAGI, Abhishek, CHARI, Murali Ramaswamy
• Granted Patent: US 9,405,972 B2
• Time in Patent Office: Days
• US Class Current: 701/491
• CPC Class Codes: G01C 21/1654 with electromagnetic compass
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 25/005 initial alignment, calibration or starting-up of inertial devices
• CPC Class Codes: G01S 19/14 specially adapted for specific applications
• CPC Class Codes: G01S 5/0264 at least one of the systems being a non-radio wave positioning system
• CPC Class Codes: G01S 5/0294 Trajectory determination or predictive filtering, e.g. target tracking or Kalman filtering
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings

Abstract

Embodiments disclosed pertain to the use of user equipment (UE) for the generation of a 3D exterior envelope of a structure based on captured images and a measurement set associated with each captured image. In some embodiments, a sequence of exterior images of a structure is captured and a corresponding measurement set comprising Inertial Measurement Unit (IMU) measurements, wireless measurements (including Global Navigation Satellite (GNSS) measurements) and/or other non-wireless sensor measurements may be obtained concurrently. A closed-loop trajectory of the UE in global coordinates may be determined and a 3D structural envelope of the structure may be obtained based on the closed loop trajectory and feature points in a subset of images selected from the sequence of exterior images of the structure.

Claims

1. A method on a user equipment (UE), the method comprising:

capturing a plurality of images of the exterior of a structure when traversing a plurality of locations in the vicinity of the structure;

capturing a plurality of measurement sets, wherein each measurement set corresponds to at least one image and comprises at least one of Inertial Measurement Unit (IMU) measurements or available wireless measurements with correction information for the wireless measurements;

estimating, based, in part, on the captured images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the UE;

sending the estimated trajectory of the UE, and one or more of the 3D structural envelope, captured images, or the corresponding measurement sets, to a server wirelessly coupled to the UE; and

receiving, based, in part, on the estimated trajectory of the UE, captured images and measurement sets, a corrected trajectory of the UE and a 3D structural envelope of the structure registered to absolute coordinates.

2. The method of claim 1, wherein the wireless measurements comprise:

Global Navigation Satellite System (GNSS) measurements comprising differentially corrected code and carrier phase observables and wherein the correction information for the wireless measurements comprises one or more of;

GNSS code and carrier differential corrections, GNSS precise orbital and clock information, and GNSS atmospheric corrections.

3. The method of claim 2, wherein estimating the trajectory of the UE comprises:

applying, for each location in a subset of locations on the trajectory, one or more of the;

GNSS code and carrier differential corrections, GNSS precise orbital and clock information, or GNSS atmospheric corrections to GNSS measurements at the location;

obtaining, for each location in the subset, a corresponding corrected location based on the corrected GNSS measurements; and

determining a trajectory based, in part, on the corrected location corresponding to each location in the subset.

4. The method of claim 2, wherein the correction information for the wireless measurements is received by the UE from a network server.

5. The method of claim 2, wherein the UE sends the wireless measurements to a network server and receives corrected wireless measurement information from the network server.

6. The method of claim 1, wherein the wireless measurements further comprise Wireless Wide Area Network (WWAN) measurements comprising one or more of:

Observed Time Difference of Arrival (OTDOA) measurements, or Reference Signal Time Difference (RSTD) measurements, or

Advanced Forward Link Trilateralation (AFLT) measurements, or

hybrid-AFLT measurements.

7. The method of claim 1, wherein correction information for the wireless measurements comprises antenna pattern information for the UE.

8. The method of claim 1, wherein the corrected trajectory corresponds to a closed-loop trajectory of the UE.

9. The method of claim 1, wherein receiving the corrected trajectory and 3D structural envelope of the structure registered to absolute coordinates comprises:

receiving an outdoor map comprising the corrected trajectory and the 3D structural envelope, wherein the outdoor map comprises a plurality map layers registered to the absolute coordinates.

10. The method of claim 9, wherein the plurality of layers includes at least two of:

a plan view map;

or

a 3D structural map, with external 3D structural envelope information for neighboring structures;

attenuation or delay of WWAN signals associated with the structure, or

annotations for any variation in WWAN signal strength in the vicinity of the structure;

or

a Line Of Sight (LOS) map layer, indicating, for each absolute coordinate in a first plurality of absolute coordinates on the map, corresponding WLAN antennas in a line of sight relative to the absolute coordinate.

11. The method of claim 1, wherein estimating the trajectory and 3D structural envelope of the structure comprises:

applying Visual Simultaneous Localization and Mapping (VSLAM) techniques to a subset of the plurality of images of the structure to determine a scene geometry and a 6 Degrees Of Freedom (6DOF) pose of the UE relative to each image in the subset and a scene geometry, the pose being determined based on keypoints in the subset of images; and

obtaining the external 3D structural envelope of the structure based, in part, on the scene geometry; and

obtaining the estimated trajectory based, in part, on the estimated 6DOF pose.

12. The method of claim 11, wherein obtaining the estimated trajectory comprises:

providing the 6DOF pose to an Extended Kalman Filter (EKF), wherein the EKF determines a pose of the UE in absolute coordinates, based in part, on the 6DOF pose, IMU measurements and available wireless measurements.

13. A User Equipment (UE) comprising:

a camera configured to capture a plurality of images of the exterior of a structure when traversing a plurality of locations in the vicinity of the structure;

a plurality of sensors, the sensors comprising an Inertial Measurement Unit (IMU);

a wireless module configured to take measurements of available wireless signals, and acquire correction information for the wireless measurements;

a processor, coupled to the camera, sensors and wireless module, wherein the processor is configured to;

obtain the plurality of images of the exterior of a structure;

obtain a plurality of measurement sets, wherein each measurement set corresponds to at least one image in the plurality of images, and each measurement set comprises at least one of IMU measurements and available wireless measurements with correction information for the wireless measurements;

estimate, based, in part, on the captured images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the UE;

send the estimated trajectory of the UE, and one or more of the 3D structural envelope, captured images, or the corresponding measurement sets, to a server wirelessly coupled to the UE; and

receive, based, in part, on the estimated trajectory of the UE, captured images and measurement sets, a corrected trajectory of the UE and a 3D structural envelope of the structure registered to absolute coordinates.

14. The UE of claim 13, wherein the wireless measurements comprise:

Global Navigation Satellite System (GNSS) measurements comprising differentially corrected code and carrier phase observables and wherein the correction information for the wireless measurements comprises one or more of;

GNSS code and carrier differential corrections, GNSS precise orbital and clock information, and GNSS atmospheric corrections.

15. The UE of claim 14, wherein, to estimate the trajectory of the UE, the processor is configured to:

apply, for each location in a subset of locations on the trajectory, one or more of the;

GNSS code and carrier differential corrections, GNSS precise orbital and clock information, or GNSS atmospheric corrections to GNSS measurements at the location;

obtain, for each location in the subset, a corresponding corrected location based on the corrected GNSS measurements; and

determine a trajectory based, in part, on the corrected location corresponding to each location in the subset.

16. The UE of claim 14, wherein the correction information for the wireless measurements is received by the wireless module from a network server.

17. The UE of claim 14, wherein:

the wireless module sends the wireless measurements to a network server and receives corrected wireless measurement information from the network server.

18. The UE of claim 13, wherein the wireless measurements further comprise Wireless Wide Area Network (WWAN) measurements comprising one or more of:

Observed Time Difference of Arrival (OTDOA) measurements, or Reference Signal Time Difference (RSTD) measurements, or

Advanced Forward Link Trilateralation (AFLT) measurements, or

hybrid-AFLT measurements.

19. The UE of claim 13, wherein correction information for the wireless measurements comprises antenna pattern information for the UE.

20. The UE of claim 13, wherein the corrected trajectory corresponds to a closed-loop trajectory of the UE.

21. The UE of claim 13, wherein to receive the corrected trajectory and 3D structural envelope of the structure registered to absolute coordinates, the processor is configured to:

receive an outdoor map comprising the corrected trajectory and the 3D structural envelope, wherein the outdoor map comprises a plurality map layers registered to the absolute coordinates.

22. The UE of claim 21, wherein the plurality of layers includes at least two of:

a plan view map;

or

a 3D structural map, with external 3D structural envelope information for neighboring structures;

attenuation or delay of WWAN signals associated with the structure, or

annotations for any variation in WWAN signal strength in the vicinity of the structure;

or

a Line Of Sight (LOS) map layer, indicating, for each absolute coordinate in a first plurality of absolute coordinates on the map, corresponding WLAN antennas in a line of sight relative to the absolute coordinate.

23. The UE of claim 13, wherein, to estimate the trajectory of the UE and 3D structural envelope of the structure, the processor is configured to:

apply Visual Simultaneous Localization and Mapping (VSLAM) techniques to a subset of the plurality of images of the structure to determine a scene geometry and a 6 Degrees Of Freedom (6DOF) pose of the UE relative to each image in the subset and a scene geometry, the pose being determined based on keypoints in the subset of images; and

obtain the external 3D structural envelope of the structure based, in part, on the scene geometry; and

obtain the estimated trajectory based, in part, on the estimated 6DOF pose.

24. The UE of claim 23, wherein, to obtain the estimated trajectory of the UE, the processor is configured to:

provide the 6DOF pose to an Extended Kalman Filter (EKF), wherein the EKF determines a pose of the UE in absolute coordinates, based in part, on the 6DOF pose, IMU measurements and available wireless measurements.

25. A User Equipment (UE) comprising:

imaging means configured to capture a plurality of images of the exterior of a structure when traversing a plurality of locations in the vicinity of the structure;

sensing means, the sensing means comprising an Inertial Measurement Unit (IMU) means;

wireless measurement means configured to take measurements of available wireless signals, and acquire correction information for the wireless measurements;

means for obtaining the plurality of images of the exterior of a structure;

means for obtaining a plurality of measurement sets, wherein each measurement set corresponds to at least one image in the plurality of images, and each measurement set comprises at least one of IMU measurements and available wireless measurements with correction information for the wireless measurements;

means for estimating, based, in part, on the captured images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the UE;

means for sending the estimated trajectory of the UE, and one or more of the 3D structural envelope, captured images, or the corresponding measurement sets, to a server wirelessly coupled to the UE; and

means for receiving, based, in part, on the estimated trajectory of the UE, captured images and measurement sets, a corrected trajectory of the UE and a 3D structural envelope of the structure registered to absolute coordinates.

26. The UE of claim 25, wherein the wireless measurements comprise:

Global Navigation Satellite System (GNSS) measurements comprising differentially corrected code and carrier phase observables and wherein the correction information for the wireless measurements comprises one or more of;

GNSS code and carrier differential corrections, GNSS precise orbital and clock information, and GNSS atmospheric corrections.

27. A non-transitory computer-readable medium comprising instructions, which, when executed by a processor, cause the processor to be configured to:

capture a plurality of images of the exterior of a structure when traversing a plurality of locations in the vicinity of the structure;

capture a plurality of measurement sets, wherein each measurement set corresponds to at least one image and comprises at least one of Inertial Measurement Unit (IMU) measurements or available wireless measurements with correction information for the wireless measurements;

estimate, based, in part, on the captured images and the corresponding plurality of measurement sets, a 3D structural envelope of the structure and a trajectory of the UE;

send the estimated trajectory of the UE, and one or more of the 3D structural envelope, captured images, or the corresponding measurement sets, to a server wirelessly coupled to the UE; and

receive, based, in part, on the estimated trajectory of the UE, captured images and measurement sets, a corrected trajectory of the UE and a 3D structural envelope of the structure registered to absolute coordinates.

28. The computer-readable medium of claim 27, wherein the wireless measurements comprise:

Global Navigation Satellite System (GNSS) measurements comprising differentially corrected code and carrier phase observables and wherein the correction information for the wireless measurements comprises one or more of;

GNSS code and carrier differential corrections, GNSS precise orbital and clock information, and GNSS atmospheric corrections.

29. The computer-readable medium of claim 28, wherein the instructions to estimate a trajectory of the UE cause the processor to:

apply, for each location in a subset of locations on the trajectory, one or more of the;

GNSS code and carrier differential corrections, GNSS precise orbital and clock information, or GNSS atmospheric corrections to GNSS measurements at the location;

obtain, for each location in the subset, a corresponding corrected location based on the corrected GNSS measurements; and

determine a trajectory based, in part, on the corrected location corresponding to each location in the subset.

30. The computer-readable medium of claim 28, wherein the correction information for the wireless measurements is received by the UE from a network server.