Sensor-aided wide-area localization on mobile devices

• US 9,031,283 B2
• Filed: 12/18/2012
• Issued: 05/12/2015
• Est. Priority Date: 07/12/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Arth, Clemens, Mulloni, Alessandro, Reitmayr, Gerhard, Schmalstieg, Dieter
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): SAINI, AMANDEEP SINGH
• Application Number:
  US13/719,012
• Publication Number:
  US 20140016821A1
• Time in Patent Office: 875 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A mobile device uses vision and orientation sensor data jointly for six degree of freedom localization, e.g., in wide-area environments. An image or video stream is captured while receiving geographic orientation data and may be used to generate a panoramic cylindrical map of an environment. A bin of model features stored in a database is accessed based on the geographic orientation data. The model features are from a pre-generated reconstruction of the environment produced from extracted features from a plurality of images of the environment. The reconstruction is registered to a global orientation and the model features are stored in bins based on similar geographic orientations. Features from the panoramic cylindrical map are matched to model features in the bin to produce a set of corresponding features, which are used to determine a position and an orientation of the camera.

Claims

1. A method comprising:

capturing an image of an environment with a camera;

receiving geographic orientation data associated with the image;

accessing a bin of model features from a database based on the geographic orientation data, wherein the model features are from a pre-generated three-dimensional model of the environment and are binned in the database based on geographic orientation of the model features;

extracting features from the image;

comparing the features extracted from the image to model features in the bin of model features to produce a set of corresponding features; and

using the set of corresponding features to determine a position and an orientation of the camera.

2. The method of claim 1, further comprising:

producing at least a portion of a panoramic cylindrical map of the environment using the features extracted from the image;

wherein comparing the features extracted from the image to the model features in the bin of model features uses features from the portion of the panoramic cylindrical map.

3. The method of claim 1, wherein the image of the environment is a frame from a video stream.

4. The method of claim 1, further comprising:

receiving position data associated with the image; and

wherein accessing the bin of model features from the database based on the geographic orientation data comprises accessing a data set block of model features from the database based on the position data, wherein the data set block of model features comprises a plurality of bins of model features, and using the geographic orientation data to access the bin of model features from the plurality of bins of model features in the data set block.

5. The method of claim 1, further comprising:

determining a normal vector for each feature; and

associating the geographic orientation data to the normal vector for each feature;

wherein accessing the bin of model features from the database based on the geographic orientation data comprises using the geographic orientation data associated with the normal vector for each feature to retrieve model features from a corresponding bin of model features with normal vectors in a range of geographic orientations that includes the geographic orientation data associated with the normal vector.

6. The method of claim 5, further comprising:

determining a gravity vector for each feature; and

using the gravity vector when comparing the features extracted from the image to the model features in the bin of model features to produce the set of corresponding features, wherein each model feature has an associated gravity vector.

7. An apparatus comprising:

a camera capable of capturing images of an environment;

an orientation sensor capable of providing geographic orientation data; and

a processor coupled to the camera and the orientation sensor, the processor configured to extract features from an image, determine a geographic orientation for each feature;

access a bin of model features for a database based on the geographic orientation of each feature, wherein the model features are from a pre-generated three-dimensional model of the environment and are binned in the database based on the geographic orientation of the model features;

compare the features extracted from the image to the bin of model features to produce a set of corresponding features; and

use the set of corresponding features to determine a position and an orientation of the camera.

8. The apparatus of claim 7, wherein the processor is further configured to:

produce at least a portion of a panoramic cylindrical map of the environment using the features extracted from the image;

wherein the processor is configured to compare the features extracted from the image to the model features in the bin of model features by configured to use features from the portion of the panoramic cylindrical map.

9. The apparatus of claim 7, further comprising:

a position sensor capable of determining a coarse position of the apparatus;

wherein the processor is coupled to the position sensor and is further configured to access a data set block of model features from the database based on the coarse position, wherein the data set block of model features comprises a plurality of bins of model features, and the processor is configured to use the geographic orientation data to access the bin of model features from the plurality of bins of model features in the data set block.

10. The apparatus of claim 7, wherein the processor is further configured to determine a normal vector for each feature;

and associate the geographic orientation data to the normal vector for each feature;

wherein the processor is configured to access the bin of model features from the database based on the geographic orientation data by being configured to use the geographic orientation data associated with the normal vector for each feature to retrieve model features from a corresponding bin of model features with normal vectors in a range of geographic orientations that includes the geographic orientation data associated with the normal vector.

11. The apparatus of claim 10, further comprising:

an accelerometer to determine a direction of gravity;

wherein the processor is coupled to the accelerometer and is further configured to determine a gravity vector for each feature and to use the gravity vector when comparing the features extracted from the image to the model features in the bin of model features to produce the set of corresponding features, wherein each model feature has an associated gravity vector.

12. The apparatus of claim 7, further comprising:

a wireless interface capable of communicating with a remote server coupled to the database, wherein the processor is configured to access the bin of model features from the database through the wireless interface.

13. An apparatus comprising:

means for capturing an image of an environment;

means for receiving geographic orientation data associated with the image;

means for accessing a bin of model features from a database based on the geographic orientation data, wherein the model features are from a pre-generated three-dimensional model of the environment and are binned in the database based on geographic orientation of the model features;

means for extracting features from the image;

means for comparing the features extracted from the image to model features in the bin of model features to produce a set of corresponding features; and

means for using the set of corresponding features to determine a position and an orientation of the apparatus.

14. The apparatus of claim 13, further comprising means for producing at least a portion of a panoramic cylindrical map of the environment using the features extracted from the image, wherein the means for comparing the features extracted from the image to the model features in the bin of model features uses features from the portion of the panoramic cylindrical map.

15. The apparatus of claim 13, further comprising:

means for receiving position data; and

wherein the means for accessing the bin of model features from the database based on the geographic orientation data accesses a data set block of model features from the database based on the position data, wherein the data set block of model features comprises a plurality of bins of model features, and uses the geographic orientation data to access the bin of model features from the plurality of bins of model features in the data set block.

16. The apparatus of claim 13, further comprising:

means for determining a normal vector for each feature; and

means for associating the geographic orientation data to the normal vector for each feature;

wherein the means for accessing the bin of model features from the database based on the geographic orientation data uses the geographic orientation data associated with the normal vector for each feature to retrieve model features from a corresponding bin of model features with normal vectors in a range of geographic orientations that includes the geographic orientation data associated with the normal vector.

17. The apparatus of claim 16, further comprising:

means for determining a gravity vector for each feature; and

means for using the gravity vector when comparing the features extracted from the image to the model features in the bin of model features to produce the set of corresponding features, wherein each model feature has an associated gravity vector.

18. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to receive geographic orientation data associated with an image of an environment captured with a camera;

program code to access a bin of model features from a database based on the geographic orientation data, wherein the model features are from a pre-generated three-dimensional model of the environment and are binned in the database based on geographic orientation of the model features;

program code to extract features from the image;

program code to compare the features extracted from the image to model features in the bin of model features to produce a set of corresponding features; and

program code to use the set of corresponding features to determine a position and an orientation of the camera.

19. The non-transitory computer-readable medium of claim 18, further comprising:

program code to receive position data associated with the image; and

wherein the program code to access the bin of model features from the database based on the geographic orientation data comprises program code to access a data set block of model features from the database based on the position data, wherein the data set block of model features comprises a plurality of bins of model features, and use the geographic orientation data to access the bin of model features from the plurality of bins of model features in the data set block.

20. The non-transitory computer-readable medium of claim 18, further comprising:

program code to determine a normal vector for each feature; and

program code to associate the geographic orientation data to the normal vector for each feature;

wherein the program code to access the bin of model features from the database based on the geographic orientation data comprises program code to use the geographic orientation data associated with the normal vector for each feature to retrieve model features from a corresponding bin of model features with normal vectors in a range of geographic orientations that includes the geographic orientation data associated with the normal vector.

21. A method comprising:

obtaining a plurality of images of an environment;

extracting features from the plurality of images;

generating a reconstruction of the environment using the features;

registering the reconstruction of the environment to a global orientation;

associating a geographic orientation with each feature; and

storing the reconstruction of the environment with the features binned together into a plurality of bins, wherein each bin is associated with a different predetermined range of geographic orientation.

22. The method of claim 21, wherein associating the geographic orientation with each feature comprises:

determining a normal vector for each feature; and

associating an orientation of the normal vector for each feature with respect to the geographic orientation.

23. The method of claim 22, wherein determining the normal vector for a feature comprises:

determining vectors connecting a three-dimensional position of the feature and positions of all cameras that capture images of the three-dimensional position in the plurality of images of the environment; and

calculating a mean of the vectors as the normal vector.

24. The method of claim 21, further comprising:

storing the reconstruction of the environment in a plurality of data set blocks based on position, wherein each data set block comprises a different plurality of bins of features;

receiving position data from a mobile device; and

retrieving a data set block associated with the position data.

25. An apparatus comprising:

a database;

a processor coupled to the database, the processor being configured to generate a reconstruction of an environment using features extracted from a plurality of images of the environment, register the reconstruction of the environment to a global orientation, associate a geographic orientation with each feature; and

store in the database the reconstruction of the environment with the features binned together into a plurality of bins, wherein each bin is associated with a different predetermined range of geographic orientation.

26. The apparatus of claim 25, wherein the processor is configured to associate the geographic orientation with each feature by being configured to determine a normal vector for each feature;

and associate an orientation of the normal vector for each feature with respect to the geographic orientation.

27. The apparatus of claim 26, wherein the processor is configured to determine the normal vector for the feature by being configured to determine vectors connecting the 3D point of the feature and all cameras imaging the 3D point;

and calculate a mean of the vectors as the normal vector.

28. The apparatus of claim 25, further comprising:

an external interface capable of receiving position information from a mobile device; and

wherein the processor is further configured to store the reconstruction of the environment in a plurality of data set blocks based on position, wherein each data set block comprises a different plurality of bins of features.

29. An apparatus comprising:

means for obtaining a plurality of images of an environment;

means for extracting features from the plurality of images;

means for generating a reconstruction of the environment using the features;

means for registering the reconstruction of the environment to global orientation;

means for associating a geographic orientation with each feature; and

means for storing the reconstruction of the environment with the features binned together into a plurality of bins, wherein each bin is associated with a different predetermined range of geographic orientation.

30. The apparatus of claim 29, wherein the means for associating the geographic orientation with each feature comprises means for determining a normal vector for each feature;

and means for associating an orientation of the normal vector for each feature with respect to the geographic orientation.

31. The apparatus of claim 30, wherein the means for determining the normal vector for a feature determines vectors connecting a three-dimensional position of the feature and positions of all cameras that capture images of the three-dimensional position in the plurality of images of the environment;

and calculates a mean of the vectors as the normal vector.

32. The apparatus of claim 29, wherein the means for storing the reconstruction of the environment stores the reconstruction in a plurality of data set blocks based on position, wherein each data set block comprises a different plurality of bins of features, the apparatus further comprising:

means for receiving position information from a mobile device; and

means for retrieving a data set block associated with the position information.

33. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to extracting features from a plurality of images of an environment;

program code to generate a reconstruction of the environment using the features;

program code to register the reconstruction of the environment to global orientation;

program code to associate a geographic orientation with each feature; and

program code to store the reconstruction of the environment with the features binned together into a plurality of bins, wherein each bin is associated with a different predetermined range of geographic orientation.

34. The non-transitory computer-readable medium of claim 33, wherein the program code to associate the geographic orientation with each feature comprises:

program code to determine a normal vector for each feature; and

program code to associate an orientation of the normal vector for each feature with respect to the geographic orientation.

35. The non-transitory computer-readable medium of claim 34, wherein the program code to determine the normal vector for the feature comprises:

program code to determine vectors connecting a three-dimensional position of the feature and positions of all cameras that capture images of the three-dimensional position in the plurality of images of the environment; and

program code to calculate a mean of the vectors as the normal vector.

36. The non-transitory computer-readable medium of claim 33, further comprising:

program code to store the reconstruction of the environment in a plurality of data set blocks based on position, wherein each data set block comprises a different plurality of bins of features;

program code to receive position data from a mobile device; and

program code to retrieve a data set block associated with the position data.