Techniques for ground-level photo geolocation using digital elevation

• US 9,165,217 B2
• Filed: 01/18/2013
• Issued: 10/20/2015
• Est. Priority Date: 01/18/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cao, Liangliang, Hua, Gang, Smith, John R., Codella, Noel C. F.
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US13/744,688
• Publication Number:
  US 20140205186A1
• Time in Patent Office: 1,005 Days
• Field of Search: 707/732, 707/723, 707/722, 707/705
• US Class Current: 1/1
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/188 Vegetation

Abstract

Techniques for generating cross-modality semantic classifiers and using those cross-modality semantic classifiers for ground level photo geo-location using digital elevation are provided. In one aspect, a method for generating cross-modality semantic classifiers is provided. The method includes the steps of: (a) using Geographic Information Service (GIS) data to label satellite images; (b) using the satellite images labeled with the GIS data as training data to generate semantic classifiers for a satellite modality; (c) using the GIS data to label Global Positioning System (GPS) tagged ground level photos; (d) using the GPS tagged ground level photos labeled with the GIS data as training data to generate semantic classifiers for a ground level photo modality, wherein the semantic classifiers for the satellite modality and the ground level photo modality are the cross-modality semantic classifiers.

Claims

1. A method for generating cross-modality semantic classifiers, the method comprising the steps of:

(a) using Geographic Information Service (GIS) data to label satellite images;

(b) using the satellite images labeled with the GIS data as training data to generate semantic classifiers for a satellite modality;

(c) using the GIS data to label Global Positioning System (GPS) tagged ground level photos; and

(d) using the GPS tagged ground level photos labeled with the GIS data as training data to generate semantic classifiers for a ground level photo modality,

wherein the semantic classifiers for the satellite modality and the ground level photo modality are the cross-modality semantic classifiers.

2. The method of claim 1, wherein the satellite images are of known locations.

3. The method of claim 1, further comprising the step of:

(e) obtaining the GPS tagged ground level photos from a Ground Level Photo Repository.

4. The method of claim 1, further comprising the steps of:

(f) using the cross-modality semantic classifiers to geo-locate a query ground level photo of an unknown location;

(g) tagging the query ground level photo which has been geo-located with GPS information resulting in a GPS tagged query ground level photo; and

(h) repeating steps (a)-(d) including the GPS tagged query ground level photo in the training data being used to generate the semantic classifiers for the ground level photo modality.

5. A method for geo-locating a query ground level photo of an unknown location from combined elevation data and satellite imagery, the method comprising the steps of:

(a) parsing the query ground level photo into one or more semantic regions;

(b) assigning semantic labels to the semantic regions;

(c) using cross-modality semantic classifiers to identify geo-spatial regions in the combined elevation data and satellite imagery that have at least one semantic classifier in common with the query ground level photo; and

(d) performing matches of the query ground level photo with the combined elevation data and satellite imagery for each of the geo-spatial regions identified in step (c).

6. The method of claim 5, wherein IBM Multimedia Analysis and Retrieval System (IMARS)-based semantic classifiers are used to label the semantic regions.

7. The method of claim 5, wherein step (d) is performed for an array of positions within each of the each of the geo-spatial regions identified in step (c).

8. The method of claim 7, further comprising the step of:

(e) estimating matches of the query ground level photo with the combined elevation data and satellite imagery at each of the positions within each of the geo-spatial regions identified in step (c).

9. The method of claim 8, wherein the matches in step (e) are estimated using a spherical vector cross-correlation mapping between processed edges on the query ground level photo and the combined elevation data and satellite imagery.

10. The method of claim 8, further comprising the step of:

(f) confirming an accuracy in the matches performed in step (e) using a finer-grained matching process than in step (e).

11. The method of claim 9, wherein the matches performed in step (e) are confirmed using a silhouette map matching process.

12. The method of claim 10, wherein one or more inaccuracies in the matches performed in step (e) are detected, the method further comprising the step of:

(g) repeating steps (e)-(f) iteratively until no inaccuracies are detected.

13. The method of claim 5, further comprising the step of:

(h) generating the cross-modality semantic classifiers by i) using GIS data to label satellite images;

ii) using the satellite images labeled with the GIS data as training data to generate semantic classifiers for a satellite modality;

iii) using the GIS data to label GPS tagged ground level photos; and

iv) using the GPS tagged ground level photos labeled with the GIS data as training data to generate semantic classifiers for a ground level photo modality, wherein the semantic classifiers for the satellite modality and the ground level photo modality are the cross-modality semantic classifiers.

14. A method for geo-locating a query ground level photo of an unknown location from combined elevation data and satellite imagery, the method comprising steps of:

(a) parsing the query ground level photo into one or more semantic regions;

(b) assigning semantic labels to the semantic regions;

(c) using cross-modality semantic classifiers to identify geo-spatial regions in the combined elevation data and satellite imagery that have at least one semantic classifier in common with the query ground level photo;

(d) estimating matches of the query ground level photo with the combined elevation data and satellite imagery at each of an array of positions within each of the geo-spatial regions identified in step (c); and

(e) confirming an accuracy in the matches performed in step (d) using a finer-grained matching process than in step (d).

15. The method of claim 14, wherein IMARS-based semantic classifiers are used to label the semantic regions.

16. The method of claim 14, wherein the matches in step (d) are estimated using a spherical vector cross-correlation mapping between processed edges on the query ground level photo and the combined elevation data and satellite imagery.

17. The method of claim 14, wherein the matches performed in step (d) are confirmed using a silhouette map matching process.

18. The method of claim 14, wherein one or more inaccuracies in the matches performed in step (d) are detected, the method further comprising the step of:

(f) repeating steps (d)-(e) iteratively until no inaccuracies are detected.

19. The method of claim 14, further comprising the step of:

(g) generating the cross-modality semantic classifiers by i) using GIS data to label satellite images;

ii) using the satellite images labeled with the GIS data as training data to generate semantic classifiers for a satellite modality;

iii) using the GIS data to label GPS tagged ground level photos; and

iv) using the GPS tagged ground level photos labeled with the GIS data as training data to generate semantic classifiers for a ground level photo modality, wherein the semantic classifiers for the satellite modality and the ground level photo modality are the cross-modality semantic classifiers.