Location of image capture device and object features in a captured image

• US 9,324,003 B2
• Filed: 03/14/2013
• Issued: 04/26/2016
• Est. Priority Date: 09/14/2009
• Status: Active Grant

• Current Assignee: Trimble Inc.
• Original Assignee: Trimble Navigation Limited (Trimble Inc.)
• Inventors: France, Peter G., Weisenburger, Shawn D.
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US13/829,085
• Publication Number:
  US 20130243250A1
• Time in Patent Office: 1,139 Days
• Field of Search: 382/113, 382/195, 382/233, 382/294, 382/305, 348/113, 348/117, 348/118, 348/130, 348/143, 348/153, 348/239, 348/573
• US Class Current: 1/1
• CPC Class Codes: G01C 15/00 Surveying instruments or accessories not provided for in groups G01C1/00 - G01C13/00
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

A method for matching a region on an object of interest with a geolocation in a coordinate system is disclosed. In one embodiment, an image of a region on an object of interest is captured on an image capture device. The image is processed to detect a located feature using a feature detection algorithm. Further processing of the located feature is performed to derive a first feature descriptor using a feature descriptor extraction algorithm. The feature descriptor is stored in a memory. A database of feature descriptors having geolocation information associated with the feature descriptors is searched for a match to the first feature descriptor. The geolocation information is then made available for access.

Claims

1. A method for matching a region on an object of interest with a geolocation in a coordinate system, said method comprising:

capturing an image of a region of an object of interest using an image capture device at a first location;

processing said image to detect a located feature using a feature detection algorithm, wherein said feature detection algorithm is configured to detect said located feature at a two-dimension pixel coordinate in said image for further processing;

further processing said located feature to derive a first feature descriptor using a feature descriptor extraction algorithm, wherein said feature descriptor extraction algorithm is configured to resample an area in said image around said located feature and derive a compact representation that is usable to identify said located feature and to distinguish said located feature from other features in said image;

storing said first feature descriptor in a memory;

searching a database of feature descriptors having geolocation information associated with the feature descriptors, for a match to said first feature descriptor;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said first feature descriptor; and

making said geolocation information available for access.

2. The method of claim 1 further comprising:

sending said first feature descriptor from said image capture device to an image database; and

matching said first feature descriptor with at least one feature descriptor stored at said image database, said at least one feature descriptor stored at said database having geolocation information associated therewith.

3. The method of claim 2 further comprising:

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with said at least one feature descriptor;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one feature descriptor; and

conveying by said image database said geographic position and said orientation to said image capture device.

4. The method of claim 2 further comprising:

sending by said image database to said image capture device the geolocation information associated with said at least one feature descriptor;

storing the geolocation information associated with said at least one feature descriptor upon a database disposed upon said image capture device;

and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one feature descriptor.

5. The method of claim 2 further comprising:

sending an estimated location of said image capture device to said image database; and

selecting by said image database a subset of feature descriptors to match with said first feature descriptor sent by said image capture device based upon said estimated location.

6. The method of claim 5 further comprising:

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with at least one of said subset of feature descriptors;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

conveying by said image database said geographic position and said orientation to said image capture device.

7. The method of claim 5 further comprising:

sending by said image database to said image capture device the geolocation information associated with at least one of said subset of feature descriptors;

storing the geolocation information associated with said at least one of said subset of feature descriptors upon a database disposed upon said image capture device;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors.

8. The method of claim 5 further comprising:

sending an estimated orientation of said image capture device to said database; and

selecting by said image database a subset of feature descriptors to match with said first feature descriptor sent by said image capture device based upon said estimated orientation, each of said subset of feature descriptors having geolocation information associated therewith.

9. The method of claim 8 further comprising:

matching said first feature descriptor with at least one of said subset of feature descriptors;

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

conveying by said image database said geographic position and said orientation to said image capture device.

10. The method of claim 8 further comprising:

sending by said image database to said image capture device the geolocation information associated with at least one of said subset of feature descriptors;

storing the geolocation information associated with said at least one of said subset of feature descriptors upon a database disposed upon said image capture device;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors.

11. The method of claim 1 further comprising:

sending an estimated location of said image capture device to an image database; and

selecting by said image database a subset of feature descriptors to match with said first feature descriptor sent by said image capture device based upon said estimated location, each of said subset of feature descriptors having geolocation information associated therewith.

12. The method of claim 11 further comprising:

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with at least one of said subset of feature descriptors;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

conveying by said image database said geographic position and said orientation to said image capture device.

13. The method of claim 11 further comprising:

sending by said image database to said image capture device the geolocation information associated with at least one of said subset of feature descriptors;

storing the geolocation information associated with said at least one of said subset of feature descriptors upon a database disposed upon said image capture device;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors.

14. The method of claim 11 further comprising:

sending an estimated orientation of said image capture device to said database; and

selecting by said image database a subset of feature descriptors to match with said first feature descriptor sent by said image capture device based upon said estimated orientation, each of said subset of feature descriptors having geolocation information associated therewith.

15. The method of claim 14 further comprising:

matching said first feature descriptor with at least one of said subset of feature descriptors;

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

conveying by said image database said geographic position and said orientation to said image capture device.

16. The method of claim 14 further comprising:

sending by said image database to said image capture device the geolocation information associated with at least one of said subset of feature descriptors;

storing the geolocation information associated with said at least one of said subset of feature descriptors upon a database disposed upon said image capture device;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors.

17. The method of claim 1, wherein said image further comprises a second object of interest, said method further comprising:

capturing a second image of said object of interest and a second object of interest at a second location;

processing said second image to detect said located feature using said feature detection algorithm;

further processing said located feature of said second image to derive said first feature descriptor using said feature descriptor extraction algorithm;

determining a geographic position and an orientation of said image capture device at said first location and a second geographic position and a second orientation of an image capture device at said second location; and

using a bundle adjustment algorithm to calculate a geographic position of said second object of interest based upon said geographic position and said orientation of said image capture device at said first location and said second geographic position and said second orientation of an image capture device at said second location.

18. The method of claim 1 further comprising:

using said geolocation information for a plurality of points of interest to derive geolocation information for a second object of interest which is co-planar with said plurality of points of interest.

19. A non-transitory computer-readable storage medium comprising computer executable code for directing a processor to executed a method for matching a region on an object of interest with a geolocation in a coordinate system, said method comprising:

capturing an image of a region of an object of interest using an image capture device at a first location;

processing said image to detect a located feature using a feature detection algorithm, wherein said feature detection algorithm is configured to detect said located feature at a two-dimension pixel coordinate in said image for further processing;

further processing said located feature to derive a first feature descriptor using a feature descriptor extraction algorithm, wherein said feature descriptor extraction algorithm is configured to resample an area in said image around said located feature and derive a compact representation that is usable to identify said located feature and to distinguish said located feature from other features in said image;

storing said first feature descriptor in a memory;

searching a database of feature descriptors having geolocation information associated with the feature descriptors, for a match to said first feature descriptor;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said first feature descriptor; and

making said geolocation information available for access.

20. The non-transitory computer-readable storage medium of claim 19, wherein said method further comprises:

sending said first feature descriptor from said image capture device to an image database; and

matching said first feature descriptor with at least one feature descriptor stored at said image database, said at least one feature descriptor stored at said database having geolocation information associated therewith.

21. The non-transitory computer-readable storage medium of claim 20, wherein said method further comprises:

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with said at least one feature descriptor;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one feature descriptor; and

conveying by said image database said geographic position and said orientation to said image capture device.

22. The non-transitory computer-readable storage medium of claim 20, wherein said method further comprises:

sending by said image database to said image capture device the geolocation information associated with said at least one feature descriptor;

storing the geolocation information associated with said at least one feature descriptor upon a database disposed upon said image capture device;

and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one feature descriptor.

23. The non-transitory computer-readable storage medium of claim 20, wherein said method further comprises:

sending an estimated location of said image capture device to said image database; and

selecting by said image database a subset of feature descriptors to match with said first feature descriptor sent by said image capture device based upon said estimated location.

24. The non-transitory computer-readable storage medium of claim 23, wherein said method further comprises:

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with at least one of said subset of feature descriptors;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

conveying by said image database said geographic position and said orientation to said image capture device.

25. The non-transitory computer-readable storage medium of claim 23, wherein said method further comprises:

sending by said image database to said image capture device the geolocation information associated with at least one of said subset of feature descriptors;

storing the geolocation information associated with said at least one of said subset of feature descriptors upon a database disposed upon said image capture device;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors.

26. The non-transitory computer-readable storage medium of claim 23, wherein said method further comprises:

sending an estimated orientation of said image capture device to said database; and

selecting by said image database a subset of feature descriptors to match with said first feature descriptor sent by said image capture device based upon said estimated orientation, each of said subset of feature descriptors having geolocation information associated therewith.

27. The non-transitory computer-readable storage medium of claim 26, wherein said method further comprises:

matching said first feature descriptor with at least one of said subset of feature descriptors;

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

conveying by said image database said geographic position and said orientation to said image capture device.

28. The non-transitory computer-readable storage medium of claim 26, wherein said method further comprises:

sending by said image database to said image capture device the geolocation information associated with at least one of said subset of feature descriptors;

storing the geolocation information associated with said at least one of said subset of feature descriptors upon a database disposed upon said image capture device;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors.

29. The non-transitory computer-readable storage medium of claim 19, wherein said method further comprises:

sending an estimated location of said image capture device to an image database; and

selecting by said image database a subset of feature descriptors to match with said first feature descriptor sent by said image capture device based upon said estimated location, each of said subset of feature descriptors having geolocation information associated therewith.

30. The non-transitory computer-readable storage medium of claim 29, wherein said method further comprises:

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with at least one of said subset of feature descriptors;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

conveying by said image database said geographic position and said orientation to said image capture device.

31. The non-transitory computer-readable storage medium of claim 29, wherein said method further comprises:

sending by said image database to said image capture device the geolocation information associated with at least one of said subset of feature descriptors;

storing the geolocation information associated with said at least one of said subset of feature descriptors upon a database disposed upon said image capture device;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors.

32. The non-transitory computer-readable storage medium of claim 29, wherein said method further comprises:

sending an estimated orientation of said image capture device to said database; and

selecting by said image database a subset of feature descriptors to match with said first feature descriptor sent by said image capture device based upon said estimated orientation, each of said subset of feature descriptors having geolocation information associated therewith.

33. The non-transitory computer-readable storage medium of claim 32, wherein said method further comprises:

matching said first feature descriptor with at least one of said subset of feature descriptors;

determining by said image database a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors;

determining by said image database an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

conveying by said image database said geographic position and said orientation to said image capture device.

34. The non-transitory computer-readable storage medium of claim 32, wherein said method further comprises:

sending by said image database to said image capture device the geolocation information associated with at least one of said subset of feature descriptors;

storing the geolocation information associated with said at least one of said subset of feature descriptors upon a database disposed upon said image capture device;

determining by said image capture device a geographic position of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors; and

determining by said image capture device an orientation of said image capture device based upon the geolocation information associated with said at least one of said subset of feature descriptors.

35. The non-transitory computer-readable storage medium of claim 19, wherein said image further comprises a second object of interest, said method further comprising:

capturing a second image of said object of interest and a second object of interest at a second location;

processing said second image to detect said located feature using said feature detection algorithm;

further processing said located feature of said second image to derive said first feature descriptor using said feature descriptor extraction algorithm;

determining a geographic position and an orientation of said image capture device at said first location and a second geographic position and a second orientation of an image capture device at said second location; and

using a bundle adjustment algorithm to calculate a geographic position of said second object of interest based upon said geographic position and said orientation of said image capture device at said first location and said second geographic position and said second orientation of an image capture device at said second location.

36. The non-transitory computer-readable storage medium of claim 19, wherein said method further comprises:

using said geolocation information for a plurality of points of interest to derive geolocation information for a second object of interest which is co-planar with said plurality of points of interest.

37. A system for determining the location of an image capture device, said system comprising:

an image capture device comprising;

a camera;

a processor coupled with a bus and configured to;

process an image captured by said camera to detect a located feature using a feature detection algorithm, wherein said feature detection algorithm is configured to detect said located feature at a two-dimension pixel coordinate in said image for further processing;

further process said located feature to derive a first feature descriptor using a feature descriptor extraction algorithm, wherein said feature descriptor extraction algorithm is configured to resample an area in said image around said located feature and derive a compact representation that is usable to identify said located feature and to distinguish said located feature from other features in said image; and

a memory coupled with said bus; and

an image database comprising;

an image database memory coupled with an image database bus; and

an image database processor coupled with said image database bus, said image database processor configured to access said first feature descriptor of an image received from said image capture device and match said first feature descriptor with a corresponding feature descriptor having geolocation information associated therewith, said image database processor further configured to make said geolocation information available for access as a result of matching said first feature descriptor with said corresponding feature descriptor and to determine a geographic position of said image capture device based upon the geolocation information associated with said first feature descriptor.

38. The system of claim 37, wherein said image database processor receives said first feature descriptor from said image capture device.

39. The system of claim 37, wherein said image database processor is further configured to generate an orientation of said image capture device based upon said geolocation information and convey said geographic position and said orientation to said image capture device.

40. The system of claim 37 wherein said image database processor is further configured to convey said corresponding feature descriptor and geolocation information to said image capture device.

41. The system of claim 37 wherein said image database processor is further configured to identify a subset of a plurality of feature descriptors having geolocation information associated therewith based upon an estimated location of said image capture device.