Salient feature based vehicle positioning

• US 10,599,149 B2
• Filed: 01/11/2017
• Issued: 03/24/2020
• Est. Priority Date: 10/09/2015
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: Zhao, Cong
• Primary Examiner(s): Soofi, Yazan A
• Application Number:
  US15/404,038
• Publication Number:
  US 20170123425A1
• Time in Patent Office: 1,168 Days
• Field of Search: 701 3
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0202 specially adapted to aircraft
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones

Abstract

Apparatus, methods, and computer readable mediums for determining a position of a vehicle are provided. Imaging devices may be coupled to movable objects, such as unmanned aerial vehicles (UAVs). One or more images may be captured by an imaging device, such as a camera. Salient features within the one or more images may be determined. A distance from the vehicle to the salient features may be calculated and a position of the vehicle may be determined.

Claims

1. A method of determining a position of a vehicle, said method comprising:

receiving a plurality of images captured by one or more vision sensors, the one or more vision sensors being movable relative to the vehicle;

selecting, with aid of one or more processors, one or more images from the plurality of images;

determining, based on sensor data received from one or more sensors, a movement of the one or more vision sensors relative to the vehicle during the one or more images being taken;

generating a saliency map, with aid of the one or more processors processing the one or more images by filtering according to one or more contrast features of the one or more images and compensating the movement of the one or more vision sensors relative to the vehicle;

measuring, with aid of the one or more processors, a gradient of a gray level for each pixel in one or more portions of the saliency map;

identifying, with aid of the one or more processors, a plurality of stationary salient features in the saliency map according to the gradient, wherein the plurality of stationary salient features are substantially stationary within an environment;

calculating, with aid of the one or more processors, a distance from the vehicle to each of the plurality of stationary salient features; and

determining, with aid of the one or more processors, a position of the vehicle based on the distance from the vehicle to each of the plurality of stationary salient features.

2. The method of claim 1, wherein the step of identifying the plurality of stationary salient features includes (1) identifying, with aid of one or more processors, a plurality of salient features in the saliency map according to the gradient;

and (2) identifying, with aid of the one or more processors, the plurality of stationary salient features from the plurality of salient features.

3. The method of claim 2, wherein the plurality of salient features are identified when the gradient of the gray level of one or more pixels exceeds a threshold.

4. The method of claim 2, wherein the plurality of stationary salient features are identified when there is no positional change for the plurality of salient features in consecutive frames in the plurality of images.

5. The method of claim 4, wherein positions of the plurality of salient features in the consecutive frames are countervailed with one or more of the following:

(1) displacement of the vehicle, or (2) displacement of a carrier that supports the one or more vision sensors relative to a body of the vehicle.

6. The method of claim 1, wherein the plurality of stationary salient features are a plurality of objects or a plurality of salient regions.

7. The method of claim 1, wherein the one or more vision sensors include one or more of the following:

monocular sensor, RGB sensor, and binocular sensor.

8. The method of claim 1, further comprising determining a virtual coordinate in a local coordinate system for each stationary salient feature with the vehicle as the origin of the local coordinate system.

9. The method of claim 8, wherein the virtual coordinate of the stationary salient feature is at a center of the stationary salient feature or a center of mass of the stationary salient feature.

10. The method of claim 8, wherein the local coordinate system is a two-dimensional coordinate system.

11. The method of claim 1, wherein calculating the distance is performed by measuring a depth of the stationary salient features according to a parallax.

12. The method of claim 11, wherein the parallax is observed in consecutive frames in the plurality of images.

13. The method of claim 1, wherein the plurality of images are captured during movement of the vehicle.

14. The method of claim 1, wherein the plurality of images are two or more images captured with a different field of view.

19. The method of claim 1, further comprising:

determining a plurality of stationary salient feature candidates in the saliency map; and

generating a bounding box for each of the stationary salient feature candidates in the saliency map, the bounding box including a triangle shape and encompassing the corresponding stationary salient feature candidate,

wherein the one or more portions of the saliency map includes a portion encompassed by each of the bounding boxes.

15. An apparatus for determining a position of a vehicle, said apparatus comprising:

one or more vision sensors configured to capture a plurality of images and being movable relative to the vehicle; and

one or more processors individually or collectively configured to;

select one or more images from the plurality of images;

determine, based on sensor data received from one or more sensors, a movement of the one or more vision sensors relative to the vehicle during the one or more images being taken;

process the one or more images by filtering according to one or more contrast features of the one or more images to generate a saliency map and compensating the movement of the one or more vision sensors relative to the vehicle;

measure a gradient of a gray level for each pixel in one or more portions of the saliency map;

identify a plurality of stationary salient features in the saliency map according to the gradient, wherein the plurality of stationary salient features are substantially stationary within an environment;

calculate a distance from the vehicle to each of the plurality of stationary salient features; and

determine a position of the vehicle based on the distance from the vehicle to each of the plurality of stationary salient features.

16. The apparatus of claim 15, wherein the vehicle is an unmanned aerial vehicle (UAV).

17. The apparatus of claim 15, wherein the one or more processors are configured to identify the plurality of stationary salient features by obtaining salient features from the saliency map and identifying the plurality of stationary salient features from the salient features.

20. The apparatus of claim 15, wherein the one or more processors are further configured to:

determine a plurality of stationary salient feature candidates in the saliency map; and

generate a bounding box for each of the stationary salient feature candidates in the saliency map, the bounding box including a triangle shape and encompassing the corresponding stationary salient feature candidate,

wherein the one or more portions of the saliency map includes a portion encompassed by each of the bounding boxes.

18. A method of determining a position of a vehicle, said method comprising:

receiving a plurality of images captured by one or more vision sensors;

selecting, with aid of one or more processors, one or more images from the plurality of images;

generating a saliency map, with aid of the one or more processors processing the one or more images by filtering according to one or more contrast features of the one or more images;

determining a plurality of stationary salient feature candidates in the saliency map;

generating a bounding box for each of the stationary salient feature candidates in the saliency map, the bounding box including a triangle shape and encompassing the corresponding stationary salient feature candidate;

measuring, with aid of the one or more processors, a gradient of a gray level for each pixel in the bounding box for each of the stationary salient feature candidates in the saliency map;

identifying, with aid of the one or more processors, a plurality of stationary salient features in the saliency map according to the gradient, wherein the plurality of stationary salient features are substantially stationary within an environment;

calculating, with aid of the one or more processors, a distance from the vehicle to each of the plurality of stationary salient features; and

determining, with aid of the one or more processors, a position of the vehicle based on the distance from the vehicle to each of the plurality of stationary salient features.