Apparatus and method for estimating location of mobile body and generating map of mobile body environment using upper image of mobile body environment, and computer readable recording medium storing computer program controlling the apparatus

• US 8,831,872 B2
• Filed: 01/13/2006
• Issued: 09/09/2014
• Est. Priority Date: 01/25/2005
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Hong, Sungi, Lee, Hyoungki, Myeong, Hyeon
• Primary Examiner(s): MUSTAFA, IMRAN K
• Application Number:
  US11/331,058
• Publication Number:
  US 20060165276A1
• Time in Patent Office: 3,161 Days
• Field of Search: 701/209, 382/153
• US Class Current: 701/409
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30

Abstract

An apparatus and method for estimating a location of a mobile body and generating a map of a mobile body environment using an upper image of the mobile body environment, and a computer readable recording medium storing a computer program for controlling the apparatus. The apparatus includes: a landmark generating unit observing corner points from the upper image obtained by photographing in an upper vertical direction in the mobile body environment and respectively generating landmarks from the observed corner points; and a location and map operation unit estimating the location of the mobile body and generating the map of the mobile body environment from the landmarks.

Claims

1. An apparatus for estimating a location of a mobile body and generating a map of a mobile body environment using an upper image of the mobile body environment, the apparatus comprising:

a landmark generating unit to observe corner points from the upper image obtained by photographing in an upper vertical direction in the mobile body environment and to respectively generate landmarks from the observed corner points; and

a location and map operation unit to estimate the location of the mobile body and to generate the map of the mobile body environment from the landmarks,

wherein the landmark generating unit selects one or more landmarks and one or more landmark candidates that belong to a previous upper image, among previously registered landmarks and landmark candidates, predicts locations of the selected landmarks and the selected landmark candidates in a current upper image, and searches for corner points that have been used to respectively generate landmarks or landmark candidates that belong to the previous upper image, among the observed corner points by using coordinates of the corner points that have been observed at the current upper image and local feature descriptors that classify the corner points, and registers the landmark candidate that has been observed as the corner point more than a predetermined number of times among the landmark candidates as a landmark,

wherein the landmark candidate is distinguished from the corner point, the landmark and the landmark candidate,

wherein the landmark candidate is registered among the corner points that have not been used to generate both the landmark and the landmark candidate in the previous upper image by using the predicted locations of the selected landmarks and the selected landmark candidates,

wherein the corner point is registered as the landmark candidate when a distance from the corner point to the landmark or the landmark candidate is greater than a first predetermined distance or when a distance from the local feature descriptor of the corner point and the local feature descriptor of the landmark or the landmark candidate is greater than a second predetermined distance, and

wherein the current upper image is an upper image that has been obtained currently and the previous upper image is an upper image that has been obtained previously.

2. The apparatus of claim 1, further comprising an odometry processing unit to control an odometry of the mobile body and to output information about the odometry,

wherein the location and map operation unit estimates the location of the mobile body and generates the map of the mobile body environment from the information about the odometry inputted from the odometry processing unit and the landmark inputted from the landmark generating unit and outputs information about the landmark to the landmark generating unit.

3. The apparatus of claim 1, wherein the landmark generating unit comprises:

an image obtaining unit to obtain an image in the upper vertical direction in the mobile body environment;

an image buffer to buffer the obtained image inputted from the image obtaining unit and to output the buffered result as the previous upper image;

an image feature information extracting unit to extract at least one of coordinates of the corner points of the current upper image that has been currently obtained by the image obtaining unit and the local feature descriptors as image feature information;

a landmark and landmark candidate predicting unit to predict the landmark and the landmark candidate in the previous upper image from the current upper image;

an observing and registering unit to compare information about the predicted landmark and landmark candidate with the image feature information, to observe and search for corner points that have been used to respectively generate landmarks or landmark candidates in the previous upper image among the corner points using the compared result, and to register the corner points that have not been used to generate the landmarks or the landmark candidates in the previous upper image as new landmark candidates; and

a landmark registering unit to register the landmark candidate that has been observed as the corner point more than a predetermined number of times among the landmark candidates as the landmark.

4. The apparatus of claim 3, wherein the image obtaining unit comprises:

an upper image obtaining unit to obtain an image in the upper vertical direction of the mobile body environment; and

a distortion calibrating unit to calibrate a distortion of the obtained upper image.

5. The apparatus of claim 4, wherein the image obtaining unit further comprises a low pass filter to low pass filter the distortion-calibrated result.

6. The apparatus of claim 4, wherein the distortion calibrating unit sets a region of interest in the obtained upper image and calibrates the distortion only in the region of interest.

7. The apparatus of claim 3, wherein the image obtaining unit comprises:

an upper image obtaining unit to obtain an image in the upper vertical direction of the mobile body environment; and

a low pass filter to low pass filter the obtained upper image.

8. The apparatus of claim 3, wherein the image feature information extracting unit comprises:

a corner point detecting unit to detect the corner points from the current upper image and to output coordinates of the detected corner points; and

a descriptor operation unit to analyze images around the detected corner points and to perform an operation on the local feature descriptors at each of the corner points using the analyzed result.

9. The apparatus of claim 8, wherein the corner point detecting unit determines a point at which a value R is greater than 0 and which becomes a local maximum as the corner point, the value R being given by:

R=det(M)−

k(tr(M))²

wherein tr is a matrix trace, det is a matrix determinant, and M is given by;

10. The apparatus of claim 9, wherein k is 0.04.

11. The apparatus of claim 8, wherein the local feature descriptor on which an operation has been performed by the descriptor operation unit is a scale invariant feature transform descriptor.

12. The apparatus of claim 3, wherein the landmark and landmark candidate predicting unit comprises:

an image coordinate converting unit to convert three-dimensional coordinates of locations of the landmark and the landmark candidate that have been registered in the previous upper image into two-dimensional coordinates based on an image coordinate system of the previous upper image using the location and the azimuth angle of the mobile body when the previous upper image is inputted;

a selecting unit to select only the landmark and the landmark candidate that belong to the previous upper image, among the landmarks and the landmark candidates having the converted two-dimensional coordinates inputted from the image coordinate converting unit; and

a location predicting unit to calculate the amount of variation between the current upper image inputted from the image obtaining unit and the previous upper image inputted from the image buffer using the selected result and to predict the landmark and the landmark candidate that may exist in the current upper image using the calculated amount of variation.

13. The apparatus of claim 12, wherein the location predicting unit predicts locations of the landmark and the landmark candidate that may exist in the current upper image using an optical flow tracker.

14. The apparatus of claim 3, wherein the observing and registering unit comprises:

a landmark and landmark candidate searching unit to search for the nearest landmark or landmark candidate to the corner point, with regard to each of the corner points;

a first distance calculating unit to calculate a distance between the corner point and the searched landmark or landmark candidate, with regard to each of the corner points;

a first comparing unit to compare the distance calculated by the first distance calculating unit with a first predetermined distance;

a second distance calculating unit to calculate a distance between a local feature descriptor of the corner point and a local feature distance of the searched landmark or landmark candidate in response to the result compared by the first comparing unit;

a second comparing unit to compare the distance calculated by the second distance calculating unit with a second predetermined distance;

a landmark and landmark candidate identifying unit to identify whether the landmark or the landmark candidate exists at the nearest location to the corner point in response to the result compared by the second comparing unit;

a counter to count a landmark flag that indicates how many times the landmark has been observed, in an ascending series or counting a landmark flag that indicates how many times the landmark candidate has been observed, in an ascending series, in response to the identified result; and

a landmark candidate initializing unit to search for the corner points that have not been used to generate the landmark or the landmark candidate in the previous upper image among the corner points as the new landmark candidate and to register information about the new landmark candidate in response to the results compared by the first and second comparing units.

15. The apparatus of claim 3, wherein the landmark generating unit further comprises a landmark candidate processing unit to remove overlapping landmark candidates from the registered landmark candidates and to estimate a three-dimensional image coordinate from a two-dimensional image coordinate of non-overlapping landmark candidates, and

wherein the landmark registering unit registers landmark candidates that have been observed as the corner points more than the predetermined number of times among the non-overlapping landmark candidates as the landmarks.

16. The apparatus of claim 15, wherein the landmark candidate processing unit comprises:

a third distance calculating unit to calculate a minimum distance between the nearest landmark candidates, with regard to each of the registered landmark candidates;

a third comparing unit to compare each minimum distance with a third predetermined distance;

a landmark candidate removing unit to determine whether the landmark candidates are overlapped with one another in response to the result compared by the third comparing unit, and to remove the overlapping landmark candidates in response to the determining result; and

a coordinate estimating unit to estimate a three-dimensional image coordinate of the non-overlapping landmark candidates, in response to the result compared by the third comparing unit.

17. The apparatus of claim 1, wherein the location and map operation unit estimates the location of the mobile body from the landmarks and generates the map of the mobile body environment using an extended Kalman filter.

18. The apparatus of claim 1, wherein the landmark generating unit registers the landmark candidate or the landmark by using pairs which bind the predicted and nearest landmarks or landmark candidates to the respective corner points.

19. The apparatus of claim 1, wherein the landmark generating unit comprises a landmark and landmark candidate predicting unit to predict the landmark and the landmark candidate in the previous upper image from the current upper image.