Localization system and method of mobile robot based on camera and landmarks

• US 20070150097A1
• Filed: 08/23/2006
• Published: 06/28/2007
• Est. Priority Date: 12/08/2005
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Cho, Young Jo, Yu, Won Pil, Lee, Jae Young, Chae, Heesung
• Granted Patent: US 7,634,336 B2
• Time in Patent Office: Days
• US Class Current: 700/245
• CPC Class Codes: G05D 1/0234 using optical markers or beacons optical beacons per se G01S1/70
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30

Abstract

A localization system and method of a mobile robot using a camera and artificial landmarks in a home and a general office environment (or working zone) is provided. The localization system includes artificial landmarks having an LED flash function in an invisible wavelength band, a camera with a wide-angle lens, a module flashing landmarks attached at the ceiling and identifying positions and IDs of the landmarks from an image photographed by the camera having a filter, a module calculating position and orientation of the robot using two landmarks of the image in a stop state, a module, when a ceiling to which the landmarks are attached has different heights, a position of the robot, and a module, when a new landmark is attached in the working zone, calculating a position of the new landmark on an absolute coordinate.

Claims

1. A localization system of a mobile robot, comprising:

n number of landmarks attached at a ceiling and having wireless transmitting and receiving functions,

a camera photographing the n number of landmarks;

a landmark detection part which flashes the landmarks attached at the ceiling and obtains positions and IDs of the landmarks from an image photographed by the camera to detect at least two landmarks;

a robot position detection part calculating the position of the mobile robot using the detected landmarks;

a landmark position prediction part which, when a new landmark is attached within a working zone, calculates a position of the new landmark on an absolute coordinate;

a topology map building part which builds a topology map of the mobile robot using the calculated position of the new landmark; and

a robot controller controlling a navigation of the mobile robot using the built topology map.

2. The localization system of claim 1, wherein the n number of landmarks each have a specific ID built therein, comprises an infrared LED emitting light in a specific wavelength band, and wireless transmitting and receiving modules, and the camera comprises a camera filter which passes a light with the specific wavelength band to detect the infrared LED of the landmark.

3. The localization system of claim 1, wherein the landmark detection part detects at least two landmarks existing on an image of the camera by applying a binary search to the n number of landmarks attached at the ceiling.

4. The localization system of claim 1, wherein the robot position detection part comprises:

a first module calculating current position and orientation of the mobile robot using an image of the detected two landmarks when the mobile robot is in a stop state;

a second module calculating a position of the mobile robot in real time when the mobile robot is in a moving state; and

a third module calculating a position of the mobile robot when the ceiling to which the n number of landmarks are attached has different heights.

5. The localization system of claim 4, wherein the first module calculates the current position ‘

Pw’

of the mobile robot using the below equation 12;

P_w=R_we·

s(R_ie·

(P_i−

T_ie))+T_we 

 

(12),

where P_w is a real position of the mobile robot on a world coordinate that is the calculated final position of the mobile robot, P_i is a position of the mobile robot on an image coordinate, R_we is a rotational vector informing how much an extra coordinate rotationally moves with respect to the world coordinate, R_ie is a rotational vector informing how much the image coordinate rotationally moves with respect to the extra coordinate, T_ie is a parallel movement vector informing how much the image coordinate moves in parallel with respect to the extra coordinate, T_we is a parallel movement vector informing how much the extra coordinate moves in parallel with respect to the world (absolute) coordinate, and s is a distance ratio between the two landmarks on the image and the real two landmarks.

6. The localization system of claim 5, wherein the first module calculate the orientation ‘

θ

_r’

of the mobile robot using the below equation 8;

$\begin{array}{cc}{\mathrm{\theta <br><br>}}_r={\mathrm{\theta <br><br>}}_2-{\mathrm{\theta <br><br>}}_1+\frac{\mathrm{\pi <br><br>}}{2}.& \left(8\right)\end{array}$

7. The localization system of claim 4, wherein the second module calculates the position of the mobile robot by setting a mask with a constant size based on the image coordinates of the obtained two landmarks, and searching only an area of the mask set for the image coordinates of the landmarks after the moving of the mobile robot.

8. The localization system of claim 4, wherein the third module calculates the position of the mobile robot by correcting the image coordinates of the n number of landmarks using the below equation 11:

$\begin{array}{cc}{P}^{\mathrm{\prime <br><br>}}<br><br>\left(u_j<br><br>v_j\right)=\frac{P·<br><br>h_1}{h_2},& \left(11\right)\end{array}$

where P′

is the position of the mobile robot obtained from the corrected image coordinates of the n number of landmarks, P is the position of the mobile robot obtained from the n number of landmarks before being corrected, and h1 and h2 are heights of the ceiling.

9. The localization system of claim 1, wherein when a new landmark is attached in the working zone, the landmark position prediction part identifies an ID of the new landmark, calculates a position ‘

Pw’

of the mobile robot using the below equation 12 and calculates a position ‘

G_w’

of the new landmark on the world coordinate by the below equation 13;

P_w=R_we·

s(R_ie·

(P_i−

T_ie))+T_we 

 

(12),

G_w=R_we·

s(R_ie(G_i−

T_ie))+T_we 

 

(13),

where P_w is a real position of the mobile robot on a world coordinate that is the calculated final position of the mobile robot, P_i is a position of the mobile robot on an image coordinate, R_we is a rotational vector informing how much an extra coordinate rotationally moves with respect to the world coordinate, R_ie is a rotational vector informing how much the image coordinate rotationally moves with respect to the extra coordinate, T_ie is a parallel movement vector informing how much the image coordinate moves in parallel with respect to the extra coordinate, T_we is a parallel movement vector informing how much the extra coordinate moves in parallel with respect to the world (absolute) coordinate, s is a distance ratio between the two landmarks on the image and the real two landmarks, Gi is a position of the new landmark on the image coordinate, and Gw is a real position (i.e., a position on the world coordinate) of the new landmark.

10. The localization system of claim 9, wherein the topology map building part builds the topology map by setting the landmarks as nodes and obtaining a distance information between the nodes.

11. The localization system of claim 2, wherein the LED of each of the n number of landmarks is flashed by each of the n number of landmarks and a wireless communication function of the camera.

12. A localization method of a mobile robot using a camera and a landmark, the localization method comprising the steps of

(a) detecting the landmark from an image of the landmark photographed by the camera to calculate a position of a mobile robot;

(b) when a new landmark is added to a working zone, obtaining a position of the added landmark using the calculated position of the mobile robot;

(c) setting the added landmark as a node to build a topology map; and

(d) controlling a navigation of the mobile robot using the built topology map and the calculated position of the mobile robot.

13. The localization method of claim 12, wherein the step (a) comprises:

(e) acquiring an ID of the landmark using a wireless transmitting and receiving module of the mobile robot, flashing the landmark and photographing an image of the landmark of a ceiling;

(f) performing a binary search for the photographed image of the landmark to detect at least two landmarks; and

(g) calculating position and orientation of the mobile robot using the detected at least two landmarks.

14. The localization method of claim 13, wherein in the step (e), the landmark comprises an infrared LED and the camera comprises an infrared filter which passes only a specific wavelength band, and the mobile robot flashes the infrared LED and then photographs the landmark.

15. The localization method of claim 13, wherein the step (g) comprises:

when the mobile robot is in a stop state, calculating a final position ‘

P_w’

of the mobile robot using the detected at least two landmarks through the below equation 12;

P_w=R_we·

s(R_ie·

(P_i−

T_ie))+T_we 

 

(12),

when the mobile robot is in a moving state, calculating the position of the mobile robot by setting a mask with a predetermined size based on image coordinates of the at least two landmarks and searching an area of the mask set for the image coordinates of the landmarks after a moving of the mobile robot,

where P_w is a real position of the mobile robot on a world coordinate that is the calculated final position of the mobile robot, P_i is a position of the mobile robot on an image coordinate, R_we is a rotational vector informing how much an extra coordinate rotationally moves with respect to the world coordinate, R_ie is a rotational vector informing how much the image coordinate rotationally moves with respect to the extra coordinate, T_ie is a parallel movement vector informing how much the image coordinate moves in parallel with respect to the extra coordinate, T_we is a parallel movement vector informing how much the extra coordinate moves in parallel with respect to the world (absolute) coordinate, and s is a distance ratio between the two landmarks on the image and the real two landmarks.

16. The localization method of claim 15, wherein the step (g) comprises, when the ceiling to which the landmarks are attached has different heights, calculating the position of the mobile robot by correcting the image coordinates of the landmarks through the below equation 11:

$\begin{array}{cc}{P}^{\mathrm{\prime <br><br>}}<br><br>\left(u_j<br><br>v_j\right)=\frac{P·<br><br>h_1}{h_2},& \left(11\right)\end{array}$

where P′

is the position of the mobile robot obtained from the corrected image coordinates of the n number of landmarks, P is the position of the mobile robot obtained from the n number of landmarks before being corrected, and h1 and h2 are heights of the ceiling.