Autonomous vehicle and motion control therefor

• US 7,440,585 B2
• Filed: 05/02/2003
• Issued: 10/21/2008
• Est. Priority Date: 11/21/2002
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Son, Young, Kwak, Joo-young, Roh, Kyoung-sig
• Primary Examiner(s): Johns; Andrew W.
• Assistant Examiner(s): Bitar; Nancy
• Application Number:
  US10/427,973
• Publication Number:
  US 20040101161A1
• Time in Patent Office: 1,999 Days
• Field of Search: 359/619, 359/618, 382/103, 382/104, 382/181, 382/154, 382/128, 382/284, 382/305, 356/139.04, 356/4.01, 356/139.07, 348/43
• US Class Current: 382/103
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0259 using magnetic or electromagnetic means
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

There are provided an autonomous vehicle, and an apparatus and method for estimating the motion of the autonomous vehicle and detecting three-dimensional (3D) information of an object appearing in front of the moving autonomous vehicle. The autonomous vehicle measures its orientation using an acceleration sensor and a magnetic flux sensor, and extracts epipolar geometry information using the measured orientation information. Since the corresponding points between images required for extracting the epipolar geometry information can be reduced to two, it is possible to more easily and correctly obtain motion information of the autonomous vehicle and 3D information of an object in front of the autonomous vehicle.

Claims

1. An autonomous vehicle, comprising:

a corresponding point detection unit for detecting corresponding points from at least two consecutive images obtained through a camera

an orientation measuring unit for computing a rotation matrix based on orientation information of the autonomous vehicle;

an epipolar computation unit for computing epipolar geometry information based on the rotation matrix and translation information;

a motion analysis unit for analyzing motion of the autonomous vehicle based on the computed epipolar geometry information; and

a three-dimensional (3D) information analysis unit for analyzing 3D information of the object existing in front of the autonomous vehicle based on the computed epipolar geometry information,

wherein when a relationship between a coordinate system of the camera and a coordinate system of the images is represented by using the rotation matrix and the translation information, the translation information is determined by applying the rotation matrix and coordinates of at least two corresponding points to the relationship, where the orientation information represents orientation information between the two consecutive images, and the translation information represents translation information with respect to an earliest obtained image of the two consecutive images.

2. The autonomous vehicle according to claim 1, wherein the camera has the same coordinate axes as the autonomous vehicle.

3. The autonomous vehicle according to claim 1, wherein the images are obtained through a single camera at a predetermined frame rate.

4. The autonomous vehicle according to claim 1, wherein images are obtained simultaneously through a stereo camera or at least two or more cameras.

5. The autonomous vehicle according to claim 1, wherein the orientation measuring unit includes an acceleration sensor for measuring gravity and a magnetic flux sensor for measuring terrestrial magnetism, and wherein the orientation information of the autonomous vehicle is computed with respect to a fixed ground coordinate system using the absolute values of the values obtained through the acceleration sensor and the magnetic flux sensor.

6. The autonomous vehicle according to claim 1, wherein the epipolar computation unit computes the epipolar geometry information based on the orientation information computed by the orientation measuring unit and the information on at least two corresponding points.

7. The autonomous vehicle according to claim 6, wherein the epipolar S geometry information is computed in accordance with the following equations if a matrix expressing the orientation of the autonomous vehicle with respect to any two consecutive images can be measured:

$\left[\begin{array}{c}f<br><br><br><br>{\stackrel{\_}{v}}_1-\stackrel{\_}{f}<br><br><br><br>v_2\\ f<br><br><br><br>{\stackrel{\_}{u}}_1-\stackrel{\_}{f}<br><br><br><br>u_2\end{array}\right]<br><br>\left[\begin{array}{c}{T}_1\\ T_2\end{array}\right]=[{\stackrel{\_}{u}}_1<br><br>v_2-{\stackrel{\_}{v}}_1<br><br>u_2$

where f is the focal length of the camera, T₁ and T₂ are translation vector elements representing translation information between the two consecutive images, u₁ and ν

₁ are x and y coordinates on a plane of the image obtained earlier between the two consecutive images, and u₂ and ν

₂ are x and y coordinates on a plane of the image obtained later between the two consecutive images.

8. The autonomous vehicle according to claim 1, wherein the 3D information of the object in front of the autonomous vehicle includes information on a 3D shape of the object.

9. An apparatus for estimating the motion of an autonomous vehicle, comprising:

a corresponding point detection unit for detecting corresponding points from at least two consecutive images obtained through a camera;

an orientation measuring unit for computing orientation information of the autonomous vehicle;

an epipolar computation unit for computing epipolar geometry information based on the rotation matrix and translation information; and

a motion analysis unit for analyzing the motion of the autonomous vehicle based on the computed epipolar geometry information,

wherein when a relationship between a coordinate system of the camera and a coordinate system of the images is represented by using the rotation matrix and the translation information, the translation information is determined by applying the rotation matrix and coordinates of at least two corresponding points to the relationship, where the orientation information represents orientation information between the two consecutive images, and the translation information represents translation information with respect to an earliest obtained image of the two consecutive images.

10. The apparatus for estimating the motion of an autonomous vehicle according to claim 9, wherein the camera has the same coordinate axes with the autonomous vehicle.

11. The apparatus for estimating the motion of an autonomous vehicle according to claim 9, wherein the images are obtained through a single camera at a predetermined frame rate.

12. The apparatus for estimating the motion of an autonomous vehicle according to claim 9, wherein images are obtained simultaneously through a stereo camera or at least two or more cameras.

13. The apparatus for estimating the motion of an autonomous vehicle according to claim 9, wherein the orientation measuring unit includes an acceleration sensor for measuring gravity and a magnetic flux sensor for measuring terrestrial magnetism, and wherein the orientation information of the autonomous vehicle is computed with respect to a fixed ground coordinate system using the absolute values of the values obtained through the acceleration sensor and the magnetic flux sensor.

14. The apparatus for estimating the motion of an autonomous vehicle according to claim 9, wherein the epipolar computation unit computes the epipolar geometry information based on the orientation information computed by the orientation measuring unit and information on at least two corresponding points.

15. The apparatus for estimating the motion of an autonomous vehicle according to claim 14, wherein the epipolar geometry information is computed in accordance with the following equations if a matrix expressing the orientation of the autonomous vehicle with respect to any two consecutive images can be measured:

$\left[\begin{array}{c}f<br><br><br><br>{\stackrel{\_}{v}}_1-\stackrel{\_}{f}<br><br><br><br>v_2\\ f<br><br><br><br>{\stackrel{\_}{u}}_1-\stackrel{\_}{f}<br><br><br><br>u_2\end{array}\right]<br><br>\left[\begin{array}{c}{T}_1\\ T_2\end{array}\right]=[{\stackrel{\_}{u}}_1<br><br>v_2-{\stackrel{\_}{v}}_1<br><br>u_2$

where f is the focal length of the camera, T₁ and T₂ are translation vector elements representing translation information between the two consecutive images, u₁ and ν

₁ are x and y coordinates on a plane of the image obtained earlier between the two consecutive images, and u₂ and ν

₂ are x and y coordinates on a plane of the image obtained later between the two consecutive images.

16. An apparatus for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle, comprising:

a corresponding point detection unit for detecting corresponding points from at least two consecutive images obtained through a camera;

an orientation measuring unit for computing a rotation matrix based on orientation information of the autonomous vehicle;

an epipolar computation unit for computing epipolar geometry information based on the rotation matrix and translation information; and

a 3D information analysis unit for analyzing 3D information of the object existing in front of the autonomous vehicle based on the computed epipolar geometry information,

wherein when a relationship between a coordinate system of the camera and a coordinate system of the images is represented by using the rotation matrix and the translation information, the translation information is determined by applying the rotation matrix and coordinates of at least two corresponding points to the relationship, where the orientation information represents orientation information between the two consecutive images, and the translation information represents translation information with respect to an earliest obtained image of the two consecutive images.

17. The apparatus for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 16, wherein the camera has the same coordinate axes with the autonomous vehicle.

18. The apparatus for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 16, wherein the images are obtained through a single camera at a predetermined frame rate.

19. The apparatus for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 16, wherein images are obtained simultaneously through a stereo camera or at least two or more cameras.

20. The apparatus for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 16, wherein the orientation measuring unit includes an acceleration sensor for measuring gravity and a magnetic flux sensor for measuring terrestrial magnetism, and wherein the orientation information of the autonomous vehicle is computed with respect to a fixed ground coordinate system using the absolute values of the values obtained through the acceleration sensor and the magnetic flux sensor.

21. The apparatus for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 16, wherein the epipolar computation unit computes the epipolar geometry information based on the orientation information computed by the orientation measuring unit and the information on at least two corresponding points.

22. The apparatus for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 21, wherein the epipolar geometry information is computed in accordance with the following equations if a matrix expressing the orientation of the autonomous vehicle with respect to any two consecutive images can be measured:

$\left[\begin{array}{c}f<br><br><br><br>\stackrel{\_}{v_1}-\stackrel{\_}{f}<br><br><br><br>v_2\\ f<br><br><br><br>\stackrel{\_}{u_1}-\stackrel{\_}{f}<br><br><br><br>u_2\end{array}\right]<br><br>\left[\begin{array}{c}{T}_1\\ T_2\end{array}\right]=[\stackrel{\_}{u_1}<br><br>v_2-\stackrel{\_}{v_1}<br><br>u_2$

where f is the focal length of the camera, T₁ and T₂ are translation vector elements representing translation information between the two consecutive images, u₁ and ν

₁ are x and y coordinates on a plane of the image obtained earlier between the two consecutive images, and u₂ and ν

₂ are x and y coordinates on a plane of the image obtained later between the two consecutive images.

23. The apparatus for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 16, wherein the 3D information of the object in front of the autonomous vehicle includes information on a distance between the autonomous vehicle and the object, and information on a 3D shape of the object.

24. A method for controlling the motion of an autonomous vehicle comprising the steps of:

(a) detecting corresponding points from at least two consecutive images obtained through a camera;

(b) computing a rotation matrix based on orientation information of the autonomous vehicle;

(c) computing epipolar geometry information based on the rotation matrix and translation information;

(d) analyzing the motion of the autonomous vehicle based on the computed epipolar geometry information; and

(e) analyzing three-dimensional information of an object existing in front of the autonomous vehicle based on the computed epipolar geometry information,

wherein when a relationship between a coordinate system of the camera and a coordinate system of the images is represented by using the rotation matrix and the translation information, the translation information is determined by applying the rotation matrix and coordinates of at least two corresponding points to the relationship, where the orientation information represents orientation information between the two consecutive images, and the translation information represents translation information with respect to an earliest obtained image of the two consecutive images.

25. The method for controlling the motion of an autonomous vehicle according to claim 24, wherein the camera has the same coordinate axes with the autonomous vehicle.

26. The method for controlling the motion of an autonomous vehicle according to claim 24, wherein the images are obtained through a single camera at a predetermined frame rate.

27. The method for controlling the motion of an autonomous vehicle according to claim 24, wherein images are obtained simultaneously through a stereo camera or at least two or more cameras.

28. The method for controlling the motion of an autonomous vehicle according to claim 24, wherein step (b) includes:

(b-i) measuring gravity;

(b-2) measuring terrestrial magnetism; and

(b-3) computing the orientation information of the autonomous vehicle with respect to a fixed ground coordinate system using the absolute values of the values obtained in steps (b-1) and (b-2).

29. The method for controlling the motion of an autonomous vehicle according to claim 24, wherein step (c) includes computing the epipolar geometry information based on the orientation information computed in step (b) and the information on at least two corresponding points.

30. The method for controlling the motion of an autonomous vehicle according to claim 29, wherein the epipolar geometry information is computed in accordance with the following equations if a matrix expressing the orientation of the autonomous vehicle with respect to any two consecutive images can be measured:

$\left[\begin{array}{c}f<br><br><br><br>\stackrel{\_}{v_1}-\stackrel{\_}{f}<br><br><br><br>v_2\\ f<br><br><br><br>\stackrel{\_}{u_1}-\stackrel{\_}{f}<br><br><br><br>u_2\end{array}\right]<br><br>\left[\begin{array}{c}{T}_1\\ T_2\end{array}\right]=[\stackrel{\_}{u_1}<br><br>v_2-\stackrel{\_}{v_1}<br><br>u_2$

where f is the focal length of the camera, T₁ and T₂ are translation vector elements representing translation information between the two consecutive images, u₁ and ν

₁ are x and y coordinates on a plane of the image obtained earlier between the two consecutive images, and u₂ and ν

₂ are x and y coordinates on a plane of the image obtained later between the two consecutive images.

31. The method for controlling the motion of an autonomous vehicle according to claim 24, wherein the 3D information of the object in front of the autonomous vehicle includes information on a distance between the autonomous vehicle and the object, and information on a 3D shape of the object.

47. A computer readable recording medium that stores a computer program to implement a method according to claim 24.

32. A method for estimating the motion of an autonomous vehicle, comprising the steps of:

(a) detecting corresponding points from at least two consecutive images obtained through a camera;

(b) computing a rotation matrix based on orientation information of the autonomous vehicle;

(c) computing epipolar geometry information based on the rotation matrix and translation information; and

(d) analyzing motion of the autonomous vehicle based on the computed epipolar geometry information,

wherein when a relationship between a coordinate system of the camera and a coordinate system of the images is represented by using the rotation matrix and the translation information, the translation information is determined by applying the rotation matrix and coordinates of at least two corresponding points to the relationship, where the orientation information represents orientation information between the two consecutive images, and the translation information represents translation information with respect to an earliest obtained image of the two consecutive images.

33. The method for estimating the motion of an autonomous vehicle according to claim 32, wherein the camera has the same coordinate axes with the autonomous vehicle.

34. The method for estimating the motion of an autonomous vehicle according to claim 32, wherein the images are obtained through a single camera at a predetermined frame rate.

35. The method for estimating the motion of an autonomous vehicle according to claim 32, wherein images are obtained simultaneously through a stereo camera or at least two or more cameras.

36. The method for estimating the motion of an autonomous vehicle according to claim 32, wherein step (b) includes:

(b-1) measuring gravity;

(b-2) measuring terrestrial magnetism; and

(b-3) computing the orientation information of the autonomous vehicle with respect to a fixed ground coordinate system using the absolute values of the values obtained in steps (b-1) and (b-2).

37. The method for estimating the motion of an autonomous vehicle according to claim 32, wherein step (c) includes computing the epipolar geometry information based on the orientation information computed in step (b) and the information on at least two corresponding points.

38. The method for estimating the motion of an autonomous vehicle according to claim 37, wherein the epipolar geometry information is computed in accordance with the following equations if a matrix expressing the orientation of the autonomous vehicle with respect to any two consecutive images can be measured:

$\left[\begin{array}{c}f<br><br><br><br>\stackrel{\_}{v_1}-\stackrel{\_}{f}<br><br><br><br>v_2\\ f<br><br><br><br>\stackrel{\_}{u_1}-\stackrel{\_}{f}<br><br><br><br>u_2\end{array}\right]<br><br>\left[\begin{array}{c}{T}_1\\ T_2\end{array}\right]=[\stackrel{\_}{u_1}<br><br>v_2-\stackrel{\_}{v_1}<br><br>u_2$

where f is the focal length of the camera, T₁ and T₂ are translation vector elements representing translation information between the two consecutive images, u₁ and ν

₁ are x and y coordinates on a plane of the image obtained earlier between the two consecutive images, and u₂ and ν

₂ are x and y coordinates on a plane of the image obtained later between the two consecutive images.

48. A computer readable recording medium that stores a computer program to implement a method according to claim 32.

39. A method for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle, comprising the steps of:

(a) detecting corresponding points from at least two consecutive images obtained through a camera;

(b) computing a rotation matrix based on orientation information of the autonomous vehicle;

(c) computing epipolar geometry information based on the rotation matrix and translation information; and

(d) analyzing 3D information of the object existing in front of the autonomous vehicle based on the computed epipolar geometry information,

wherein when a relationship between a coordinate system of the camera and a coordinate system of the images is represented by using the rotation matrix and the translation information, the translation information is determined by applying the rotation matrix and coordinates of at least two corresponding points to the relationship, where the orientation information represents orientation information between the two consecutive images, and the translation information represents translation information with respect to an earliest obtained image of the two consecutive images.

40. The method for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 39, wherein the camera has the same coordinate axes with the autonomous vehicle.

41. The method for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 39, wherein the images are obtained through a single camera at a predetermined frame rate.

42. The method for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 39, wherein images are obtained simultaneously through a stereo camera or at least two or more cameras.

43. The method for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 41 wherein step (b) includes:

(b-1) measuring gravity;

(b-2) measuring terrestrial magnetism; and

(b-3) computing the orientation information of the autonomous vehicle with respect to a fixed ground coordinate system using absolute values of the values obtained in steps (b-1) and (b-2).

44. The method for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 39, wherein step (c) includes computing the epipolar geometry information based on the orientation information computed in step (b) and the information on at least two corresponding points.

45. The method for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 44, wherein the epipolar geometry information is computed in accordance with the following equations if a matrix expressing the orientation of the autonomous vehicle with respect to any two consecutive images can be measured:

$\left[\begin{array}{c}f<br><br><br><br>\stackrel{\_}{v_1}-\stackrel{\_}{f}<br><br><br><br>v_2\\ f<br><br><br><br>\stackrel{\_}{u_1}-\stackrel{\_}{f}<br><br><br><br>u_2\end{array}\right]<br><br>\left[\begin{array}{c}{T}_1\\ T_2\end{array}\right]=[\stackrel{\_}{u_1}<br><br>v_2-\stackrel{\_}{v_1}<br><br>u_2$

where f is the focal length of the camera, T₁ and T₂ are translation vector elements representing translation information between the two consecutive images, u₁ and ν

₁ are x and y coordinates on a plane of the image obtained earlier between the two consecutive images, and u₂ and ν

₂ are x and y coordinates on a plane of the image obtained later between the two consecutive images.

46. The method for detecting three-dimensional (3D) information of an object existing in front of an autonomous vehicle according to claim 39, wherein the 3D information of the object in front of the autonomous vehicle includes-information on a distance between the autonomous vehicle and the object, and information on a 3D shape of the object.

49. A computer readable recording medium that stores a computer program to implement a method according to claim 39.