STEREO CAMERA-BASED AUTONOMOUS DRIVING METHOD AND APPARATUS

• US 20180052457A1
• Filed: 02/23/2017
• Published: 02/22/2018
• Est. Priority Date: 08/16/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: KIM, Changhyun, KANG, Hyoa, SHIN, Changwoo, CHO, Baek Hwan, JI, Derek Daehyun
• Granted Patent: US 10,444,752 B2
• Time in Patent Office: Days
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G05D 1/0055 with safety arrangements
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Disclosed is a stereo camera-based autonomous driving method and apparatus, the method including estimating a driving situation of a vehicle, determining a parameter to control a stereo camera width of a stereo camera based on the estimated driving situation, controlling a capturer configured to control arrangement between two cameras of the stereo camera for a first direction based on the determined parameter, and measuring a depth of an object located in the first direction based on two images respectively captured by the two cameras with the controlled arrangement.

Claims

1. An autonomous driving method comprising:

estimating a driving situation of a vehicle;

determining a parameter to control a stereo camera width of a stereo camera based on the estimated driving situation;

controlling a capturer configured to control arrangement between two cameras of the stereo camera for a first direction based on the determined parameter; and

measuring a depth of an object located in the first direction based on two images respectively captured by the two cameras with the controlled arrangement.

2. The method of claim 1, wherein the controlling of the capturer includes selecting the two cameras, from among a plurality of cameras facing the first direction in a camera array of the capturer, that are separated apart from each other by a distance corresponding to the determined parameter.

3. The method of claim 2, wherein the plurality of cameras are fixedly arranged in the capturer and each linearly separated by a first distance, and the distance corresponding to the determined parameter is evenly divisible by the first distance.

4. The method of claim 1, wherein the two cameras are configured in the capturer to be selectively moved by the capturer, and

the controlling of the capturer includes selectively moving at least one of the two cameras such that the two cameras are separated apart from each other by a distance corresponding to the determined parameter.

5. The method of claim 4, wherein the two cameras are configured in the capturer to be selectively moved by the capturer along a predetermined route, and

the controlling of the capturer includes selectively moving the at least one of the two cameras along the predetermined route such that the two cameras are separated apart from each other by the distance corresponding to the determined parameter.

6. The method of claim 1, wherein the determining of the parameter includes:

determining a stopping distance of the vehicle based on the estimated driving situation; and

determining the parameter based on a predetermined constraint associated with the stopping distance.

7. The method of claim 6, wherein the predetermined constraint associated with the stopping distance is based on consideration of a maximum measurable depth corresponding to the stereo camera width and a relationship between the stopping distance and a safe distance based on a depth error of the maximum measurable depth.

8. The method of claim 1, wherein the determining of the parameter includes:

determining a threshold for a blind spot area based on the estimated driving situation; and

determining the parameter based on a predetermined constraint associated with the threshold for the blind spot area.

9. The method of claim 8, wherein the determining of the threshold for the blind spot area includes determining a threshold blind spot distance based on the estimated driving situation, and the predetermined constraint is associated with the threshold blind spot distance.

10. The method of claim 9, wherein the predetermined constraint associated with the threshold blind spot distance is based on consideration of a relationship between the threshold blind spot distance and a blind spot distance corresponding to the stereo camera width.

11. The method of claim 1, wherein the determining of the parameter includes:

determining a stopping distance and a threshold for a blind spot area based on the estimated driving situation;

determining a minimum value for the parameter based on a predetermined constraint associated with the stopping distance;

determining a maximum value for the parameter based on a predetermined constraint associated with the threshold for the blind spot area; and

determining the parameter to be a value equal to either or between the minimum value and the maximum value.

12. The method of claim 11, wherein the determining of the threshold includes determining a threshold blind spot distance, and the predetermined constraint is associated with the threshold blind spot distance.

13. The method of claim 1, wherein, when the estimated driving situation is the vehicle driving at high speed, the determining of the parameter includes determining the parameter such that a depth error of a maximum measurable depth corresponding to the stereo camera width is reduced over a depth error of another maximum measurable depth corresponding another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a lower speed.

14. The method of claim 1, wherein, when the estimated driving situation is the vehicle driving at low speed, the determining of the parameter includes determining the parameter such that a blind spot distance corresponding to the stereo camera width is reduced over a blind spot distance corresponding another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a higher speed.

15. The method of claim 1, wherein the estimated driving situation is determined based on at least one of a determination of a speed of the vehicle, a moving direction of the vehicle, a location of the vehicle, or a user manipulation.

16. The method of claim 1, wherein the estimated driving situation includes at least one of the vehicle driving straight ahead, changing lanes, changing direction, driving on a highway, driving at high speed, driving at low speed, driving in a city, driving with a weather affecting condition, or driving with a road affecting condition, and

the determining of the parameter includes determining the parameter based on the estimated driving situation and corresponding prioritized consideration between an extent of depth error of a maximum measurable depth corresponding to the stereo camera width and an extent of a blind spot area corresponding to the stereo camera width.

17. The method of claim 1, further comprising:

predicting a driving direction of the vehicle based on the estimated driving situation,

wherein the determining of the parameter includes determining the parameter based on a visual field direction of the stereo camera corresponding to the predicted driving direction.

18. The method of claim 17, further comprising selectively turning off one or more additional stereo cameras of the vehicle respectively for one or more other visual field directions that are determined not needed for driving of the vehicle in the predicted driving direction.

19. The method of claim 1, further comprising:

controlling the capturer or another capturer so that two additional cameras perform stereo capturing in a second direction, opposite to the first direction, with a stereo camera width based on the determined parameter; and

measuring a depth of an object located in the second direction based on output images of the two additional cameras,

wherein the capturer or the other capturer is configured to perform capturing in the second direction in coordination with the captured two images in the first direction.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

21. An autonomous driving apparatus comprising:

a capturer including at least two cameras, the capturer being configured to control arrangement between the two cameras; and

a controller configured to determine a parameter related to a stereo camera width, of a stereo camera represented by the two cameras, based on an estimated driving situation of a vehicle, configured to control the capturer to control the arrangement between the two cameras based on the determined parameter, and configured to measure a depth of an object located in a first direction based on two images respectively captured by the two cameras with the controlled arrangement.

22. The apparatus of claim 21, wherein the controller is further configured to determine the estimated driving situation based on sensed information or one or more driving situation estimations from sensors of the vehicle and/or from sensors or devices external of the vehicle.

23. The apparatus of claim 21, wherein the capturer includes a camera array including a plurality of cameras facing in the first direction, and

the controller is configured to select, from among the plurality of cameras, the two cameras that are separated apart from each other by a distance corresponding to the determined parameter.

24. The apparatus of claim 23, wherein the plurality of cameras are fixedly arranged in the capturer and each linearly separated by a first distance, and the distance corresponding to the determined parameter is evenly divisible by the first distance.

25. The apparatus of claim 21, wherein the two cameras are configured in the capturer to be selectively moved by the capturer, and

the controller is configured to control the capturer to move one or more of the two cameras such that the two cameras are separated from each other by a distance corresponding to the determined parameter.

26. The apparatus of claim 25, wherein the two cameras are configured in the capturer to be selectively moved by the capturer along a predetermined route, and

the controller is configured to control the capturer to move the one or more of the two cameras along the predetermined route such that the two cameras are separated from each other by the distance corresponding to the determined parameter.

27. The apparatus of claim 21, wherein the controller is configured to determine a stopping distance of the vehicle based on the estimated driving situation and determine the parameter based on a predetermined constraint associated with the stopping distance.

28. The apparatus of claim 27, wherein the predetermined constraint associated with the stopping distance is based on consideration of a maximum measurable depth corresponding to the stereo camera width and a relationship between the stopping distance and a safe distance based on a depth error of the maximum measurable depth.

29. The apparatus of claim 21, wherein the controller is configured to determine a threshold for a blind spot area based on the estimated driving situation and determine the parameter based on a predetermined constraint associated with the threshold for the blind spot area.

30. The apparatus of claim 29, wherein the determining of the threshold for the blind spot area includes determining a threshold blind spot distance based on the estimated driving situation, and the predetermined constraint is associated with the threshold blind spot distance.

31. The apparatus of claim 30, wherein the predetermined constraint associated with the threshold blind spot distance is based on consideration of a relationship between the threshold blind spot distance and a blind spot distance corresponding to the stereo camera width.

32. The apparatus of claim 21, wherein the controller is configured to determine a stopping distance and a threshold for a blind spot area based on the estimated driving situation, determine a minimum value for the parameter based on a predetermined constraint associated with the stopping distance, determine a maximum value for the parameter based on a predetermined constraint associated with the threshold for the blind spot area, and determine the parameter to be a value equal to either or between the minimum value and the maximum value.

33. The apparatus of claim 32, wherein the determining of the threshold includes determining a threshold blind spot distance, and the predetermined constraint is associated with the threshold blind spot distance.

34. The apparatus of claim 21, wherein, when the estimated driving situation is the vehicle driving at high speed, the controller is configured to determine the parameter such that a depth error of a maximum measurable depth corresponding to the stereo camera width is reduced over a depth error of another maximum measurable depth corresponding to another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a lower speed.

35. The apparatus of claim 21, wherein, when the estimated driving situation is the vehicle driving at low speed, the controller is configured to determine the parameter such that a blind spot distance corresponding to the stereo camera width is reduced over a blind spot distance corresponding to another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a higher speed.

36. The apparatus of claim 21, further comprising:

a sensor configured to sense at least one of a speed of the vehicle, a moving direction of the vehicle, a location of the vehicle, or a user manipulation,

wherein the estimated driving situation is determined based on at least one of the sensed speed of the vehicle, moving direction of the vehicle, location of the vehicle, or user manipulation.

37. The apparatus of claim 21, wherein the estimated driving situation includes at least one of the vehicle driving straight ahead, changing lanes, changing direction, driving on a highway, driving at high speed, driving at low speed, driving in a city, driving with a weather affecting condition, or driving with a road affecting condition, and

the controller is configured to determine the parameter based on the estimated driving situation in prioritized consideration between an extent of depth error of a maximum measurable depth corresponding to the stereo camera width and an extent of a blind spot area corresponding to the stereo camera width.

38. The apparatus of claim 21, wherein the controller is configured to predict a driving direction of the vehicle based on the estimated driving situation and determine the parameter based on a visual field direction of the stereo camera corresponding to the predicted driving direction.

39. The apparatus of claim 38, wherein the controller is configured to selectively turn off one or more additional stereo cameras of the vehicle respectively for one or more other visual field directions that are determined not needed for driving of the vehicle in the predicted driving direction.

40. The apparatus of claim 21, wherein the capturer or another capturer is configured to include two additional cameras to perform stereo capturing in a second direction, opposite the first direction, with a stereo camera width based on the determined parameter, and

the controller is configured to control the capturer or the other capturer in coordination with the captured two images in the first direction so that the two additional cameras perform the capturing in the second direction with the stereo camera width based on the determined parameter, and configured to measure a depth of an object located in the second direction based on output images of the two additional cameras.

41. An autonomous driving apparatus, the apparatus comprising:

a processor configured to determine a parameter to control a stereo camera width of a stereo camera based on the estimated driving situation of a vehicle, control a capturer configured to control arrangement between two cameras of the stereo camera for a first direction based on the determined parameter, and measure a depth of an object located in the first direction based on two images respectively captured by the two cameras with the controlled arrangement.

42. The apparatus of claim 41, further comprising a memory configured to store instructions,

wherein the processor is further configured to execute the instructions to configure the processor to perform the determining of the parameter to control the stereo camera width of the stereo camera based on the estimated driving situation, the controlling of the capturer to control the arrangement between the two cameras of the stereo camera for the first direction based on the determined parameter, and the measuring of the depth of the object located in the first direction based on the two images respectively captured by the two cameras with the controlled arrangement.

43. The apparatus of claim 41, wherein the processor is further configured to determine the estimated driving situation based on sensed information or one or more driving situation estimations from sensors of the vehicle and/or from sensors or devices external of the vehicle.

44. The apparatus of claim 43, further comprising the capturer, which includes the two cameras, and/or the sensors of the vehicle.

45. The apparatus of claim 43, wherein the apparatus includes the capturer and the capturer includes a camera array including a plurality of cameras facing in the first direction, and

the processor is further configured to select, from among the plurality of cameras, the two cameras that are separated apart from each other by a distance corresponding to the determined parameter.

46. The apparatus of claim 45, wherein the plurality of cameras are fixedly arranged in the capturer and each linearly separated by a first distance, and the distance corresponding to the determined parameter is evenly divisible by the first distance.

47. The apparatus of claim 43, wherein the apparatus includes the capturer and the two cameras are configured in the capturer to be selectively moved by the capturer, and

the processor is further configured to control the capturer to move one or more of the two cameras such that the two cameras are separated from each other by a distance corresponding to the determined parameter.

48. The apparatus of claim 47, wherein the two cameras are configured in the capturer to be selectively moved by the capturer along a predetermined route, and

the processor is configured to control the capturer to move the one or more of the two cameras along the predetermined route such that the two cameras are separated from each other by the distance corresponding to the determined parameter.

49. The apparatus of claim 48, wherein, for the selective moving of the two cameras by the capturer, the processor is further configured to control the capturer to rotate at least one of the two cameras in a prioritized direction based on the estimated driving situation.

50. The apparatus of claim 43, wherein the apparatus includes the capturer and the two cameras are configured in the capturer to be selectively moved by the capturer, and

the processor is configured to control the capturer to rotate at least one of the two cameras in a prioritized direction corresponding to a direction indicated by a user manipulation of a turn signal level of the vehicle or based on which of left or right turn signal indicators are activated in the vehicle.

51. The apparatus of claim 41, wherein the processor is further configured to determine a stopping distance and a threshold for a blind spot area based on the estimated driving situation, determine a minimum value for the parameter based on a predetermined constraint associated with the stopping distance, determine a maximum value for the parameter based on a predetermined constraint associated with the threshold for the blind spot area, and determine the parameter to be a value equal to either or between the minimum value and the maximum value.

52. The apparatus of claim 51, wherein the determining of the threshold includes determining a threshold blind spot distance, and the predetermined constraint is associated with the threshold blind spot distance.

53. The apparatus of claim 41, wherein, when the estimated driving situation is the vehicle driving at high speed, the processor is configured to determine the parameter such that a depth error of a maximum measurable depth corresponding to the stereo camera width is reduced over a depth error of another maximum measurable depth corresponding to another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a lower speed.

54. The apparatus of claim 41, wherein, when the estimated driving situation is the vehicle driving at low speed, the processor is configured to determine the parameter such that a blind spot distance corresponding to the stereo camera width is reduced over a blind spot distance corresponding to another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a higher speed.

55. The apparatus of claim 41, wherein the processor is further configured to determine surrounding environment information of the vehicle based on the measured depth and to control the determined surrounding environment information to be displayed on a display of the vehicle.

56. The apparatus of claim 41, wherein the vehicle is an automobile.