Stereo camera-based autonomous driving method and apparatus

• US 10,444,752 B2
• Filed: 02/23/2017
• Issued: 10/15/2019
• Est. Priority Date: 08/16/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Changhyun, Kang, Hyoa, Shin, Changwoo, Cho, Baek Hwan, Ji, Derek Daehyun
• Primary Examiner(s): Rahman, Mohammad J
• Application Number:
  US15/440,138
• Publication Number:
  US 20180052457A1
• Time in Patent Office: 964 Days
• Field of Search: None
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G05D 1/0055 with safety arrangements
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Disclosed is a stereo camera-based autonomous driving method and apparatus, the method including estimating a driving situation of a vehicle, determining a parameter to control a stereo camera width of a stereo camera based on the estimated driving situation, controlling a capturer configured to control arrangement between two cameras of the stereo camera for a first direction based on the determined parameter, and measuring a depth of an object located in the first direction based on two images respectively captured by the two cameras with the controlled arrangement.

Claims

1. An autonomous driving method comprising:

estimating a driving situation of a vehicle based on at least a driving speed of the vehicle;

determining a parameter to control a stereo camera width of a stereo camera based on the driving situation;

controlling a capturer, comprising two cameras of the stereo camera, configured to control an arrangement between the two cameras for a first direction based on the parameter; and

measuring a depth of an object located in the first direction based on two images respectively captured by the two cameras with the controlled arrangement,

wherein the controlling of the capturer further comprises;

detecting a vehicle or an obstacle in the first direction based on the estimated driving situation;

assigning a priority to the two cameras located in the first direction;

selecting the two cameras that are separated apart from each other by a distance corresponding to the parameter based on the assigned priority; and

operating a remaining cameras other than the selected two cameras in a standby or power off mode.

2. The method of claim 1, wherein the two cameras are selected from among a plurality of cameras facing the first direction in a camera array of the capturer.

3. The method of claim 2, wherein the plurality of cameras are fixedly arranged in the capturer and each is linearly separated by a first distance, and the distance corresponding to the parameter is evenly divisible by the first distance.

4. The method of claim 1, wherein the two cameras are configured in the capturer to be selectively moved by the capturer, and

the controlling of the capturer comprises selectively moving at least one of the two cameras such that the two cameras are separated apart from each other by a distance corresponding to the parameter.

5. The method of claim 4, wherein the two cameras are configured in the capturer to be selectively moved by the capturer along a route, and

the controlling of the capturer comprises selectively moving the at least one of the two cameras along the route such that the two cameras are separated apart from each other by the distance corresponding to the parameter.

6. The method of claim 1, wherein the determining of the parameter comprises:

determining a stopping distance of the vehicle based on the driving situation; and

determining the parameter based on a constraint associated with the stopping distance.

7. The method of claim 6, wherein the constraint associated with the stopping distance is based on a maximum measurable depth corresponding to the stereo camera width and a relationship between the stopping distance and a safe distance based on a depth error of the maximum measurable depth.

8. The method of claim 1, wherein the determining of the parameter comprises:

determining a threshold for a blind spot area based on the driving situation; and

determining the parameter based on a constraint associated with the threshold for the blind spot area.

9. The method of claim 8, wherein the determining of the threshold for the blind spot area comprises determining a threshold blind spot distance based on the driving situation, and the constraint is associated with the threshold blind spot distance.

10. The method of claim 9, wherein the constraint associated with the threshold blind spot distance is based on a relationship between the threshold blind spot distance and a blind spot distance corresponding to the stereo camera width.

11. The method of claim 1, wherein the determining of the parameter comprises:

determining a stopping distance and a threshold for a blind spot area based on the driving situation;

determining a minimum value for the parameter based on a first constraint associated with the stopping distance;

determining a maximum value for the parameter based on a second constraint associated with the threshold for the blind spot area; and

determining the parameter to be a value equal to either or between the minimum value and the maximum value.

12. The method of claim 11, wherein the determining of the threshold comprises determining a threshold blind spot distance, and the second constraint is associated with the threshold blind spot distance.

13. The method of claim 1, wherein the determining of the parameter comprise determining the parameter such that a depth error of a maximum measurable depth corresponding to the stereo camera width is reduced over a depth error of another maximum measurable depth corresponding to another stereo camera width determined for a different driving situation corresponding to the vehicle driving at a lower speed, in response the driving situation being the vehicle driving at high speed.

14. The method of claim 1, wherein the determining of the parameter comprises determining the parameter such that a blind spot distance corresponding to the stereo camera width is reduced over a blind spot distance corresponding to another stereo camera width determined for a different driving situation corresponding to the vehicle driving at a higher speed, in response to the driving situation being the vehicle driving at low speed.

15. The method of claim 1, wherein the driving situation is determined based on the driving speed and at least one of a moving direction of the vehicle, a location of the vehicle, or a user manipulation.

16. The method of claim 1, wherein the driving situation comprises at least one of the vehicle driving straight ahead, changing lanes, changing direction, driving on a highway, driving at high speed, driving at low speed, driving in a city, driving with a weather affecting condition, or driving with a road affecting condition, and

the determining of the parameter comprises determining the parameter based on the driving situation and corresponding prioritized consideration between an extent of depth error of a maximum measurable depth corresponding to the stereo camera width and an extent of a blind spot area corresponding to the stereo camera width.

17. The method of claim 1, further comprising:

predicting a driving direction of the vehicle based on the driving situation,

wherein the determining of the parameter comprises determining the parameter based on a visual field direction of the stereo camera corresponding to the predicted driving direction.

18. The method of claim 17, further comprising selectively turning off one or more additional stereo cameras of the vehicle for one or more other visual field directions that are not needed for driving of the vehicle in the predicted driving direction.

19. The method of claim 1, further comprising:

controlling the capturer or another capturer so that two additional cameras perform stereo capturing in a second direction, opposite to the first direction, with a stereo camera width based on the parameter; and

measuring a depth of an object located in the second direction based on output images of the two additional cameras,

wherein the capturer or the other capturer is configured to perform capturing in the second direction in coordination with the captured two images in the first direction.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

21. An autonomous driving apparatus comprising:

a capturer comprising at least two cameras, the capturer being configured to control arrangement between the at least two cameras; and

a controller configured to;

determine a parameter related to a stereo camera width, of a stereo camera represented by the at least two cameras, based on an estimated driving situation of a vehicle, to control the capturer to control the arrangement between the at least two cameras based on the determined parameter, and to measure a depth of an object located in a first direction based on two images respectively captured by the at least two cameras with the controlled arrangement,

wherein the estimated driving situation is determined based on at least a driving speed of the vehicle, and

wherein the controller is further configured to;

detect a vehicle or an obstacle in the first direction based on the estimated driving situation;

assign a priority to the two cameras located in the first direction;

select the at least two cameras that are separated apart from each other by a distance corresponding to the parameter based on the assigned priority; and

operate a remaining cameras other than the selected two cameras in a standby or power off mode.

22. The apparatus of claim 21, wherein the controller is further configured to determine the estimated driving situation based on sensed information or one or more driving situation estimations from sensors of the vehicle and/or from sensors or devices external of the vehicle.

23. The apparatus of claim 21, wherein the capturer comprises a camera array including a plurality of cameras facing in the first direction, and

the controller is configured to select, from among the plurality of cameras, the at least two cameras.

24. The apparatus of claim 23, wherein the plurality of cameras are fixedly arranged in the capturer and each linearly separated by a first distance, and the distance corresponding to the determined parameter is evenly divisible by the first distance.

25. The apparatus of claim 21, wherein the at least two cameras are configured in the capturer to be selectively moved by the capturer, and

the controller is configured to control the capturer to move one or more of the at least two cameras such that the at least two cameras are separated from each other by a distance corresponding to the determined parameter.

26. The apparatus of claim 25, wherein the at least two cameras are configured in the capturer to be selectively moved by the capturer along a predetermined route, and

the controller is configured to control the capturer to move the one or more of the at least two cameras along the predetermined route such that the at least two cameras are separated from each other by the distance corresponding to the determined parameter.

27. The apparatus of claim 21, wherein the controller is configured to determine a stopping distance of the vehicle based on the estimated driving situation and determine the parameter based on a predetermined constraint associated with the stopping distance.

28. The apparatus of claim 27, wherein the predetermined constraint associated with the stopping distance is based on consideration of a maximum measurable depth corresponding to the stereo camera width and a relationship between the stopping distance and a safe distance based on a depth error of the maximum measurable depth.

29. The apparatus of claim 21, wherein the controller is configured to determine a threshold for a blind spot area based on the estimated driving situation and determine the parameter based on a predetermined constraint associated with the threshold for the blind spot area.

30. The apparatus of claim 29, wherein the determining of the threshold for the blind spot area comprises determining a threshold blind spot distance based on the estimated driving situation, and the predetermined constraint is associated with the threshold blind spot distance.

31. The apparatus of claim 30, wherein the predetermined constraint associated with the threshold blind spot distance is based on consideration of a relationship between the threshold blind spot distance and a blind spot distance corresponding to the stereo camera width.

32. The apparatus of claim 21, wherein the controller is configured to determine a stopping distance and a threshold for a blind spot area based on the estimated driving situation, determine a minimum value for the parameter based on a predetermined constraint associated with the stopping distance, determine a maximum value for the parameter based on a predetermined constraint associated with the threshold for the blind spot area, and determine the parameter to be a value equal to either or between the minimum value and the maximum value.

33. The apparatus of claim 32, wherein the determining of the threshold comprises determining a threshold blind spot distance, and the predetermined constraint is associated with the threshold blind spot distance.

34. The apparatus of claim 21, wherein, when the estimated driving situation is the vehicle driving at high speed, the controller is configured to determine the parameter such that a depth error of a maximum measurable depth corresponding to the stereo camera width is reduced over a depth error of another maximum measurable depth corresponding to another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a lower speed.

35. The apparatus of claim 21, wherein, when the estimated driving situation is the vehicle driving at low speed, the controller is configured to determine the parameter such that a blind spot distance corresponding to the stereo camera width is reduced over a blind spot distance corresponding to another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a higher speed.

36. The apparatus of claim 21, further comprising:

a sensor configured to sense at least one of the driving speed of the vehicle, a moving direction of the vehicle, a location of the vehicle, or a user manipulation, wherein the estimated driving situation is determined based on the driving speed and at least one of the moving direction of the vehicle, the location of the vehicle, or the user manipulation.

37. The apparatus of claim 21, wherein the estimated driving situation comprises at least one of the vehicle driving straight ahead, changing lanes, changing direction, driving on a highway, driving at high speed, driving at low speed, driving in a city, driving with a weather affecting condition, or driving with a road affecting condition, and

the controller is configured to determine the parameter based on the estimated driving situation in prioritized consideration between an extent of depth error of a maximum measurable depth corresponding to the stereo camera width and an extent of a blind spot area corresponding to the stereo camera width.

38. The apparatus of claim 21, wherein the controller is configured to predict a driving direction of the vehicle based on the estimated driving situation and determine the parameter based on a visual field direction of the stereo camera corresponding to the predicted driving direction.

39. The apparatus of claim 38, wherein the controller is configured to selectively turn off one or more additional stereo cameras of the vehicle respectively for one or more other visual field directions that are determined not needed for driving of the vehicle in the predicted driving direction.

40. The apparatus of claim 21, wherein the capturer or another capturer is configured to include two additional cameras to perform stereo capturing in a second direction, opposite the first direction, with a stereo camera width based on the determined parameter, and

the controller is configured to control the capturer or the other capturer in coordination with the captured two images in the first direction so that the two additional cameras perform the capturing in the second direction with the stereo camera width based on the determined parameter, and configured to measure a depth of an object located in the second direction based on output images of the two additional cameras.

41. An autonomous driving apparatus, the apparatus comprising:

a processor configured;

to determine a parameter to control a stereo camera width of a stereo camera based on an estimated driving situation of a vehicle, wherein the estimated driving situation is determined based on at least a driving speed of the vehicle,

to control a capturer configured to control arrangement between two cameras of the stereo camera for a first direction based on the parameter, and

to measure a depth of an object located in the first direction based on two images respectively captured by the two cameras with the controlled arrangement,

wherein the processor is further configured to;

detect a vehicle or an obstacle in the first direction based on the estimated driving situation;

assign a priority to the two cameras located in the first direction;

select the two cameras that are separated apart from each other by a distance corresponding to the parameter based on the assigned priority; and

operate a remaining cameras other than the selected two cameras in a standby or power off mode.

42. The apparatus of claim 41, further comprising a memory configured to store instructions,

wherein the processor is further configured to execute the instructions to configure the processor to perform the determining of the parameter to control the stereo camera width of the stereo camera based on the estimated driving situation.

43. The apparatus of claim 41, wherein the processor is further configured to determine the estimated driving situation based on sensed information or one or more driving situation estimations from sensors of the vehicle and/or from sensors or devices external of the vehicle.

44. The apparatus of claim 43, further comprising the capturer, which comprises the two cameras, and/or the sensors of the vehicle.

45. The apparatus of claim 43, wherein the apparatus comprises the capturer and the capturer comprises a camera array including a plurality of cameras facing in the first direction, and

the processor is further configured to select, from among the plurality of cameras, the two cameras that are separated apart from each other by a distance corresponding to the determined parameter.

46. The apparatus of claim 45, wherein the plurality of cameras are fixedly arranged in the capturer and each linearly separated by a first distance, and the distance corresponding to the determined parameter is evenly divisible by the first distance.

47. The apparatus of claim 43, wherein the apparatus comprises the capturer and the two cameras are configured in the capturer to be selectively moved by the capturer, and

the processor is further configured to control the capturer to move one or more of the two cameras such that the two cameras are separated from each other by a distance corresponding to the determined parameter.

48. The apparatus of claim 47, wherein the two cameras are configured in the capturer to be selectively moved by the capturer along a predetermined route, and the processor is configured to control the capturer to move the one or more of the two cameras along the predetermined route such that the two cameras are separated from each other by the distance corresponding to the determined parameter.

49. The apparatus of claim 48, wherein, for the selective moving of the two cameras by the capturer, the processor is further configured to control the capturer to rotate at least one of the two cameras in a prioritized direction based on the estimated driving situation.

50. The apparatus of claim 43, wherein the apparatus comprises the capturer and the two cameras are configured in the capturer to be selectively moved by the capturer, and

the processor is configured to control the capturer to rotate at least one of the two cameras in a prioritized direction corresponding to a direction indicated by a user manipulation of a turn signal level of the vehicle or based on which of left or right turn signal indicators are activated in the vehicle.

51. The apparatus of claim 41, wherein the processor is further configured to determine a stopping distance and a threshold for a blind spot area based on the estimated driving situation, determine a minimum value for the parameter based on a predetermined constraint associated with the stopping distance, determine a maximum value for the parameter based on a predetermined constraint associated with the threshold for the blind spot area, and determine the parameter to be a value equal to either or between the minimum value and the maximum value.

52. The apparatus of claim 51, wherein the determining of the threshold comprises determining a threshold blind spot distance, and the predetermined constraint is associated with the threshold blind spot distance.

53. The apparatus of claim 41, wherein, when the estimated driving situation is the vehicle driving at high speed, the processor is configured to determine the parameter such that a depth error of a maximum measurable depth corresponding to the stereo camera width is reduced over a depth error of another maximum measurable depth corresponding to another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a lower speed.

54. The apparatus of claim 41, wherein, when the estimated driving situation is the vehicle driving at low speed, the processor is configured to determine the parameter such that a blind spot distance corresponding to the stereo camera width is reduced over a blind spot distance corresponding to another stereo camera width determined for a different estimated driving situation corresponding to the vehicle driving at a higher speed.

55. The apparatus of claim 41, wherein the processor is further configured to determine surrounding environment information of the vehicle based on the measured depth and to control the determined surrounding environment information to be displayed on a display of the vehicle.

56. The apparatus of claim 41, wherein the vehicle is an automobile.