Method for warning vehicle of risk of lane change and alarm device using the same

• US 10,300,851 B1
• Filed: 10/04/2018
• Issued: 05/28/2019
• Est. Priority Date: 10/04/2018
• Status: Active Grant

• Current Assignee: Stradvision, Inc.
• Original Assignee: Stradvision, Inc.
• Inventors: Kim, Kye-Hyeon, Kim, Yongjoong, Kim, Insu, Kim, Hak-Kyoung, Nam, Woonhyun, Boo, Sukhoon, Sung, Myungchul, Yeo, Donghun, Ryu, Wooju, Jang, Taewoong, Jeong, Kyungjoong, Je, Hongmo, Cho, Hojin
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US16/151,760
• Time in Patent Office: 236 Days
• Field of Search: 382103, 382104, 382155-159
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A method for warning a vehicle of a risk of lane change is provided. The method includes steps of: (a) an alarm device, if at least one rear image captured by a running vehicle is acquired, segmenting the rear image by using a learned convolutional neural network (CNN) to thereby obtain a segmentation image corresponding to the rear image; (b) the alarm device checking at least one free space ratio in at least one blind spot by referring to the segmentation image, wherein the free space ratio is determined as a ratio of a road area without an object in the blind spot to a whole area of the blind spot; and (c) the alarm device, if the free space ratio is less than or equal to at least one predetermined threshold value, warning a driver of the vehicle of the risk of lane change.

Claims

1. A method for warning a vehicle of a risk of lane change, comprising steps of:

(a) an alarm device, if at least one rear image captured by a running vehicle is acquired, segmenting the rear image by using a learned convolutional neural network (CNN) to thereby obtain a segmentation image corresponding to the rear image;

(b) the alarm device checking at least one free space ratio in at least one blind spot by referring to the segmentation image, wherein the free space ratio is determined as a ratio of a road area without an object in the blind spot to a whole area of the blind spot; and

(c) the alarm device, if the free space ratio is less than or equal to at least one predetermined threshold value, warning a driver of the vehicle of the risk of lane change.

2. The method of claim 1, wherein the blind spot includes a first sub-blind spot to an n-th sub-blind spot which are set by distances in a direction away from a rear of the vehicle,

wherein each of a first threshold value to an n-th threshold value, as each threshold value for the free space ratio, is set for each of the first sub-blind spot to the n-th sub-blind spot, and

wherein, at the step of (c), the alarm device warns the driver differently according to a distance between the vehicle and a k-th sub-blind spot having the free space ratio less than or equal to a k-th threshold value.

3. The method of claim 2, wherein, on condition that (i) a case that the free space ratio of the k-th sub-blind spot is less than or equal to the k-th threshold value, and the free space ratio of (k−

1)-th sub-blind spot is 100% is referred to as a first case, (ii) a case that the free space ratio of the k-th sub-blind spot is less than or equal to the k-th threshold value, and the free space ratio of (k−

1)-th sub-blind spot is less than or equal to the (k−

1)-th threshold value is referred to as a second case, and (iii) a case that the free space ratio of the k-th sub-blind spot is 100%, and the free space ratio of (k−

1)-th sub-blind spot is less than or equal to the (k−

1)-th threshold value is referred to as a third case, the alarm device informs the driver that an object is closer to the vehicle in the second case and the third case as compared to the vehicle in the first case.

4. The method of claim 2, wherein, on condition that the free space ratio of the k-th sub-blind spot exceeds the k-th threshold value and is less than or equal to a k′

-th threshold value which is set to be a value greater than the k-th threshold value, if the free space ratio of a (k+1)-th sub-blind spot is less than or equal to a (k+1)-th threshold value, the alarm device warns the driver differently according to the distance between the vehicle and the k-th sub-blind spot.

5. The method of claim 2, wherein the alarm device obtains movement information of the object within the first sub-blind spot to the n-th sub-blind spot by referring to each rate of change in the free space ratio of each of the first sub-blind spot to the n-th sub-blind spot, and

wherein the alarm device warns the driver according to a distance between the k-th sub-blind spot and the vehicle by referring to the obtained movement information of the object.

6. The method of claim 5, wherein, if the free space ratio in the (k+1)-th sub-blind spot increases from a state in which the free space ratio is less than or equal to a (k+1)-th threshold value and if the free space ratio in the k-th sub-blind spot decreases, the alarm device determines that the object is moving from the (k+1)-th sub-blind spot to the k-th sub-blind spot,

wherein, if the free space ratio in the (k−

1)-th sub-blind spot increases from a state in which the free space ratio is less than or equal to a (k−

1)-th threshold value and if the free space ratio in the k-th sub-blind spot decreases, the alarm device determines that the object is moving from the (k−

1)-th sub-blind spot to the k-th sub-blind spot, and

wherein the alarm device warns the driver according to the distance between the k-th sub-blind spot and the vehicle by further referring to the movement direction of the object in the k-th sub-blind spot.

7. The method of claim 2, wherein the alarm device refers to a rate of change in the free space ratio in the k-th sub-blind spot to thereby check a velocity or an acceleration of the object, and refers to the velocity or the acceleration of the object to thereby warn the driver according to the distance between the k-th sub-blind spot and the vehicle.

8. The method of claim 7, wherein, on condition that it is determined that the object is approaching the vehicle as a result of checking the velocity or the acceleration of the object, if the free space ratio of the k-th sub-blind spot is less than or equal to a k″

-th threshold value which is set to be a value greater than the k-th threshold value, the alarm device warns the driver according to the distance between the k-th sub-blind spot and the vehicle.

9. The method of claim 1, wherein, if the rear image represents a rear image obtained from a t-th frame, at the step of (c), the alarm device checks a relative movement direction of the object corresponding to a running direction of the vehicle by using (i) a t-th ratio which represents the free space ratio in the blind spot obtained from the rear image in the t-th frame and (ii) each of a (t−

1)-th ratio to a (t−

i)-th ratio which represents the free space ratio in the blind spot obtained from the rear image in a (t−

1)-th frame to the rear image in a (t−

i)-th frame, and warns the driver of the risk of the lane change by referring to the relative movement direction of the object.

10. The method of claim 9, wherein the alarm device analyzes an increase rate of from the (t−

i)-th rate to the t-th rate, and, if the increase rate is a negative value, determines that the object has moved from an external side of the blind spot to an inner side of the blind spot, and, if the increase rate is a positive value, determines that the object has moved from the inner side of the blind spot to the external side of the blind spot, and wherein the alarm device warns the driver of the risk of the lane change by further referring to the determined movement direction of the object.

11. The method of claim 10, wherein the alarm device analyzes the segmentation image at a specific frame which is selected among the (t−

i)-th frame to the t-th frame by referring to information on which of the (t−

i)-th rate to the t-th rate changes from a rate of 100% to a rate of less than 100%, wherein the alarm device determines whether an area where the object is located in the blind spot is an area adjacent to the vehicle or an area away from the vehicle to thereby check a location of entrance of the object into the blind spot, then checks a movement direction of the object relative to that of the vehicle by referring to the location of the entrance of the object into the blind spot and the movement direction of the object, and wherein the alarm device warns the driver of the risk of the lane change by further referring to the movement direction of the object relative to that of the vehicle.

12. The method of claim 1, wherein, at the step of (a), the warning device applies at least one convolutional operation to the rear image to thereby generate at least one feature map as a downsampled feature map and then applies at least one deconvolutional operation to the downsampled feature map to thereby segment the road area and the object in the rear image.

13. An alarm device for warning a vehicle of a risk of lane change, comprising:

a communication part for acquiring at least one rear image captured by a running vehicle; and

a processor for performing processes of (I) segmenting the rear image by using a learned convolutional neural network (CNN) to thereby obtain a segmentation image corresponding to the rear image, (II) checking at least one free space ratio in at least one blind spot by referring to the segmentation image, wherein the free space ratio is determined as a ratio of a road area without an object in the blind spot to a whole area of the blind spot, and (III) if the free space ratio is less than or equal to at least one predetermined threshold value, warning a driver of the vehicle of the risk of lane change.

14. The alarm device of claim 13, wherein the blind spot includes a first sub-blind spot to an n-th sub-blind spot which are set by distances in a direction away from a rear of the vehicle,

wherein each of a first threshold value to an n-th threshold value, as each threshold value for the free space ratio, is set for each of the first sub-blind spot to the n-th sub-blind spot, and

wherein, at the process of (III), the processor warns the driver differently according to a distance between the vehicle and a k-th sub-blind spot having the free space ratio less than or equal to a k-th threshold value.

15. The alarm device of claim 14, wherein, on condition that (i) a case that the free space ratio of the k-th sub-blind spot is less than or equal to the k-th threshold value, and the free space ratio of (k−

1)-th sub-blind spot is 100% is referred to as a first case, (ii) a case that the free space ratio of the k-th sub-blind spot is less than or equal to the k-th threshold value, and the free space ratio of (k−

1)-th sub-blind spot is less than or equal to the (k−

1)-th threshold value is referred to as a second case, and (iii) a case that the free space ratio of the k-th sub-blind spot is 100%, and the free space ratio of (k−

1)-th sub-blind spot is less than or equal to the (k−

1)-th threshold value is referred to as a third case, the processor informs the driver that an object is closer to the vehicle in the second case and the third case as compared to the vehicle in the first case.

16. The alarm device of claim 14, wherein, on condition that the free space ratio of the k-th sub-blind spot exceeds the k-th threshold value and is less than or equal to a k′

-th threshold value which is set to be a value greater than the k-th threshold value, if the free space ratio of a (k+1)-th sub-blind spot is less than or equal to a (k+1)-th threshold value, the processor warns the driver differently according to the distance between the vehicle and the k-th sub-blind spot.

17. The alarm device of claim 14, wherein the processor obtains movement information of the object within the first sub-blind spot to the n-th sub-blind spot by referring to each rate of change in the free space ratio of each of the first sub-blind spot to the n-th sub-blind spot, and

wherein the processor warns the driver according to a distance between the k-th sub-blind spot and the vehicle by referring to the obtained movement information of the object.

18. The alarm device of claim 17, wherein, if the free space ratio in the (k+1)-th sub-blind spot increases from a state in which the free space ratio is less than or equal to a (k+1)-th threshold value and if the free space ratio in the k-th sub-blind spot decreases, the processor determines that the object is moving from the (k+1)-th sub-blind spot to the k-th sub-blind spot,

wherein, if the free space ratio in the (k−

1)-th sub-blind spot increases from a state in which the free space ratio is less than or equal to a (k−

1)-th threshold value and if the free space ratio in the k-th sub-blind spot decreases, the processor determines that the object is moving from the (k−

1)-th sub-blind spot to the k-th sub-blind spot, and

wherein the processor warns the driver according to the distance between the k-th sub-blind spot and the vehicle by further referring to the movement direction of the object in the k-th sub-blind spot.

19. The alarm device of claim 14, wherein the processor refers to a rate of change in the free space ratio in the k-th sub-blind spot to thereby check a velocity or an acceleration of the object, and refers to the velocity or the acceleration of the object to thereby warn the driver according to the distance between the k-th sub-blind spot and the vehicle.

20. The alarm device of claim 19, wherein, on condition that it is determined that the object is approaching the vehicle as a result of checking the velocity or the acceleration of the object, if the free space ratio of the k-th sub-blind spot is less than or equal to a k″

-th threshold value which is set to be a value greater than the k-th threshold value, the processor warns the driver according to the distance between the k-th sub-blind spot and the vehicle.

21. The alarm device of claim 13, wherein, if the rear image represents a rear image obtained from a t-th frame, at the process of (III), the processor checks a relative movement direction of the object corresponding to a running direction of the vehicle by using (i) a t-th ratio which represents the free space ratio in the blind spot obtained from the rear image in the t-th frame and (ii) each of a (t−

1)-th ratio to a (t−

i)-th ratio which represents the free space ratio in the blind spot obtained from the rear image in a (t−

1)-th frame to the rear image in a (t−

i)-th frame, and warns the driver of the risk of the lane change by referring to the relative movement direction of the object.

22. The alarm device of claim 21, wherein the processor analyzes an increase rate of from the (t−

i)-th rate to the t-th rate, and, if the increase rate is a negative value, determines that the object has moved from an external side of the blind spot to an inner side of the blind spot, and, if the increase rate is a positive value, determines that the object has moved from the inner side of the blind spot to the external side of the blind spot, and wherein the processor warns the driver of the risk of the lane change by further referring to the determined movement direction of the object.

23. The alarm device of claim 22, wherein the processor analyzes the segmentation image at a specific frame which is selected among the (t−

i)-th frame to the t-th frame by referring to information on which of the (t−

i)-th rate to the t-th rate changes from a rate of 100% to a rate of less than 100%, wherein the processor determines whether an area where the object is located in the blind spot is an area adjacent to the vehicle or an area away from the vehicle to thereby check a location of entrance of the object into the blind spot, then checks a movement direction of the object relative to that of the vehicle by referring to the location of the entrance of the object into the blind spot and the movement direction of the object, and wherein the processor warns the driver of the risk of the lane change by further referring to the movement direction of the object relative to that of the vehicle.

24. The alarm device of claim 13, wherein, at the process of (I), the processor applies at least one convolutional operation to the rear image to thereby generate at least one feature map as a downsampled feature map and then applies at least one deconvolutional operation to the downsampled feature map to thereby segment the road area and the object in the rear image.