Method and apparatus for determining abnormal object

• US 10,546,201 B2
• Filed: 05/01/2017
• Issued: 01/28/2020
• Est. Priority Date: 11/29/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kang, Hyoa, Kim, Changhyun
• Primary Examiner(s): Fujita, Katrina R
• Application Number:
  US15/583,339
• Publication Number:
  US 20180150701A1
• Time in Patent Office: 1,002 Days
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/126 Evolutionary algorithms, e.g. genetic algorithms or genetic programming
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

Disclosed is a method and apparatus for determining an abnormal object, the method including selecting a candidate object from target objects extracted from a two-dimensional (2D) image of a front view captured from a host vehicle, generating a three-dimensional (3D) model of the candidate object, determining, based on the 3D model, whether the candidate object corresponds to an abnormal object that interferes with driving of the host vehicle, and outputting the abnormal object, in response to the candidate object corresponding to the abnormal object.

Claims

1. A method of determining an abnormal object, the method comprising:

selecting a candidate object from target objects extracted from a two-dimensional (2D) image of a front view captured from a host vehicle;

estimating a three-dimensional (3D) information of the candidate object from the 2D image;

generating a 3D model based on the estimated 3D information;

determining, based on the 3D model, whether the candidate object corresponds to an abnormal object that interferes with driving of the host vehicle; and

outputting the abnormal object, in response to the candidate object corresponding to the abnormal object,

wherein the selecting the candidate object comprises selecting a first target object from among the target objects as the candidate object in response to a difference being greater than or equal to a threshold, the difference being between a size of a portion of the first target object not occluded by another vehicle in the 2D image to be used to generate the 3D model and a size of a portion of a second target object of the target objects that is not occluded by another vehicles in the 2D image to be used to generate the 3D model.

2. The method of claim 1, wherein the selecting of the candidate object comprises selecting the candidate object from the target objects based on any one or any combination of any two or more of a degree of curve of a lane, a distance from the host vehicle, and types of nearby vehicles.

3. The method of claim 1, wherein the selecting of the candidate object comprises selecting the candidate object from the target objects using a neural network trained to output a candidate object in response to an input of the 2D image.

4. The method of claim 3, wherein the neural network is adaptively trained based the determining of whether the candidate object corresponds to the abnormal object.

5. The method of claim 1, wherein the selecting of the candidate object comprises selecting the candidate object from the target objects based on an abnormal score from a previous image captured earlier than the 2D image, and

the abnormal score determines whether a target object corresponds to the abnormal object.

6. The method of claim 1, wherein the selecting of the candidate object comprises selecting the candidate object from the target objects based on driving patterns of the target objects acquired from the 2D image and a previous image captured earlier than the 2D image.

7. The method of claim 1, wherein the determining of whether the candidate object corresponds to the abnormal object comprises determining whether the candidate object corresponds to the abnormal object based on any one or any combination of any two or more of whether the candidate object is travelling in a lane identical or adjacent to that of the host vehicle, a moving direction, a moving speed, a distance from the host vehicle, and a size of the candidate object.

8. The method of claim 7, wherein the determining of whether the candidate object corresponds to the abnormal object comprises determining whether the candidate object corresponds to the abnormal object based any one or any combination of any two or more of a shape of a road on which the host vehicle is travelling, a traffic accident frequency, a traffic condition, and a driving pattern of a user of the host vehicle.

9. The method of claim 1, wherein, in response to the candidate object being a person, the determining of whether the candidate object corresponds to the abnormal object comprises determining whether the candidate object corresponds to the abnormal object using any one or any combination of any two or more of a moving direction, a moving speed, a distance from the host vehicle, a pose, a direction of a face, and a height of the person.

10. The method of claim 1, wherein the outputting of the abnormal object comprises any one or any combination of any two or more of displaying the abnormal object on a display device in the host vehicle, replaying a warning sound indicating a presence of the abnormal object through a speaker device in the host vehicle, delivering tactile feedback on the abnormal object by a steering handle of the host vehicle, delivering information on the abnormal object by a vibration of a seat of the host vehicle, and providing the abnormal object to a driving system that controls or guides the driving of the host vehicle.

11. The method of claim 1, wherein the selecting of the candidate object comprises selecting a target object travelling in a pattern different than remaining target objects.

12. A non-transitory computer-readable medium storing instructions that, when executed by a processor, causes the processor to perform the method of claim 1.

13. An apparatus for determining an abnormal object, the apparatus comprising:

an image sensor configured to capture a front image from a host vehicle and to generate a two-dimensional (2D) image; and

a processor configured to

select a candidate object from target objects extracted from the 2D image,

estimate a 3D information of the candidate object from the 2D image,

generate a 3D model based on the estimated 3D information,

determine, based on the 3D model, whether the candidate object corresponds to an abnormal object that interferes with driving of the host vehicle, and

output the abnormal object, in response to the candidate object corresponding to the abnormal object,

wherein the processor is further configured to select a first target object from among the target objects as the candidate object in response to a difference being greater than or equal to a threshold, the difference being between a size of a portion of the first target object not occluded by another vehicle in the 2D image to be used to generate the 3D model and a size of a portion of a second target object of the target objects that is not occluded by another vehicles in the 2D image to be used to generate the 3D model.

14. The apparatus of claim 13, wherein the processor is further configured to select the candidate object from the target objects using a neural network trained to output a candidate object in response to an input of the 2D image.

15. The apparatus of claim 14, wherein the neural network is adaptively trained based a determination of whether the candidate object corresponds to the abnormal object.

16. The apparatus of claim 13, wherein the processor is further configured to select the candidate object from the target objects based on an abnormal score from a previous image captured earlier than the 2D image, and

the abnormal score determines whether a target object corresponds to the abnormal object.

17. The apparatus of claim 13, wherein the processor is further configured to determine whether the candidate object corresponds to the abnormal object based on any one or any combination of any two or more of whether the candidate object is travelling in a lane identical or adjacent to that of the host vehicle, a moving direction, a moving speed, a distance from the host vehicle, and a size of the candidate object.

18. The apparatus of claim 17, wherein the processor is further configured to determine whether the candidate object corresponds to the abnormal object based on any one or any combination of any two or more of a shape of a road on which the host vehicle is travelling, a traffic accident frequency, a traffic condition, and a driving pattern of a user of the host vehicle.