System and method for providing laser camera fusion for identifying and tracking a traffic participant

• US 10,121,082 B2
• Filed: 10/07/2015
• Issued: 11/06/2018
• Est. Priority Date: 10/07/2015
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Heisele, Bernd, Ayvaci, Alper
• Primary Examiner(s): Williams, Jeffery A
• Assistant Examiner(s): Young, Patricia I
• Application Number:
  US14/876,907
• Publication Number:
  US 20170103269A1
• Time in Patent Office: 1,126 Days
• CPC Class Codes: B60R 2300/108 using 'non-standard' camera systems, e.g. camera sensor used for additional purposes i.a. rain sensor, camera sensor split in multiple image areas
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A system and method for providing laser camera fusion for identifying and tracking a traffic participant that include receiving an image of a surrounding environment of a vehicle and a set of object coordinates of at least one object determined within the surrounding environment of the vehicle. The system and method also include determining a portion of the image as object space based on the image and the set of object coordinates and filtering the object space to identify a traffic related object. Additionally, the system and method include determining a three dimensional position of the traffic related object and classifying the traffic related object as at least one of: the traffic participant, or a non-traffic participant. The system and method further include tracking the traffic participant based on a three dimensional position of the traffic related object classified as the traffic participant.

Claims

1. A computer-implemented method for providing laser camera fusion for identifying and tracking a traffic participant, comprising:

receiving an image of a surrounding environment of a vehicle from a vehicle camera system and a set of object coordinates of at least one object determined within the surrounding environment of the vehicle from a vehicle laser projection system;

determining a portion of the image as object space based on the set of object coordinates and at least one corresponding set of image coordinates, wherein the at least one corresponding set of image coordinates are two-dimensional coordinates that represent at least one pixel of the image that corresponds to at least one location within the image that is associated to the set of object coordinates;

filtering the object space to identify a traffic related object;

determining a three dimensional position of the traffic related object, wherein determining the three dimensional position of the traffic related object includes projecting the image and projecting a plurality of laser beam points from an image laser projection system to a plurality of pixels of the projected image of the surrounding environment of the vehicle, wherein determining a three dimensional position of the traffic related object includes determining a subset of clusters of a plurality of clusters found closest to a camera of the vehicle camera system based on an evaluation of a mean laser beam length value for each cluster against at least one predetermined laser length value; and

classifying the traffic related object as at least one of;

the traffic participant, or a non-traffic participant;

tracking the traffic participant based on a three dimensional position of the traffic related object classified as the traffic participant, wherein a predicted position of the traffic participant relative to the vehicle is determined based on tracking the traffic participant; and

controlling at least one component of the vehicle to provide at least one vehicle collision avoidance function.

2. The computer-implemented method of claim 1, wherein determining the portion of the image as object space includes computing a horizon line of the image, a horizon space of the image, and a floor space of the image, wherein the horizon line is computed in a middle portion of the image, wherein the horizon space includes a portion of the image that is located above the horizon line and is located above a predetermined area where the traffic participant can be captured within the image, wherein the floor space includes a portion of the image that is located below the horizon line and includes the predetermined area where the traffic participant is captured within the image.

3. The computer-implemented method of claim 2, wherein determining the portion of the image as object space includes removing the horizon line and the horizon space from the image and determining a set of image coordinates included within the floor space of the image that corresponds to the set of object coordinates, wherein a remaining portion of the floor space of the image that includes the set of image coordinates that correspond to the set of object coordinates is determined as object space.

4. The computer-implemented method of claim 3, wherein filtering the object space includes determining at least one separate object within the image and identifying the at least one separate object as at least one of:

the traffic related object or the non-traffic related object, wherein a bounding box is computed around the traffic related object and the traffic related object is included within the bounding box, wherein the non-traffic related object is removed from the image.

5. The computer-implemented method of claim 1, wherein a subset of laser beam points of the plurality of laser beam points that are projected to a subset of pixels of the image included within the bounding box are grouped into a plurality of clusters, wherein mean three dimensional coordinates are computed for each cluster of the plurality of clusters.

6. The computer-implemented method of claim 5, wherein determining a three dimensional position of the traffic related object includes determining the subset of clusters of the plurality of clusters found closest to a camera of the vehicle camera system that captures the image based on determining that a measured length of the mean laser beam length is less than or equal to the predetermined laser length value, and determining the subset of clusters of the plurality of clusters found furthest from the camera that captures the image based on determining that the measured length of the mean laser beam length is more than the predetermined laser length value.

7. The computer-implemented method of claim 6, wherein determining a three dimensional position of the traffic related object includes removing the subsets of pixels from the bounding box that have laser beam points projected toward them from the subsets of clusters found furthest from the camera that captures the image, wherein a resized bounding box is computed around the subsets of pixels from the bounding box that have laser beams projected toward them from the subsets of clusters found closest to the camera that captures the image, wherein the traffic related object is included within the resized bounding box.

8. The computer-implemented method of claim 7, wherein classifying the traffic related object includes determining a measurement of the resized bounding box, wherein the measurement of the resized bounding box is compared against a minimum threshold measurement value and a maximum threshold measurement value, wherein the traffic related object is classified as at least one of:

the traffic participant when the measurement of the resized bounding box is within the minimum threshold measurement value and the maximum threshold measurement value, or the non-traffic participant when the measurement of the resized bounding box is outside of the minimum threshold measurement value and the maximum threshold measurement value.

9. The computer-implemented method of claim 8, wherein tracking the traffic participant includes receiving three dimensional coordinates of the traffic participant, wherein the three dimensional coordinates of the traffic participant provided at a future time frame are analyzed with previous three dimensional coordinates of the traffic participants provided at a previous time frame to determine a difference in distance traveled, wherein the difference is distance traveled is analyzed to estimate an average velocity of the traffic participant.

10. A system for providing laser camera fusion for identifying and tracking a traffic participant, comprising:

a memory storing instructions when executed by a processor cause the processor to;

receive an image of a surrounding environment of the vehicle from a vehicle camera system and a set of object coordinates of at least one object determined within the surrounding environment of the vehicle from a vehicle laser projection system;

determine a portion of the image as object space based the set of object coordinates and at least one corresponding set of image coordinates, wherein the at least one corresponding set of image coordinates are two-dimensional coordinates that represent at least one pixel of the image that corresponds to at least one location within the image that is associated to the set of object coordinates;

filter the object space to identify a traffic related object;

determine a three dimensional position of the traffic related object, wherein determining the three dimensional position of the traffic related object includes projecting the image and projecting a plurality of laser beam points from an image laser projection system to a plurality of pixels of the projected image of the surrounding environment of the vehicle, wherein determining a three dimensional position of the traffic related object includes determining a subset of clusters of a plurality of clusters found closest to a camera of the vehicle camera system based on an evaluation of a mean laser beam length value for each cluster against at least one predetermined laser length value;

classify the traffic related object as at least one of;

the traffic participant, or a non-traffic participant;

track the traffic participant based on a three dimensional position of the traffic related object classified as the traffic participant, wherein a predicted position of the traffic participant relative to the vehicle is determined based on tracking the traffic participant; and

control at least one component of the vehicle to provide at least one vehicle collision avoidance function.

11. The system of claim 10, wherein determining the portion of the image as object space includes computing a horizon line of the image, a horizon space of the image, and a floor space of the image, wherein the horizon line is computed in a middle portion of the image, wherein the horizon space includes a portion of the image that is located above the horizon line and is located above a predetermined area where the traffic participant can be captured within the image, wherein the floor space includes a portion of the image that is located below the horizon line and includes the predetermined area where a traffic participant is captured within the image.

12. The system of claim 11, wherein determining the portion of the image as object space includes removing the horizon line and the horizon space from the image and determines a set of image coordinates included within the floor space of the image that corresponds to the set of object coordinates, wherein a remaining portion of the floor space of the image that includes the set of image coordinates that correspond to the set of object coordinates is determined as object space.

13. The system of claim 12, wherein filtering the object space includes determining at least one separate object within the image and identifying the at least one separate object as at least one of:

the traffic related object or the non-traffic related object, wherein the object space filtering module computes a bounding box around the traffic related object and the traffic related object is included within the bounding box, wherein the non-traffic related object is removed from the image.

14. The system of claim 10, wherein a subset of laser beam points of the plurality of laser beam points that are projected to a subset of pixels of the image included within the bounding box are grouped into a plurality of clusters, wherein mean three dimensional coordinates are computed for each cluster of the plurality of clusters.

15. The system of claim 14, wherein determining a three dimensional position of the traffic related object includes determining the subset of clusters of the plurality of clusters found closest to a camera that captures the image based on determining that a measured length of the mean laser beam length is less than or equal to the predetermined laser length value, and determining a subset of clusters of the plurality of clusters found furthest from the camera that captures the image based on determining that the measured length of the mean laser beam length is more than the predetermined laser length value.

16. The system of claim 15, wherein determining a three dimensional position of the traffic related object includes removing the subsets of pixels from the bounding box that have laser beam points projected toward them from the subsets of clusters found furthest from the camera that captures the image, wherein the positional determinant module computes a resized bounding box around the subsets of pixels from the bounding box that have laser beams projected toward them from the subsets of clusters found closest to the camera that captures the image, wherein the traffic related object is included within the resized bounding box.

17. The system of claim 16, wherein classifying the traffic related object includes determining a measurement of the resized bounding box, wherein the measurement of the resized bounding box is compared against a minimum threshold measurement value and a maximum threshold measurement value, wherein the traffic related object is classified as at least one of:

the traffic participant when the measurement of the resized bounding box is within the minimum threshold measurement value and the maximum threshold measurement value, or the non-traffic participant when the measurement of the resized bounding box is outside of the minimum threshold measurement value and the maximum threshold measurement value.

18. The system of claim 17, wherein tracking the traffic participant includes receiving three dimensional coordinates of the traffic participant, wherein the three dimensional coordinates of the traffic participant provided at a future time frame are analyzed with previous three dimensional coordinates of the traffic participants provided at a previous time frame to determine a difference in distance traveled, wherein the difference is distance traveled is analyzed to estimate an average velocity of the traffic participant.

19. A non-transitory computer-readable storage medium storing instructions that when executed by a processor performs actions, comprising:

receiving an image of a surrounding environment of a vehicle from a vehicle camera system and a set of object coordinates of at least one object determined within the surrounding environment of the vehicle from a vehicle laser projection system;

determining a portion of the image as object space based on the set of object coordinates and at least one corresponding set of image coordinates, wherein the at least one corresponding set of image coordinates are two-dimensional coordinates that represent at least one pixel of the image that corresponds to at least one location within the image that is associated to the set of object coordinates;

filtering the object space to identify a traffic related object;

filtering the object space to identify a traffic related object;

determining a three dimensional position of the traffic related object, wherein determining the three dimensional position of the traffic related object includes projecting the image and projecting a plurality of laser beam points from an image laser projection system to a plurality of pixels of the projected image of the surrounding environment of the vehicle, wherein determining a three dimensional position of the traffic related object includes determining a subset of clusters of a plurality of clusters found closest to a camera of the vehicle camera system based on an evaluation of a mean laser beam length value for each cluster against at least one predetermined laser length value; and

classifying the traffic related object as at least one of;

a traffic participant, or a non-traffic participant;

tracking the traffic participant based on a three dimensional position of the traffic related object classified as the traffic participant, wherein a predicted position of the traffic participant relative to the vehicle is determined based on tracking the traffic participant; and

controlling at least one component of the vehicle to provide at least one vehicle collision avoidance function.

20. The non-transitory computer readable storage medium of claim 19, wherein tracking the traffic participant includes receiving three dimensional coordinates of the traffic participant, wherein the three dimensional coordinates of the traffic participant provided at a future time frame are analyzed with previous three dimensional coordinates of the traffic participants provided at a previous time frame to determine a difference in distance traveled, wherein the difference is distance traveled is analyzed to estimate an average velocity of the traffic participant.