Video and lidar target detection and tracking system and method for segmenting moving targets

• US 9,111,444 B2
• Filed: 10/31/2012
• Issued: 08/18/2015
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Kaganovich, Vitaliy M.
• Primary Examiner(s): Ratcliffe, Luke
• Assistant Examiner(s): Abraham, Samantha K
• Application Number:
  US13/665,560
• Publication Number:
  US 20140118716A1
• Time in Patent Office: 1,021 Days
• Field of Search: 356/4.01, 356/3.01, 356/5.01, 356/9, 356/625
• US Class Current: 1/1
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20032 Median filtering
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/056 with provision for distinguishing direction of travel

Abstract

Embodiments of video and LIDAR target detection and tracking system and method for segmenting targets are generally described herein. In some embodiments, the system may perform moving target detection using three-frame difference processing on frames of LIDAR range images of a scene to identify a moving target and determine an initial position estimate for the moving target. The system may also perform segmentation of the moving target using a single frame of the LIDAR range images to generate a segmented target that may include a target mask and target centroid position. After segmentation, moving target registration including motion compensation may be performed by registering the segmented target in multiple LIDAR frames to determine a target position, a target orientation and a target heading. In some embodiments, a three-dimensional model of the moving target may be generated and added to a sequence of video frames.

Claims

1. A target detection and tracking system comprising:

a moving target detection module to perform moving target detection using three-frame difference processing on frames of LIDAR range images of a scene to identify a moving target and determine an initial position estimate for the moving target,

each LIDAR range image being formed from illumination of the scene with infrared laser light and analysis of round trip travel time of the infrared laser light collected from the scene,

each LIDAR range image comprising a depth map of the scene, each LIDAR range image being unaffected by shadows, surface markings, color, and lighting; and

a segmentation module to perform segmentation of the moving target using a single frame of the LIDAR range images to generate a segmented target including a target mask and target centroid position.

2. The system of claim 1 wherein the segmentation module is configured to:

transform the single frame of the LIDAR range images within a track window to a height image wherein pixel values of the height image correspond to heights of objects in the scene;

threshold out pixels of the height image to remove ground pixels and occlusions in the height image to generate a segmented target image;

apply a median filter to the single frame of the LIDAR range image to fill in gaps;

identify depth discontinuities in the LIDAR range image after the gaps are filled;

null-out pixels of the segmented target image that correspond to the depth discontinuities; and

establish boundaries within the segmented target image to identify potential targets.

3. The system of claim 2 wherein the segmentation module is further configured to:

filter the potential targets based on a predetermined size threshold to identify at least one target of interest; and

generate the segmented target image and a target mask for each of the identified targets of interest based on the filtering.

4. The system of claim 3 wherein the segmentation module is further configured to:

determine a three-dimensional (3D) target centroid for each of the targets of interest within the range image;

set the 3D target centroid as the target location; and

update a 3D velocity of the target.

5. The system of claim 4 wherein the segmentation module is further configured to determine an image gradient of the LIDAR range image after the gaps are filled in to identify the depth discontinuities, and

wherein the boundaries are established within the segmented target image after nulling-out the pixels that correspond to depth discontinuities to identify potential targets.

6. The system of claim 2 wherein the segmentation module is further configured to:

generate histogram values of the height image;

estimate a ground height from the histogram values; and

determine one or more height thresholds for the height image for use in thresholding out pixel values.

7. The system of claim 4 further comprising a motion compensation module,

wherein after segmentation, the motion compensation module is configured to perform moving target registration including motion compensation by registering the segmented target in multiple LIDAR frames to determine a target position, a target orientation and a target heading.

8. The system of claim 7 wherein the motion compensation module is further configured to perform an interactive closest point (ICP) to refine frame-to-frame registration.

9. A video and LIDAR target detection, tracking and segmentation method comprising:

performing moving target detection using three-frame difference processing on frames of LIDAR range images of a scene to identify a moving target and determine an initial position estimate for the moving target,

each LIDAR range image being formed from illumination of the scene with infrared laser light and analysis of round trip travel time of the infrared laser light collected from the scene,

each LIDAR range image comprising a depth map of the scene, each LIDAR range image being unaffected by shadows, surface markings, color, and lighting; and

performing segmentation of the moving target using a single frame of the LIDAR range images to generate a segmented target including a target mask and target centroid position.

10. The method of claim 9 wherein performing segmentation comprises:

transforming the single frame of the LIDAR range images within a track window to a height image wherein pixel values of the height image correspond to heights of objects in the scene;

thresholding-out pixels of the height image to remove ground pixels and occlusions in the height image to generate a segmented target image;

applying a median filter to the single frame of the LIDAR range image to fill in gaps;

identifying depth discontinuities in the LIDAR range image after the gaps are filled, the depth discontinuities being identified using an image gradient of the LIDAR range image;

nulling-out pixels of the segmented target image that correspond to the depth discontinuities; and

establishing boundaries within the segmented target image to identify potential targets.

11. The method of claim 10 wherein performing segmentation further comprises:

filtering the potential targets based on a predetermined size threshold to identify at least one target of interest; and

generating the segmented target image and a target mask for each of the identified targets of interest based on the filtering.

12. The method of claim 11 wherein performing segmentation further comprises:

determining a three-dimensional (3D) target centroid for each of the targets of interest within the range image;

setting the 3D target centroid as the target location; and

updating a 3D velocity of the target.

13. The method of claim 12 wherein performing segmentation further comprises determining an image gradient of the LIDAR range image after the gaps are filled in to identify the depth discontinuities, and

wherein the boundaries are established within the segmented target image after nulling-out the pixels that correspond to depth discontinuities to identify potential targets.

14. The method of claim 10 wherein the segmentation further comprises:

generating histogram values of the height image;

estimating a ground height from the histogram values; and

determining one or more height thresholds for the height image for use in thresholding out pixel values.

15. The method of claim 12 further comprising, after segmentation, performing moving target registration including motion compensation by registering the segmented target in multiple LIDAR frames to determine a target position, a target orientation and a target heading.

16. The method of claim 15 wherein performing moving target registration further comprises performing an interactive closest point (ICP) to refine frame-to-frame registration.

17. A method for target image segmentation comprising:

transforming a LIDAR range image within a track window to a height image wherein pixel values of the height image correspond to heights of objects in a scene,

the LIDAR range image being formed from illumination of the scene with infrared laser light and analysis of round trip travel time of the infrared laser light collected from the scene,

the LIDAR range image comprising a depth map of the scene,

the LIDAR range image being unaffected by shadows, surface markings, color, and lighting;

thresholding-out pixels of the height image to remove ground pixels and occlusions in the height image to generate a segmented target image;

applying a median filter to the LIDAR range image to fill in gaps;

identifying depth discontinuities in the LIDAR range image after the gaps are filled, the depth discontinuities being identified using an image gradient of the LIDAR range image;

nulling-out pixels of the segmented target image that correspond to the depth discontinuities; and

establishing boundaries within the segmented target image to identify potential targets.

18. The method of claim 17 further comprising:

filtering the potential targets based on a predetermined size threshold to identify at least one target of interest; and

generating the segmented target image and a target mask for each of the identified targets of interest based on the filtering.

19. The method of claim 18 further comprising:

determining an image gradient of the LIDAR range image after the gaps are filled in to identify the depth discontinuities, and

wherein the boundaries are established within the segmented target image after nulling-out the pixels that correspond to depth discontinuities to identify potential targets.

20. The method of claim 18 further comprising:

generating histogram values of the height image;

estimating a ground height from the histogram values; and

determine one or more height thresholds for the height image for use in thresholding out pixel values.