Vehicle detection, tracking and localization based on enhanced anti-perspective transformation

• US 9,928,426 B1
• Filed: 09/16/2016
• Issued: 03/27/2018
• Est. Priority Date: 09/16/2016
• Status: Active Grant

• Current Assignee: Hong Kong Applied Science And Technology Research Institute Co., Ltd.
• Original Assignee: Hong Kong Applied Science And Technology Research Institute Co., Ltd.
• Inventors: Li, Hailiang, Lei, Zhi Bin, Liu, Yang, Chiu, Man Yau, Wu, Kangheng
• Primary Examiner(s): Safaipour, Bobbak
• Application Number:
  US15/267,875
• Publication Number:
  US 20180082131A1
• Time in Patent Office: 557 Days
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/32 Normalisation of the pattern dimensions
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/625 License plates

Abstract

A method for a vehicle management system with a perspective view camera generating perspective images of vehicles in a traffic direction, including performing an anti-perspective transform on the perspective images; separating each anti-perspective image into sub-images along one of a radial direction and a tangential direction relative to the traffic direction; determining a scale factor for each sub-image based upon measuring a scale of each vehicle at plural positions in one image using a frame difference method performing a scale transform for each sub-image using the corresponding scale factors for the sub-image; combining each of the scale transformed sub-images for each vehicle into corresponding enhanced anti-perspective images; performing vehicle detection for each vehicle based a combination of the enhanced anti-perspective images in which tracking is enhanced with an optimized detection box size range determined by the enhanced anti-perspective images; and performing vehicle tracking for each vehicle based on the combination of the enhanced anti-perspective images in which detecting is enhanced with the optimized detection box size range determined by the enhanced anti-perspective images.

Claims

1. A method for a vehicle management system with a perspective view camera generating perspective images of vehicles in a traffic direction, comprising:

performing an anti-perspective transform on the perspective images;

separating each anti-perspective image into sub-images along one of a radial direction and a tangential direction relative to the traffic direction;

determining a scale factor for each sub-image based upon measuring a scale of each vehicle at plural positions in one image using a frame difference method performing a scale transform for each sub-image using the corresponding scale factors for the sub-image;

combining each of the scale transformed sub-images for each vehicle into corresponding enhanced anti-perspective images;

performing vehicle detection for each vehicle based a combination of the enhanced anti-perspective images in which detection is enhanced with an optimized detection box size range determined by the enhanced anti-perspective images; and

performing vehicle tracking for each vehicle based on the combination of the enhanced anti-perspective images in which detection is enhanced with the optimized detection box size range determined by the enhanced anti-perspective images.

2. The method of claim 1, further comprising;

performing vehicle localization for each vehicle by integrating each of the detected and/or tracked results from a same one of the vehicles.

3. The method of claim 1, further comprising:

performing landmark localization to localize a vehicle plate area for each vehicle based upon at least one of the detected and tracked results; and

locating a vehicle license plate around the localized plate area of each vehicle based upon the landmark localization.

4. The method of claim 2, further comprising:

performing landmark localization to localize a vehicle plate area for each vehicle based upon at least one of the detected and tracked results; and

locating a vehicle license plate around the localized plate area of each vehicle based upon the landmark localization.

5. The method of claim 3, further comprising generating higher resolution images based on images of the located vehicle license plates.

6. The method of claim 4, further comprising generating higher resolution images based on images of the located vehicle license plates.

7. The method of claim 1, wherein the performing of the anti-perspective transform for the perspective images comprises:

performing an anti-perspective transform which is a combination of a 2D scale, a rotation, and a translation transformation which is described as;

8. The method of claim 1, wherein the performing of the scale transform for each sub-image comprises:

dividing each anti-perspective image into n parts (P₁, P₂ . . . P_n);

wherein when n is an even number, RSF<sub>n/2−

1</sub>=1, and when n is an odd number, RSF_(n+1)/2=1;

a size of n parts can be equal to each other or not;

taking the frame difference result when there is only one vehicle moving in camera view, wherein vehicle length L_i is defined by the vehicle length measured by pixels when the rear of the vehicle is inside Pi;

wherein if n is an even number, when n+1>

i>

n/2, RSF_i=L_i/L_n/2-1, and when n/2>

i>

0, RSF_i=L_n/2-1/L_i; and

if n is an odd number, when n+1>

i>

(n+1)/2, RSF_i=L_i/L_(n+1)/2, and when (n+1)/2>

i>

0, RSF_i=L_(n+1)/2/L_i.

9. The method of claim 7, wherein the performing of the scale transform for each sub-image comprises:

dividing each anti-perspective image into n parts (P₁, P₂ . . . P_n);

wherein when n is an even number, RSF_n/2-1=1, and when n is an odd number, RSF_(n+1)/2=1;

a size of n parts can be equal to each other or not;

taking the frame difference result when there is only one vehicle moving in camera view, wherein vehicle length L_i is defined by the vehicle length measured by pixels when the rear of the vehicle is inside Pi;

wherein if n is an even number, when n+1>

i>

n/2, RSF;

=L_n/2-1, and when n/2>

i>

0, RSF_i=L_n/2-1/L_i; and

if n is an odd number, when n+1>

i>

(n+1)/2, RSF_i=L_i/L_(n+1)/2, and when (n+1)/2>

i>

0, RSF_i=L_(n+1)/2/L_i.

10. The method of claim 2, wherein the performing of the vehicle localization for each vehicle comprises:

given a vehicle image I, there is a ground truth shape S corresponding to this vehicle image, training t forests to get a feature mapping function φ

^t(I,S^t-1) for each tree to generate local features, wherein t indicates that t stages are trained, and n forests are trained (here n forests are equal n experts) in each stage;

calculating the error of the last stage Δ

S^t from ground truth and previous stage estimated shape S^t-1, and also defined as the target shape increments {Δ

S^t=S−

S^t-1}, wherein the shape error Δ

S is relative to shape-indexed features;

the shape-indexed features being used for construction of a binary tree and then local features are generated from the leaves of a binary; and

training a linear projection matrix W^t to get the feature mapping between the local binary feature and shape error Δ

S by linear regression.

11. The method of claim 4, wherein the performing of the vehicle localization for each vehicle comprises:

given a vehicle image I, there is a ground truth shape S corresponding to this vehicle image, training t forests to get a feature mapping function φ

^t(I,S^t-1) for each tree to generate local features, wherein t indicates that t stages are trained, and n forests are trained (here n forests are equal n experts) in each stage;

calculating the error of the last stage Δ

S^t from ground truth and previous stage estimated shape S^t-1, and also defined as the target shape increments {Δ

S^t=S−

S^t-1}, wherein the shape error Δ

S is relative to shape-indexed features;

the shape-indexed features being used for construction of a binary tree and then local features are generated from the leaves of a binary; and

training a linear projection matrix W^t to get the feature mapping between the local binary feature and shape error Δ

S by linear regression.

12. The method of claim 3, further comprising recognizing colors of the located vehicle license plates based on images of the located vehicle license plates.

13. The method of claim 4, further comprising recognizing colors of the located vehicle license plates based on images of the located vehicle license plates.

14. An intelligent traffic surveillance system for vehicle management with a perspective view camera generating perspective images of vehicles in a traffic direction, comprising:

an anti-perspective transform unit to perform an anti-perspective transform on the perspective images, and separating each anti-perspective image into sub-images along one of a radial direction and a tangential direction relative to the traffic direction;

a scale factor determining unit to determine a scale factor for each sub-image based upon measuring a scale of each vehicle at plural in one image using a frame difference method performing a scale transform for each sub-image using the corresponding scale factors for the sub-image;

an enhanced anti-perspective transformation unit combining each of the scale transformed sub-images for each vehicle into corresponding enhanced anti-perspective images;

a vehicle detector performing vehicle detection for each vehicle based a combination of the enhanced anti-perspective images in which detecting is enhanced with an optimized detection box size range determined by the enhanced anti-perspective images; and

a vehicle tracker performing vehicle tracking for each vehicle based on the combination of the enhanced anti-perspective images in which tracking is enhanced with the optimized detection box size range determined by the enhanced anti-perspective images.

15. The traffic surveillance system of claim 14, further comprising;

a vehicle localizer performing vehicle localization for each vehicle by integrating each of the detected and/or tracked results from a same one of the vehicles.

16. The traffic surveillance system of claim 15, wherein the vehicle localizer:

performs landmark localization to localize a vehicle plate area for each vehicle based upon at least one of the detected and tracked results; and

locates a vehicle license plate around the localized plate area of each vehicle based upon the landmark localization.

17. The traffic surveillance system of claim 16, further comprising a color recognizer recognizing colors of the located vehicle license plates based on images of the located vehicle license plates.