Dynamic Bayesian Networks for vehicle classification in video

• US 9,466,000 B2
• Filed: 05/21/2012
• Issued: 10/11/2016
• Est. Priority Date: 05/19/2011
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Bhanu, Bir, Kafai, Mehran
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): HUANG, FRANK F
• Application Number:
  US14/117,678
• Publication Number:
  US 20140085475A1
• Time in Patent Office: 1,604 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights

Abstract

A system and method for classification of passenger vehicles and measuring their properties, and more particularly to a stochastic multi-class vehicle classification system, which classifies a vehicle (given its direct rear-side view) into one of four classes Sedan, Pickup truck, SUV/Minivan, and unknown, and wherein a feature pool of tail light and vehicle dimensions is extracted which feeds a feature selection algorithm to define a low-dimensional feature vector, and the feature vector is then processed by a Hybrid Dynamic Bayesian Network (HDBN) to classify each vehicle.

Claims

1. A method for vehicle classification comprising:

detecting at least three subcomponents, the at least three subcomponents comprising vehicle detection, license plate extraction, and tail light extraction;

using a Gaussian mixture model approach for detection of a moving object, wherein the Gaussian mixture model comprises Gaussian distributions to determine if a pixel is more likely to belong to a background model or not, and an AND approach, which determines a pixel as background only if the pixel falls within three standard deviations for all the components in all three R, G, and B color channels;

validating detected moving objects by using a simple frame differencing approach;

removing shadows and erroneous pixels by finding a vertical axis of symmetry using an accelerated version of Loy'"'"'s symmetry and readjusting a bounding box containing a mask with respect to an axis of symmetry, wherein if the shadow is behind the vehicle, removing the shadow using geometrical assumptions such as camera location, object geometry, and ground surface geometry, and wherein given the vehicle rear mask, measuring a height and width of the bounding box, and area of the mask;

inputting the license plate corner coordinates into an algorithm for license plate extraction, and adding a Gaussian noise with constant mean 0 and variance 0.2 times width to the license plate width measurement; and

performing a Bayesian network analysis on the at least three detected subcomponents for a plurality of vehicle classifications, wherein the Bayesian network analysis is defined as a directed acyclic graph G=(V, E), where nodes represent random variables from a domain of interest and arcs symbolize direct dependencies between the random variables.

2. The method of claim 1, wherein for a Bayesian network with n nodes X₁, X₂, . . . X_n, a full joint distribution is defined as:

3. The method of claim 2, further comprising using a K2 algorithm to determine a sub-optimal structure.

4. The method of claim 3, comprising:

generating a manually structured network and comparing the sub-optimal structure to the manually structured network.

5. The method of claim 4, comprising:

adding a temporal dimension to the Bayesian network to create a Dynamic Bayesian Network (DBN structure).

6. The method of claim 5, comprising:

replicating the Bayesian network with time-dependent random variables over T time slices, and wherein a new set of arcs defining the transition model is used to determine how various random variables are related between time slices.

7. The method of claim 6, wherein the DBN structure is defined as following:

for each time slice t_{i, i=1, 2, . . . , 5};

each feature x_i^t is the parent of x_i^t+1;

C^t is the parent of C^t+1; and

all intra slice dependencies also hold as inter time slices except for arcs from time slice t hidden nodes to time slice t+1 observed nodes.

8. The method of claim 7, comprising:

generating a Hybrid Dynamic Bayesian Network (HDBN), which consists of discrete and continuous nodes, and wherein parameters of the HDBN are required before classification is performed.

9. The method of claim 8, wherein the plurality of vehicle classifications include at least one or more of the following:

Sedan, Pickup, and SUV or Minivan.

10. The method of claim 1, comprising:

generating a complete system framework, which includes data collection, feature extraction, feature selection, and classification.

11. The method of claim 10, comprising:

extracting features from images of a rear view of a vehicle.

12. The method of claim 11, comprising:

extracting tail light features, wherein the tail light features include separately for each tail light a width, a distance from the license plate, and an angle between the tail light and the license plate.

13. The method of claim 12, comprising:

extracting license plate features, the license plate features including the license plate location and a size thereof is used as a reference to enable comparison and normalize the tail light features and the vehicle size values.

14. The method of claim 1, comprising:

performing a tail light extraction by fusing two methods, a first method wherein an image is converted to hue, saturation and value (HSV) color space and classifying pixels into three main color groups red, green, and blue, and a second method, which defines the red level of each pixel as

15. The method of claim 14, comprising:

generating a bounding box surrounding each tail light by combining results from both methods and checking if the regions with high redness can be a tail light.

16. The method of claim 15, wherein if the vehicle body color is red itself, estimating a vehicle color using a hue, saturation and value (HSV) color space histogram analysis approach, which determines if the vehicle is red or not, and if a red vehicle is detected, the tail light detection component is enhanced by adding an extra level of post-processing which includes thresholding, color segmentation, removing large and small regions, and symmetry analysis, and after the tail lights are detected, separately computing width, centroid, and distance and angle with the license plate for both left and right tail lights.

17. The method of claim 16, comprising:

extracting one or more features for each image frame, and normalizing each distance with respect to the license plate width, the one or more features comprising;

perpendicular distance from license plate centroid to a line connecting two tail light centroids, right tail light width, left tail light width, right tail light license plate angle, left tail light license plate angle, right tail light license plate distance, left tail light license plate distance, bounding box width, bounding box height, license plate distance to bounding box bottom side, and/or vehicle mask area.

18. The method of claim 17, comprising:

optimizing an evaluation criterion to improve classification accuracy, shorten computational time, reduce measurements costs, and/or eliminate use of dimensionality.

19. The method of claim 18, comprising:

performing a Sequential Floating Forward Selection (SFFS), which is a deterministic statistical pattern recognition (SPR) feature selection method and returning a single suboptimal solution.

20. The method of claim 19, wherein SFFS starts from an empty set and adding features that increase accuracy and removing the least significant features during a conditional exclusion.

21. The method of claim 20, comprising:

a stopping condition, which halts the SFFS algorithm to limit the number of feature selection iterative steps to 2^n-1, wherein n is the number of features and defining a correct classification rate (CCR) threshold of b %, where b is greater than the CCR of the case when all features are used.

22. The method of claim 21, wherein the classification is a two stage approach, which consists of estimating the Gaussian distribution parameters of the distance to the nearest neighbor for all vehicles in the training dataset and determining if a vehicle test case is known or unknown by initially computing the distance to its nearest neighbor.

23. The method of claim 22, comprising:

if the distance does not lie within 4 standard deviations of a mean (μ

±

4σ

), classifying it as unknown, and if the vehicle is classified as known it is a candidate for a second stage of classification.

24. A system for classification of vehicles comprising:

a camera configured to capture images of at least one moving object; and

a computer processing unit configured to

detecting at least three subcomponents from the captured images of the at least one moving object, the at least three subcomponents comprising vehicle detection, license plate extraction, and tail light extraction;

use a Gaussian mixture model approach for detection of a moving object, wherein the Gaussian mixture model comprises Gaussian distributions to determine if a pixel is more likely to belong to a background model or not, and an AND approach, which determines a pixel as background only if the pixel falls within three standard deviations for all the components in all three R, G, and B color channels;

validating detected moving objects by using a simple frame differencinq approach;

remove shadows and erroneous pixels by finding a vertical axis of symmetry using an accelerated version of Loy'"'"'s symmetry and readiusting a bounding box containing a mask with respect to an axis of symmetry, wherein if the shadow is behind the vehicle, removing the shadow using geometrical assumptions such as camera location, object geometry, and ground surface geometry, and wherein given the vehicle rear mask, measuring a height and width of the bounding box, and area of the mask;

input the license plate corner coordinates into an algorithm for license plate extraction, and adding a Gaussian noise with constant mean 0 and variance 0.2 times width to the license plate width measurement; and

perform a Bayesian network analysis on the at least three detected subcomponents for a plurality of vehicle classifications, wherein the Bayesian network is defined as a directed acyclic graph G=(V, E), where nodes represent random variables from a domain of interest and arcs symbolize direct dependencies between random variables.

25. The system of claim 24, wherein the computer processing unit further comprises:

a memory arrangement, a processing unit and an optional display unit configured to display data and/or classification of the distinct structures in the class.

26. The system of claim 24, wherein the camera is configured to capture video images.

28. The system of claim 24, wherein for a Bayesian network with n nodes X₁, X₂, . . . X_n, a full joint distribution is defined as:

29. The system of claim 28, further comprising using a K2 algorithm to determine a sub-optimal structure;

generating a manually structured network and comparing the sub-optimal structure to the manually structured network; and

adding a temporal dimension to the Bayesian network to create a Dynamic Bayesian Network (DBN structure).

30. The system of claim 29, comprising:

replicating the Bayesian network with time-dependent random variables over T time slices, and wherein a new set of arcs defining the transition model is used to determine how various random variables are related between time slices; and

wherein the DBN structure is defined as following;

for each time slice t_{i, i=1, 2, . . . , 5};

each feature x_i^t is the parent of x_i^t+1;

C^t is the parent of C^t+1; and

all intra slice dependencies also hold as inter time slices except for arcs from time slice t hidden nodes to time slice t+1 observed nodes.

31. The system of claim 30, comprising:

generating a Hybrid Dynamic Bayesian Network (HDBN), which consists of discrete and continuous nodes, and wherein parameters of the HDBN are required before classification is performed, and wherein the plurality of vehicle classifications include at least one or more of the following;

Sedan, Pickup, and SUV or Minivan.

32. The system of claim 24, comprising:

generating a complete system framework, which includes data collection, feature extraction, feature selection, and classification;

extracting features from images of a rear view of a vehicle;

extracting tail light features, wherein the tail light features include separately for each tail light a width, a distance from the license plate, and an angle between the tail light and the license plate; and

extracting license plate features, the license plate features including the license plate location and a size thereof is used as a reference to enable comparison and normalize the tail light features and the vehicle size values.

27. A computer program product comprising a non-transitory computer usable medium having a computer readable code embodied therein for classification of passenger vehicles and measuring their properties from a rear view video frame, the computer readable program code is configured to execute a process, which:

detects at least three subcomponents from the captured images of the at least one moving object, the at least three subcomponents comprising vehicle detection, license plate extraction, and tail light extraction;

uses a Gaussian mixture model approach for detection of a moving object, wherein the Gaussian mixture model comprises Gaussian distributions to determine if a pixel is more likely to belong to a background model or not, and an AND approach, which determines a pixel as background only if the pixel falls within three standard deviations for all the components in all three R, G, and B color channels;

validates detected moving objects by using a simple frame differencing approach;

removes shadows and erroneous pixels by finding a vertical axis of symmetry using an accelerated version of Loy'"'"'s symmetry and readjusting a bounding box containing a mask with respect to an axis of symmetry, wherein if the shadow is behind the vehicle, removing the shadow using geometrical assumptions such as camera location, object geometry, and ground surface geometry, and wherein given the vehicle rear mask, measuring a height and width of the bounding box, and area of the mask;

inputs the license plate corner coordinates into an algorithm for license plate extraction, and adding a Gaussian noise with constant mean 0 and variance 0.2 times width to the license plate width measurement; and

performs a Bayesian network analysis on the at least three detected subcomponents for a plurality of vehicle classifications, which are known, wherein the Bayesian network is defined as a directed acyclic graph G=(V, E), where nodes represent random variables from a domain of interest and arcs symbolize direct dependencies between random variables.