OBJECT CLASSIFICATION WITH CONSTRAINED MULTIPLE INSTANCE SUPPORT VECTOR MACHINE

• US 20150242708A1
• Filed: 02/21/2014
• Published: 08/27/2015
• Est. Priority Date: 02/21/2014
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Duan, Kun, Marchesotti, Luca
• Granted Patent: US 9,443,169 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles

Abstract

This disclosure provides method and systems of classifying a digital image of an object. Specifically, according to one exemplary embodiment, an object classifier is trained using a constrained MI-SVM (multiple instance-support vector machine) approach whereby training images of objects are sampled to generate a collection of image regions associated with an object type and viewpoint, and the classifier is trained to determine an appropriate mid-level representation of the training image which is discriminative.

Claims

1. A computer implemented method of classifying a digital image of an object, the method comprising:

a) receiving a digital image of an object to be classified with a processor; and

b) classifying the digital image with a constrained multiple-instance support vector machine (MI-SVM) classifier, the constrained MI-SVM classifier having been automatically trained using a plurality of training images, the training images including a plurality of object types from a plurality of viewpoints, each training image including an image of an object associated with one of the plurality of object types and one of the plurality of object viewpoints, an associated object type label and an associated viewpoint label, the constrained MI-SVM classifier trained by sampling each training image to generate a bag of image regions associated with each training image, discovering s discriminative image region associated with each training image, and generating a collection of discriminative image regions for each of the plurality of object types and each of the plurality of viewpoints.

2. The computer implemented method of classifying a digital image according to claim 1, wherein a plurality of classifiers are associated with the constrained MI-SVM classifier, each classifier associated with one of the plurality of object viewpoints.

3. The computer implemented method of classifying a digital image according to claim 1, wherein the plurality of object types are associated with a plurality of vehicle types.

4. The computer implemented method of classifying a digital image according to claim 3, wherein the plurality of viewpoints include two or more of a front viewpoint of the vehicle, a rear viewpoint of the vehicle, and side viewpoint of the vehicle.

5. The computer implemented method of classifying a digital image according to claim 1, wherein the constrained MI-SVM classifier is trained using an iterative process that initially selects an initial discriminative image region for a first object type at the plurality of viewpoints and iteratively selects subsequent discriminative image regions of the first object type at the plurality of viewpoints where the selection of a subsequent discriminate image region is constrained by one or more characteristics of selected discriminative image regions associated with other viewpoints of the first object type.

6. The computer implemented method of classifying a digital image according to claim 5, wherein the one or more characteristics include location, geometry and appearance of the subsequent discriminative image region relative to the selected discriminate image region associated with other viewpoints of the first object type.

7. The computer implemented method of classifying a digital image according to claim 1, wherein the MI-SVM classifier is formulated as

$\underset{\left\{w^{\left(v\right)},C^{\left(v\right)}\right\}}{\min }$

∑

v=1V

(12



w(v)

2+C(v)

∑

I(v)

ξ

I(v))
s.t. (subject to)

Y_I·

(w^(v)·

x_I^SI+b^v)≧

1−

ξ

_I(v)

where W represents a hyperplane slope of a hyperplane associated with the MI-SVM,

C is a cross validated selective constant,

b represents an offset of the hyperplane,

I represents a bag of image regions associated with a training image,

S_I represents an image region selected for bag I;

v represents a view point,

V represents a total number of viewpoints v,

x_I^SI represents an image region selected from bag I,

Y_I represents the object type label associated with bag I, and

ξ

_I(v) represents a fixed value.

8. The computer implemented method of classifying a digital image according to claim 7, wherein an iterative process is used between training of the w^(v) and reasoning about image region selection based on positive bags including at least one discriminative image region including a CRF (Conditional Random Field), where

9. The computer implemented method of classifying a digital image according to claim 1, wherein the sampling of a plurality of image regions is one of uniformly sampling along image dimensions, uniform sampling associated with an area of the image determined to include an object, and sampling based on a detection response map.

10. The computer implemented method of classifying a digital image according to claim 1, wherein one or more of a bounding box and a DPM (Deformable Part Model) is used to train the MI-SVM.

11. The computer implemented method of classifying a digital image according to claim 1, wherein the constrained MI-SVM classifier is configured to generate a plurality of collections of discriminative image region representations, each collection including a plurality of discriminative image regions associated with an attribute of vehicle type, the attribute including one of a vehicle model, a vehicle feature and a vehicle style, and each collection associated with a semantic indicating the attribute associated with each collection.

12. An image processing system comprising:

a controller configured to receive a digital image of a vehicle including one of a plurality of vehicle types, the controller configured to execute instructions to perform a method of classifying the digital image of the vehicle as one of the plurality of vehicle types, the method comprising;

a) receiving a digital image of an object to be classified; and

b) with a processor, classifying the digital image with a constrained multiple-instance support vector machine (MI-SVM) classifier, the constrained MI-SVM classifier having been automatically trained using a plurality of training images, the training images including a plurality of object types from a plurality of viewpoints, each training image including an image of an object associated with one of the plurality of object types and one of the plurality of object viewpoints, an associated object type label and an associated viewpoint label, the constrained MI-SVM classifier trained by sampling each training image to generate a bag of image regions associated with each training image, discovering s discriminative image region associated with each training image, and generating a collection of discriminative image regions for each of the plurality of object types and each of the plurality of viewpoints.

13. The image process system according to claim 12, wherein a plurality of classifiers are associated with the constrained MI-SVM classifier, each classifier associated with one of the plurality of vehicle types.

14. The image process system according to claim 12, wherein the plurality of vehicle types are associated with models of vehicles.

15. The image process system according to claim 12, wherein the plurality of viewpoints include two or more of a front viewpoint of the vehicle, a rear viewpoint of the vehicle, and side viewpoint of the vehicle.

16. The image process system according to claim 12, wherein the constrained MI-SVM classifier is trained using an iterative process that initially selects an initial discriminative image region for a first vehicle type at the plurality of viewpoints and iteratively selects subsequent discriminative image regions of the first vehicle type at the plurality of viewpoints where the selection of a subsequent discriminate image region is constrained by one or more characteristics of selected discriminative image regions associated with other viewpoints of the first vehicle type.

17. The image processing system according to claim 16, wherein the one or more characteristics include location, geometry and appearance of the subsequent discriminative image region relative to the selected discriminated image region associated with other viewpoints of the first vehicle type.

18. The image processing system according to claim 12, wherein the MI-SVM classifier is formulated as:

$\underset{\left\{w^{\left(v\right)},C^{\left(v\right)}\right\}}{\min }<br><br>\underset{v=1}{\overset{V}{\sum <br><br>}}<br><br>\left(\frac{1}{2}<br><br>{<br><br>w^{\left(v\right)}<br><br>}^2+C^{\left(v\right)}<br><br>\underset{I^{\left(v\right)}}{\sum <br><br>}<br><br>{\mathrm{\xi <br><br>}}_{I^{\left(v\right)}}\right)$
s.t. (such that)

Y_I·

(w^(v)·

x_I^SI+b^v)≧

1−

ξ

_I(v)

where W represents a hyperplane slope of a hyperplane associated with the MI-SVM,

C is a cross validated selective constant,

b represents an offset of the hyperplane,

I represents a bag of image regions associated with a training image,

S_I represents an image region selected for bag I;

v represents a view point,

V represents a total number of viewpoints v,

x_I^SI represents an image region selected from bag I,

Y_I represents the vehicle type label associated with bag I, and

ξ

_I(v) represents a fixed value.

19. The image processing system according to claim 18, wherein an iterative process is used between training of the w^(v) and reasoning about image region selection based on positive bags including at least one discriminative image region including a CRF (Conditional Random Field), where

$E$

({LI}|{vi})=∑

I

ϕ



(LI|vI)+∑

I

∑

J

δ



(LI,LJ|vI,vJ)
where

φ

(L_I|v_I)=−

(w^(vI)·

x_I^LI+b^(vI)), and

the pairwise terms (L_I,L_J|v_I, v_J) account for geometry and appearance constraints relative to a plurality of viewpoints,

L_I and L_J represent variables based on indices of an image region selected within bag I,

E represents an energy function which is minimized,

δ

is an energy term, and

φ

is an energy term.

20. The image processing system according to claim 12, wherein the sampling of a plurality of image regions is one of uniformly sampling along image dimensions, uniform sampling associated with an area of the image determined to include a vehicle, and sampling based on a detection response map.

21. The image processing system according to claim 12, wherein one or more of a bounding box and a DPM (Deformable Part Model) is used to train the MI-SVM.

22. The image processing system according to claim 12, wherein the constrained MI-SVM classifier is configured to generate a plurality of collections of discriminative image region representations, each collection including a plurality of discriminative image regions associated with an attribute of vehicle type, the attribute including one of a vehicle model, a vehicle feature and a vehicle style, and each collection associated with a semantic indicating the attribute associated with each collection.

23. A computer implemented method of training a constrained multiple instance support vector machine (MI-SVM) classifier to classify digital images of an object, the method comprising:

a) inputting to the constrained MI-SVM classifier a plurality of training images, each training image including an object image associated with one of a plurality of object types and one of a plurality of object viewpoints, an associated object type label and associated viewpoint label;

b) the constrained MI-SVM classifier sampling each training image to generate a plurality of image regions providing a bag of image regions associated with each training image;

c) the constrained MI-SVM classifier processing the bags of image regions associated with each training image to discover a discriminative image region associated with each training image, and generate a collection of discriminative image regions for each of the plurality of object types and each of the plurality of viewpoints.

24. The computer-implemented method of training a classifier according to claim 23, wherein the plurality of object types are associated with a plurality of vehicle types.

25. The computer-implemented method of training a classifier according to claim 23, wherein the constrained MI-SVM classifier is trained using an iterative process that initially selects an initial discriminative image region for a first object type at the plurality of viewpoints and iteratively selects subsequent discriminative image regions of the first object type at the plurality of viewpoints where the selection of a subsequent discriminate image region is constrained by one or more characteristics of selected discriminative image regions associated with other viewpoints of the first object type.