Confidence weighted classifier combination for multi-modal identification

• US 20060120609A1
• Filed: 12/02/2005
• Published: 06/08/2006
• Est. Priority Date: 12/06/2004
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Ivanov, Yuri, Serre, Thomas R.
• Granted Patent: US 7,742,641 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Techniques are disclosed for multi-modal identification that utilize a classifier combination framework. One embodiment of the present invention provides a multi-modal identification system that includes a collection of classifiers that classify feature streams derived from audio and/or video sources. A classifier combination scheme is used to combine the classifier outputs having varying degrees of confidence, but in a robust way by using a confidence-based weighting scheme that operates on a “per-class” basis, rather than (or in addition to) the traditional “per-classifier” basis. The system can be distributed across several machines running independent feature classifiers on the subscription basis.

Claims

1. A multi-class classifier combination system comprising:

a plurality of multi-class classifiers, each classifier for classifying a target feature captured in at least one feature stream; and

a combination module for combining classifier outputs into a joint prediction, wherein each multi-class classifier prediction is weighted in accordance with a per-class weighting scheme prior to combining.

2. The system of claim 1 further comprising:

a labeling subsystem for labeling stored feature streams accessible to the system, in accordance with a user selected labeling scheme.

3. The system of claim 1 further comprising:

a storage for storing feature streams; and

a database manager for generating and storing for each feature stream a record including at least one of a time stamp, a file name, recording conditions, and current system parameters in the storage.

4. The system of claim 1 further comprising:

a data logging subsystem for generating feature streams including at least one audio stream and one video stream from a target scene.

6. The system of claim 1 wherein a classifier is trained on a subset of training data, and then a confidence measure is computed based on the remaining subset of the training data, and the per-class weighting scheme carried out by the combination module includes weighting the classifier output by the resulting confidence measure.

7. The system of claim 1 wherein the per-class weighting scheme carried out by the combination module is based on using a confidence measure to weigh each classifier output, the confidence measure derived from a confusion matrix that represents an empirical value of the distribution of intrinsic error of the classifier on a given data set.

8. The system of claim 1 wherein the combination module combines classifier output using at least one of voting, sum of outputs, and product of outputs combination rules.

9. The system of claim 1 wherein the per-class weighting scheme carried out by the combination module is in accordance with

$P_s\approx$

∑

λ

⁢

w λ

[ ∑

ω

~ ⁢

P λ

⁡

( ω

❘

ω

~ ) ⁢

P λ

⁡

( ω

~ ❘

x ) ] ,

where P_S is the joint prediction, x is a set of features in a given scene, P_λ

({tilde over (ω

)}|x) is the prediction of the individual classifier, w_λ

is a per-classifier weight, and confidence measure, P_λ

(ω

|{tilde over (ω

)},x), is approximated by its projection, P_λ

(ω

|{tilde over (ω

)}).

10. The system of claim 1 wherein the per-class weighting scheme carried out by the combination module is in accordance with

$P_P$

( ω

❘

x ) = 1 Z ⁢

∏

λ

⁢

[ ∑

ω

~ ⁢

P λ

⁡

( ω

❘

ω

~ ) ⁢

P λ

⁡

( ω

~ ❘

x ) ] ,

where P_p is the joint prediction using a product combination rule, x is a set of features in a given scene, P_λ

({tilde over (ω

)}|x) is the prediction of the individual classifier, Z is a normalizing constant, and confidence measure, P_λ

(ω

|{tilde over (ω

)},x), is approximated by its projection, P_λ

(ω

|{tilde over (ω

)}).

5. The system of claim 5 wherein the data logging subsystem includes a detector that triggers generation of feature streams in response to detecting a target entity being present in the target scene.

11. A method for multi-class classifier combination using predictions of a plurality of multi-class classifiers, comprising:

weighting each multi-class classifier prediction in accordance with a per-class weighting scheme; and

combining the weighted predictions from two or more multi-class classifiers into a joint prediction.

12. The method of claim 11 further comprising:

storing feature streams; and

generating and storing a record for each feature stream including at least one of a time stamp, a file name, recording conditions, and current system parameters in the storage.

13. The method of claim 11 further comprising the preliminary steps of:

generating feature streams including at least one audio stream and one video stream from a target scene;

classifying a first target feature captured in at least one feature stream using a first multi-class classifier; and

classifying a second target feature captured in at least one feature stream using a second multi-class classifier.

14. The method of claim 13 wherein generating feature streams is triggered in response to detecting a target entity being present in the target scene.

15. The method of claim 11 wherein the per-class weighting scheme is based on using a confidence measure to weigh each classifier output, the confidence measure derived from a confusion matrix that represents an empirical value of the distribution of intrinsic error of the classifier on a given data set.

16. The method of claim 11 wherein the combining is carried out using at least one of voting, sum of outputs, and product of outputs combination rules.

17. The method of claim 11 wherein the per-class weighting scheme is in accordance with

$P_s\approx$

∑

λ

⁢

w λ

[ ∑

ω

~ ⁢

P λ

⁡

( ω

❘

ω

~ ) ⁢

P λ

⁡

( ω

~ ❘

x ) ] ,

where P_S is the joint prediction, x is a set of features in a given scene, P_λ

({tilde over (ω

)}|x) is the prediction of the individual classifier, W_λ

is a per-classifier weight, and confidence measure, P_λ

(ω

|{tilde over (ω

)},x), is approximated by its projection, P_λ

(ω

|{tilde over (ω

)}).

18. The method of claim 11 wherein the per-class weighting scheme is in accordance with

$P_P$

( ω

❘

x ) = 1 Z ⁢

∏

λ

⁢

[ ∑

ω

~ ⁢

P λ

⁡

( ω

❘

ω

~ ) ⁢

P λ

⁡

( ω

~ ❘

x ) ] ,

where P_p is the joint prediction using a product combination rule, x is a set of features in a given scene, P_λ

({tilde over (ω

)}|x) is the prediction of the individual classifier, Z is a normalizing constant, and confidence measure, P_λ

(ω

|{tilde over (ω

)},x), is approximated by its projection, P_λ

(ω

|{tilde over (ω

)}).

19. The method of claim 11 further comprising the preliminary steps of:

training at least one of the multi-class classifiers on a subset of training data; and

computing a confidence measure based on the remaining subset of the training data;

wherein the per-class weighting scheme includes weighting the at least one classifier output by the resulting confidence measure.

20. A multi-class classifier combination system comprising:

a means for classifying a first target feature captured in at least one feature stream using a first multi-class classifier;

a means for classifying a second target feature captured in at least one feature stream using a second multi-class classifier;

a means for weighting each multi-class classifier prediction in accordance with a per-class weighting scheme; and

a means for combining the weighted predictions from the first and second multi-class classifiers into a joint prediction.

21. A machine-readable medium encoded with instructions, that when executed by a processor, cause the processor to carry out a multi-class classifier combination process using predictions of a plurality of multi-class classifiers, the process comprising:

weighting each multi-class classifier prediction in accordance with a per-class weighting scheme; and

combining the weighted predictions from two or more multi-class classifiers into a joint prediction.