Performing Cross-Validation Using Non-Randomly Selected Cases

• US 20140279734A1
• Filed: 03/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Abandoned Application

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Forman, George
• Application Number:
  US13/832,805
• Publication Number:
  US 20140279734A1
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/043 Distributed expert systems; Blackboards

Abstract

A technique to perform cross-validation using a set of randomly selected labeled cases and a set of non-randomly selected labeled cases. A training set for use during cross-validation can include cases from both sets. A test set for use during cross-validation can include cases from the randomly selected set but exclude cases from the non-randomly selected set.

Claims

1. A method for performing cross-validation, comprising:

given a first set of randomly selected labeled cases and a second set of actively selected labeled cases,

training a classifier on a training set comprising a first subset of the first set and a subset of the second set; and

measuring the performance of the classifier on a test set comprising a second subset of the first set that is disjoint relative to the first subset, the test set including no cases from the second set.

2. The method of claim 1, wherein the cross-validation is a version of k-fold cross-validation modified to accommodate active learning.

3. The method of claim 2, further comprising excluding the second set from the test set during each iteration of the modified version of k-fold cross-validation.

4. The method of claim 2, wherein the modified version of k-fold cross-validation comprises:

(a) dividing the first set into k folds;

(b) assigning a fold of the k folds to be the test set;

(c) assigning the remaining folds to be the training set;

(d) adding the subset of the second set to the training set;

(e) performing the training using the training set,

(f) performing the measuring using the test set; and

(g) repeating (b) through (f) until each fold of the k folds has been used as the test set.

5. The method of claim 2, further comprising:

dividing the second set into k folds;

selecting a fold of the k folds; and

excluding the selected fold from the training set during a corresponding iteration of the k-fold cross-validation, wherein the remaining folds constitute the subset of the second set.

6. The method of claim 1, wherein the subset of the second set comprises the entire second set.

7. The method of claim 1, further comprising:

performing active learning phases alternately with cross-validation phases; and

adding any cases labeled during the active learning phases to the second set.

8. The method of claim 1, further comprising:

selecting at least one case from a set of unlabeled cases for labeling during an active learning phase;

labeling the selected at least one case; and

adding the labeled, selected at least one case to the second set.

9. The method of claim 1, further comprising:

training a second classifier on the training set;

measuring the performance of the second classifier on the test set; and

selecting one of the classifier and the second classifier based on the measured performance of each.

10. A system, comprising:

a memory to store a first set of randomly sampled labeled cases and a second set of non-randomly sampled labeled cases; and

a cross-validation module to perform cross-validation using the first and second sets of cases, the cross-validation module configured to exclude the second set of non-randomly sampled labeled cases from a test set used in a test phase of the cross-validation.

11. The system of claim 10, wherein the second set of non-randomly sampled labeled cases is sampled using an active learning technique.

12. The system of claim 10, further comprising:

a labeling module to generate the second set of non-randomly sampled labeled cases by requesting that an expert assign labels to non-randomly sampled non-labeled cases selected from a population; and

a classifier module to generate at least one classifier based on the first and second sets,

wherein the cross-validation module is configured to perform the cross-validation on the generated at least one classifier.

13. The system of claim 10, wherein the cross-validation module is configured to include a subset of the second set of non-randomly sampled labeled cases in a training set used in a training phase of the cross-validation.

14. A non-transitory computer readable storage medium storing instructions that, when executed by a processor, cause a computer to perform cross-validation as follows:

train a classifier on a training set comprising a first subset of a first set of randomly selected labeled cases and a subset of a second set of actively selected labeled cases; and

measure the performance of the classifier on a test set comprising a second subset of the first set, the test set excluding actively selected cases.

15. The storage medium of claim 14, further comprising instructions that, when executed by a processor, cause the computer to:

perform a modified version of k-fold cross-validation on the first set of randomly selected labeled cases and the second set of actively labeled cases such that the test set in each iteration of the modified version of k-fold cross-validation excludes cases from the second set.