Instance-weighted mixture modeling to enhance training collections for image annotation

• US 9,646,226 B2
• Filed: 04/16/2014
• Issued: 05/09/2017
• Est. Priority Date: 04/16/2013
• Status: Active Grant

• Current Assignee: Penn State Research Foundation
• Original Assignee: Penn State Research Foundation
• Inventors: Wang, James Z., Sawant, Neela, Li, Jia
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US14/254,437
• Publication Number:
  US 20140307958A1
• Time in Patent Office: 1,119 Days
• Field of Search: 382118, 382159, 382180, 706 12, 706 20, 706 48
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting

Abstract

Automatic selection of training images is enhanced using an instance-weighted mixture modeling framework called ARTEMIS. An optimization algorithm is derived that in addition to mixture parameter estimation learns instance-weights, essentially adapting to the noise associated with each example. The mechanism of hypothetical local mapping is evoked so that data in diverse mathematical forms or modalities can be cohesively treated as the system maintains tractability in optimization. Training examples are selected from top-ranked images of a likelihood-based image ranking. Experiments indicate that ARTEMIS exhibits higher resilience to noise than several baselines for large training data collection. The performance of ARTEMIS-trained image annotation system is comparable to using manually curated datasets.

Claims

1. A method of improving precision of training data used in automated image annotation, categorization, recognition, understanding, or retrieval, comprising the steps of:

providing a digital computer;

receiving at the digital computer a plurality of digital images, all the images previously tagged with a target concept;

executing an algorithm on the digital computer to perform the following operations;

(a) extracting visual and textual features from each of the tagged images,

(b) variably weighting the images based upon the extracted features, the weight of the image reflecting a relevance of the image to the concept,

(c) learning a reference model for the images representing the target concept, wherein the reference model is a mixture model based on the weighted images, the mixture model being a parametric probabilistic model having mixture parameters, the weight of each image parameterized on the mixture parameters, and both the mixture parameters and the weights of the images being learned during the learning of the reference model to curb contributions of noisy images;

(d) retaining images with high likelihood of correct tagging based upon the reference model,

(e) iterating the steps (b) and (c) using the retained images; and

using learned reference model to train an automated image annotation, categorization, recognition, understanding, or retrieval system.

2. The method of claim 1, including the steps of

performing the following operations on the digital computer:

an initialization step wherein equal weights are assigned to all of the tagged images

retaining the tagged images by ranking the tagged images in a decreasing order of a likelihood of the image, the likelihood of each image being parameterized on the mixture parameters and reflecting the likelihood of correct tagging.

3. The method of claim 1, further comprising the step of providing a reference model for each of a plurality of concepts, wherein the tagged images are processed one concept at a time.

4. The method of claim 3, wherein an image tagged with multiple concepts is used in learning reference models of all the concepts which the image is tagged with and the image is weighted differently for different concepts.

5. The method of claim 3, wherein the reference models for the plurality of image concepts are stored in a database for future use.

6. The method of claim 1, wherein the tagged images are noisy, user-tagged images obtained from the Internet.

7. The method of claim 1, wherein the iterating stops when the algorithm converges to identify a high-density region of relevant images corresponding to a high likelihood of correct tagging, thereby improving the precision of training data selection.

8. The method of claim 1, wherein the mixture model is estimated via hypothetical local mapping.

9. The method of claim 1, wherein both the visual features and textual features are part of a jointly parameterized model using common parameters.

10. The method of claim 1, wherein the variably changing weight of each tagged image is proportional to the mixture likelihood of the tagged image.