Image search system and method for personalized photo applications using semantic networks

• US 9,330,110 B2
• Filed: 07/17/2013
• Issued: 05/03/2016
• Est. Priority Date: 07/17/2013
• Status: Expired due to Fees

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Lin, Julianna Elizabeth, Marchesotti, Luca
• Primary Examiner(s): RIFKIN, BEN M
• Application Number:
  US13/944,023
• Publication Number:
  US 20150026101A1
• Time in Patent Office: 1,021 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

A system and method for searching a finite collection of images using at least one semantic network. Upon receipt of a query from a user that includes a theme and one or more initial keywords, a set of keywords based on the theme and including the initial keywords is generated from one or more semantic networks corresponding to the theme and/or initial keywords. When the finite collection of images includes suitable metadata, a result set is generated of images corresponding to the expanded set of keywords. When the finite collection includes images lacking in metadata, a remote third-party image collection is searched with the set of keywords to obtain a result set that is used to train visual classifiers as to visual concepts associated with the keywords. The classifiers are used to classify the images in the finite collection lacking metadata and the search of the finite collection is performed with the set of keywords to generate a result set.

Claims

1. A method for searching a finite collection of images using at least one semantic network, comprising:

receiving a query from an associated user, the query comprising a theme;

identifying a set of keywords associated with the theme, each of the set of keywords being semantically linked to the theme via the at least one semantic network;

searching the finite collection of images for images having metadata relating to the identified set of keywords;

determining if the images relating to the identified set of keywords from the finite collection of images exceeds a predetermined number of images; and

determining, in response to a determination that the number of images relating to the identified set of keywords from the finite collection of images does not meet or exceed the predetermined number of images, that the finite collection of images includes additional images without metadata;

retrieving a set of images from a remote collection of images corresponding to the identified set of keywords responsive to the determination that the finite collection of images includes additional images without metadata;

training a plurality of visual classifiers to learn a set of visual concepts corresponding to the set of keywords from the retrieved set of images;

selecting a set of visual classifiers for application to each image in the finite collection of images without metadata;

classifying each image in the finite collection of images without metadata in accordance with the selected set of visual classifiers;

searching the finite collection of images with the identified set of keywords, the finite collection of images including a plurality of images having metadata associated therewith and at least one classified image; and

retrieving a result set of images corresponding to the theme from the finite collection of images responsive to a result of the searching thereof the result set including at least one classified image,

wherein at least one of the receiving, identifying, training, selecting, classifying, searching, and retrieving is performed by a computer processor.

2. The method according to claim 1, wherein the remote collection of images comprises at least one-third party image repository, and wherein each image in the remote collection of images includes metadata.

3. The method according to claim 2, wherein the query includes at least one initial keyword, and wherein set of keywords includes the at least one initial keyword.

4. The method according to claim 1, wherein selecting the set of visual classifiers further comprises:

testing each of the plurality of visual classifiers with respect to accuracy thereof, wherein selecting the set of visual classifiers includes selecting each visual classifier having a performance above a predetermined threshold, and wherein the predetermined threshold is a precision greater than 0.40.

5. The method according to claim 4, further comprising:

classifying each image in the finite collection of images in accordance with the selected set of visual classifiers; and

determining a classification score associated with each image in the finite collection of images,

wherein retrieving the set of result images comprises retrieving each image of the finite collection having a classification score above a predetermined value.

6. The method according to claim 5, wherein the at least one semantic network includes a plurality of nodes and at least one link between each of the plurality of nodes, each node corresponding to a keyword.

7. The method according to claim 6, wherein identifying the set of keywords further comprises:

determining a weight associated with each link between nodes associated with the theme; and

identifying the set of keywords corresponding to each node having an associated weight greater than a predetermined value.

8. The method according to claim 5, wherein each image in the finite collection of images includes a weight corresponding to a number of keywords associated therewith, and wherein retrieving the result set of images corresponding to the theme from the finite collection of images responsive to a result of the searching further comprises retrieving images having greater weights.

9. The method according to claim 8, wherein classifying each image in the finite collection of images further comprises adjusting the weight of an image in the finite collection of images responsive to a selection thereof by a user.

10. A system for searching a finite collection of images using at least one semantic network, comprising:

a keyword identifier component configured to identify a set of keywords semantically associated with a theme using the at least one semantic network;

a classification component configured to classify each image in the finite collection of images according to a visual concept;

a memory device which stores instructions for;

searching the finite collection of images for images having metadata relating to the identified set of keywords;

determining if the images relating to the identified set of keywords from the finite collection of images exceeds a predetermined number of images; and

determining, in response to a determination that the number of images relating to the identified set of keywords from the finite collection of images does not meet or exceed the predetermined number of images, that the finite collection of images includes additional images without metadata,

retrieving, from a remote collection of images, at least one image corresponding to each of the set of keywords responsive to the determination that the finite collection of images includes additional images without metadata,

training a plurality of classifiers for the classification component, each of the plurality of classifiers corresponding to a visual concept corresponding to at least one of the set of keywords associated with at least one of the images retrieved from the remote collection,

selecting a set of visual classifiers for application to each image without metadata in the finite collection of images,

classifying each image without metadata in the finite collection of images in accordance with a selected set of classifiers,

searching the finite collection of images in accordance with the identified set of keywords, the finite collection of images having metadata associated therewith and at least one classified image, and

retrieving a result set of images corresponding to the theme and the set of keywords from the finite collection of images responsive to a result of the searching thereof; and

a processor in communication with the memory device which executes the instructions.

11. The system according to claim 10, wherein the finite collection of images is stored in a local data storage device and the remote collection of images is stored in a remote third-party data storage device.

12. The system according to claim 10, wherein the memory device stores instructions for:

testing each of the plurality of classifiers with respect to accuracy thereof, wherein the set of classifiers includes each classifier testing above a predetermined threshold.

13. The system according to claim 12, wherein the classification component is further configured to determine a classification score associated with each image in the finite collection of images, and wherein retrieving the set of result images comprises retrieving each image of the finite collection having a classification score above a predetermined value.

14. The system according to claim 13, further comprising a weighting component configured to increase or decrease a weight of an association between a keyword and an image in the finite collection of images, wherein the weight is used by the classification component in determining the classification score associated with the image.

15. The system according to claim 14, wherein the classification component is further configured to generate metadata for at least one image in the finite collection of images in accordance with the classification thereof, wherein the classification includes a weighted association of at least one keyword with the at least one image.

16. A computer-implemented method for searching a finite collection of images using at least one semantic network, comprising:

receiving, by a computer system having a processor, a query from an associated user, the query including a theme and at least one initial keyword;

identifying a set of keywords associated with the received theme and at least one initial keyword using at least one semantic network associated with the received theme and the at least one initial keyword;

searching the finite collection of images for images having metadata relating to the identified set of keywords;

determining if the images relating to the identified set of keywords from the finite collection of images exceeds a predetermined number of images;

determining, in response to a determination that the number of images relating to the identified set of keywords from the finite collection of images does not meet or exceed the predetermined number of images, that the finite collection of images includes additional images without metadata;

retrieving, in response to a determination that the finite collection of images includes additional images without metadata, a set of images from a remote data storage having metadata including at least one of the set of semantically linked keywords;

training a plurality of visual classifiers to learn a set of visual concepts corresponding to the set of keywords from the retrieved set of images;

selecting a set of trained visual classifiers for application to each image in the finite collection of images without metadata;

classifying each of a plurality of images of the finite collection of images without metadata in accordance with the selected set of trained visual classifiers;

searching the finite collection of images with the identified set of keywords, the finite collection of images including a plurality of images having metadata associated therewith and at least one classified image; and

generating a result set of images from the finite collection of classified images in response to a search thereof with the set of keywords.