User intention modeling for interactive image retrieval

• US 8,190,604 B2
• Filed: 04/30/2008
• Issued: 05/29/2012
• Est. Priority Date: 04/03/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wen, Fang, Tang, Xiaoou
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): Davanlou, Soheila
• Application Number:
  US12/113,014
• Publication Number:
  US 20090254539A1
• Time in Patent Office: 1,490 Days
• Field of Search: 707/999.005, 707/723, 707/732, 707/E17.004, 707/915, 707/736
• US Class Current: 707/732
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content

Abstract

A system performs user intention modeling for interactive image retrieval. In one implementation, the system uses a three stage iterative technique to retrieve images from a database without using any image tags or text descriptors. First, the user submits a query image and the system models the user'"'"'s search intention and configures a customized search to retrieve relevant images. Then, the system extends a user interface for the user to designate visual features across the retrieved images. The designated visual features refine the intention model and reconfigure the search to retrieve images that match the remodeled intention. Third, the system extends another user interface through which the user can give natural feedback about the retrieved images. The three stages can be iterated to quickly assemble a set of images that accurately fulfills the user'"'"'s search intention. They system can be used for image searching without text tags, can be used for initial text tag generation, or can be used to complement a conventional tagged-image platform.

Claims

1. A method, comprising:

receiving an image from a user via a first user interface;

inferring a search intention of the user from visual contents of the image;

selecting a search method based on the inferred search intention;

deriving one or more image features from the image as search criteria;

training a support vector machine (SVM) ranking method with the search method and the one or more derived image features;

retrieving images from a database by applying the search method and the SVM ranking method;

receiving one or more image features from the user by operation of a second user interface, the one or more image features manually selected by one or more reference strokes drawn by the user that indicate the one or more image features; and

retrieving subsequent images from the database at least in part according to the one or more image features indicated by the user.

2. The method as recited in claim 1, further comprising modeling the search intention in a user intention model and selecting the search method based on the modeled search intention.

3. The method as recited in claim 2, wherein the modeling includes categorizing the inferred intention on an intentions list.

4. The method as recited in claim 2, further comprising:

refining the modeled search intention based on the image features selected via the second user interface; and

reconfiguring the search method and the search criteria, and retraining the SVM-based ranking method based on the refined modeled search intention and the image features selected at the second user interface;

wherein the retrieving of the subsequent images from the database is performed in part by applying the reconfigured search method and the retrained SVM ranking method.

5. The method as recited in claim 4, wherein the image features selected by the user'"'"'s reference strokes are selected from across multiple retrieved images.

6. The method as recited in claim 4, further comprising:

extending a third user interface to input user feedback with respect to the subsequently retrieved images, wherein the user designates images relevant to the user'"'"'s search intention;

refining the modeled search intention based on the user feedback input at the third user interface;

reconfiguring the search method and the search criteria, and retraining the SVM-based ranking method based on the refined modeled search intention and the user feedback input at the third user interface; and

retrieving images from the database by applying the reconfigured search method and the retrained SVM ranking method.

7. The method as recited in claim 6, further comprising:

comparing the user feedback with learned patterns or historical log data to find a similar pattern of user feedback;

selecting a search criterion or a search method associated with the similar pattern.

8. The method as recited in claim 6, further comprising iteratively:

receiving user input via one of the first, second, and third user interfaces;

remodeling the user'"'"'s search intention based on the user input;

reconfiguring the search method and the search criteria, and retraining the SVM-based ranking method based on the remodeled search intention;

retrieving subsequent images by applying the reconfigured search method and search criteria and the retrained SVM-based ranking method; and

repeating user input via one of the user interfaces based on the subsequently retrieved images.

9. The method as recited in claim 1, further comprising combining derived image features into an image search criterion.

10. The method as recited in claim 1, further comprising weighting each derived image feature with a relative importance.

11. A method, comprising:

crawling images in local files;

remotely crawling images on a network;

indexing the images by visual features parsed from the images, wherein the visual features are indicative of search intentions of a user modeled in an intention model;

creating an images database structured by the visual features;

receiving a query image from the user by operation of a first user interface;

inferring a search intention of the user by extending a second user interface to receive one or more visual features manually selected by reference strokes drawn by the user on the query image; and

retrieving images from the images database according to input obtained at least in part from the second user interface.

12. The method as recited in claim 11, wherein the parsed visual features include one of face detection data, face alignment data, facial Local Binary Pattern (LBP) features data, scene features data, texture features data, color features data, image attention model data, color signature data, or substantially similar visual features.

13. The method as recited in claim 11, further comprising:

modeling the user search intention against a list of user search intentions;

matching a search method to the modeled user search intention;

configuring a search criterion from the visual feature;

searching the image database using the search method and the search criterion;

ranking retrieved images returned by the searching; and

displaying highest ranked retrieved images.

14. The method as recited in claim 13, further comprising:

extending a user interface for the user to designate image features across the displayed images;

remodeling the user search intention based on the designated image features; and

searching and displaying a subsequent set of images from the database based on the remodeled user search intention.

15. The method as recited in claim 14, further comprising:

extending another user interface for collecting user feedback, wherein the user selects images relevant to the user'"'"'s search intention from the displayed images;

remodeling the user search intention based on the user feedback; and

searching and displaying a subsequent set of images from the database based on the remodeled user search intention.

16. The method as recited in claim 15, further comprising automatically submitting an image from the subsequent set of images as a new query image.

17. The method as recited in claim 15, further comprising iteratively refining the user intention model with each user input applied to each current set of retrieved images, in order to retrieve respective subsequent sets of retrieved images.

18. A system, comprising:

memory;

one or more processors operatively coupled to the memory;

an intention-modeling engine stored in the memory and executable by the one or more processors to receive a query image from a user and iteratively refine a search intention of the user based on visual features of the query image, the visual features indicated at least in part by extending a user interface allowing the user to draw strokes on the query image to designate features and regions of interest by surrounding them with the strokes; and

a retrieval engine stored in the memory and executable by the one or more processors to configure an image search based on a current state of the modeled search intention.

19. The system as recited in claim 18, wherein the intention-modeling engine includes iterative stages for soliciting user input for refining the search intention and wherein the system displays a current set of retrieved images corresponding to the currently modeled search intention.

20. The system as recited in claim 18, further comprising an image database generator, including:

a crawler to find images on local files and on the Internet;

an indexer to build a structured image database that is based on the found images and on visual features of the found images; and

a features extractor to obtain the visual features of the images, including one of face detection data, face alignment data, facial Local Binary Pattern (LBP) features data, scene features data, texture features data, color features data, image attention model data, color signature data, or substantially similar visual features.