Concept-structured image search

• US 8,392,430 B2
• Filed: 09/23/2009
• Issued: 03/05/2013
• Est. Priority Date: 09/23/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hua, Xian-Sheng, Wang, Jingdong, Xu, Hao
• Primary Examiner(s): Breene, John E
• Assistant Examiner(s): LODHI, ANDALIB FT
• Application Number:
  US12/565,313
• Publication Number:
  US 20110072048A1
• Time in Patent Office: 1,259 Days
• Field of Search: 707/748, 707/754, 707/769
• US Class Current: 707/748
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

The concept-structured image search technique described herein pertains to a technique for enabling a user to indicate their semantic intention and then retrieve and rank images from a database or other image set according to this intention. The concept-structured image search technique described herein includes a new interface for image search. With this interface, a user can freely type several key textual words in arbitrary positions on a blank image, and also describe a region for each keyword that indicates its influence scope, which is called concept structure herein. The concept-structured image search technique will return and rank images that are in accordance with the concept structure indicated by the user. One embodiment of the technique can be used to create a synthesized image without actually using the synthesized image to perform a search of an image set.

Claims

1. A computer-implemented process for creating a synthesized image by employing a concept-structured image search query, comprising:

using a computing device for,

(a) specifying a concept-structured query that comprises keywords and location of a concept for each keyword in an image;

(b) performing a text-based image search of an image set using the specified keywords;

(c) selecting groups of representative images from the image set for each of the specified keywords;

(d) performing concept-specific image center selection for each of the selected groups of representative images for each keyword to find the semantic centers of each of the groups of the representative images; and

(e) synthesizing an image that is in accordance with the specified location of the concept of each keyword by placing a representative image that best represents the selected group at the specified location of the concept of each keyword in the synthesized image, using the synthesized image to rank images returned in an image search,

(f) performing feature extraction for the synthesized image;

(g) performing image ranking of images returned in the image search by comparing the features of the synthesized image to the features of each of the images returned in the image search; and

(h) outputting a ranked list of the images of the image set that are ranked in accordance with how similar the images returned in the image search are to the synthesized image.

2. The computer-implemented process of claim 1 further comprising generating and ranking additional ranked lists of images by repeating (e) through (h).

3. The computer-implemented process of claim 1 further comprising allowing a user to manipulate the synthesized image to make the synthesized image closer to a user'"'"'s intention.

4. The computer-implemented process of claim 3 wherein the user input comprises selecting and replacing a portion of the synthesized image with a new representative image.

5. The computer-implemented process of claim 3 wherein the user input comprises adjusting boundaries of representative images used to generate the synthesized image.

6. The computer-implemented process of claim 1 wherein the location of a concept for each keyword in an image is defined by an ellipse having a center and a size.

7. The computer-implemented process of claim 6 wherein a user can manipulate the center and size of the ellipse in order to change a scope of influence of a keyword in an image.

8. A computer-implemented process for performing a concept-structured image search, comprising:

using a computing device for,

searching an image database by submitting a concept-structured query; and

returning a ranked list of one or more images of the image database based on the similarity of the images in the image database with the concept-structured search query, comprising;

performing a text-based image search of the image database using the keywords;

selecting groups of representative images for each of the specified keywords;

synthesizing an image that is in accordance with the specified location of the concept of each keyword by placing a representative image a representative image for each of the groups of representative images at the specified location of the concept of each keyword;

performing feature extraction for the synthesized image;

performing image ranking of images in the database by comparing the features of the synthesized image to the features of each of the images in the image database; and

outputting a ranked list of the images of the image database that are ranked in accordance with how similar they are to the synthesized image.

9. The computer-implemented process of claim 8 wherein the concept-structured query further comprises keywords and location of a concept for each keyword in an image.

10. The computer-implemented process of claim 9 further comprising a user changing the synthesized image to better represent the user'"'"'s concept desired in the synthesized image by replacing a representative image in the synthesized image with another representative image.

11. The computer-implemented process of claim 8 wherein the ranked list of the images is created by assigning each of the database images with a similarity score that is calculated as the weighted Euclidean Distance between the features of a database image and the features of the synthesized image.

12. The computer-implemented process of claim 8 further comprising finding the center of a group of representative images prior to stitching the representative image into the synthesized image, comprising:

segmenting each of the representative images of the group for each of the specified keywords into sub-regions;

performing clustering for the sub-regions of the group;

selecting a specified number of the largest clusters;

for each representative image of the group, forming a binary map so that the binary value is set to 1 if the corresponding sub-region belongs to the top specified number of clusters and 0 otherwise;

finding the largest connected region of each image considering only regions in the binary map whose value is 1.