CONCEPT-STRUCTURED IMAGE SEARCH

• US 20110072048A1
• Filed: 09/23/2009
• Published: 03/24/2011
• Est. Priority Date: 09/23/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Xu, Hao, Wang, Jingdong, Hua, Xian-Sheng
• Granted Patent: US 8,392,430 B2
• Time in Patent Office: Days
• US Class Current: 707/780
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

The concept-structured image search technique described herein pertains to a technique for enabling a user to indicate their semantic intention and then retrieve and rank images from a database or other image set according to this intention. The concept-structured image search technique described herein includes a new interface for image search. With this interface, a user can freely type several key textual words in arbitrary positions on a blank image, and also describe a region for each keyword that indicates its influence scope, which is called concept structure herein. The concept-structured image search technique will return and rank images that are in accordance with the concept structure indicated by the user. One embodiment of the technique can be used to create a synthesized image without actually using the synthesized image to perform a search of an image set.

Claims

1. A computer-implemented process for creating a synthesized image by employing a concept-structured image search query, comprising:

using a computing device for,

(a) specifying a concept-structured query that comprises keywords and location of a concept for each keyword in an image;

(b) performing a text-based image search of an image set using the specified keywords;

(c) selecting groups of representative images from the image set for each of the specified keywords;

(d) performing concept-specific image center selection for each of the selected groups of representative images for each keyword to find the semantic centers of each of the groups of the representative images; and

(e) synthesizing an image that is in accordance with the specified location of the concept of each keyword by placing a representative image that best represents the selected group at the specified location of the concept of each keyword in the synthesized image.

2. The computer implemented process of claim 1 further comprising using the synthesized image to rank images returned in an image search, comprising:

(f) performing feature extraction for the synthesized image;

(g) performing image ranking of images returned in the image search by comparing the features of the synthesized image to the features of each of the images returned in the image search; and

(h) outputting a ranked list of the images of the image set that are ranked in accordance with how similar the images returned in the image search are to the synthesized image.

3. The computer-implemented process of claim 2 further comprising generating and ranking additional ranked lists of images by repeating (e) through (h).

4. The computer-implemented process of claim 1 further comprising allowing a user to manipulate the synthesized image to make the synthesized image closer to a user'"'"'s intention.

5. The computer-implemented process of claim 4 wherein the user input comprises selecting and replacing a portion of the synthesized image with a new representative image.

6. The computer-implemented process of claim 4 wherein the user input comprises adjusting boundaries of representative images used to generate the synthesized image.

7. The computer-implemented process of claim 1 wherein the location of a concept for each keyword in an image is defined by an ellipse having a center and a size.

8. The computer-implemented process of claim 7 wherein a user can manipulate the center and size of the ellipse in order to change a scope of influence of a keyword in an image.

9. A computer-implemented process for performing a concept-structured image search, comprising:

using a computing device for,

searching an image database by submitting a concept-structured query; and

returning a ranked list of one or more images of the image database based on the similarity of the images in the image database with the concept-structured search query.

10. The computer-implemented process of claim 9 wherein the concept-structured query further comprises keywords and location of a concept for each keyword in an image.

11. The computer-implemented process of claim 10 wherein returning a ranked list of one or more images of the database further comprises:

performing a text-based image search of the image database using the keywords;

selecting groups of representative images for each of the specified keywords;

synthesizing an image that is in accordance with the specified location of the concept of each keyword by placing a representative image at the specified location of the concept of each keyword;

performing feature extraction for the synthesized image;

performing image ranking of images in the database by comparing the features of the synthesized image to the features of each of the images in the image database; and

outputting a ranked list of the images of the image database that are ranked in accordance with how similar they are to the synthesized image.

12. The computer-implemented process of claim 11 wherein the ranked list of the images is created by assigning each of the database images with a similarity score that is calculated as the weighted Euclidean Distance between the features of a database image and the features of the synthesized image.

13. The computer-implemented process of claim 11 further comprising finding the center of a group of representative images prior to stitching the representative image into the synthesized image, comprising:

segmenting each of the representative images of the group for each of the specified keywords into sub-regions;

performing clustering for the sub-regions of the group;

selecting a specified number of the largest clusters;

for each representative image of the group, forming a binary map so that the binary value is set to 1 if the corresponding sub-region belongs to the top specified number of clusters and 0 otherwise;

finding the largest connected region of each image considering only regions in the binary map whose value is 1.

14. The computer-implemented process of claim 10 further comprising a user changing the synthesized image to better represent the user'"'"'s concept desired in the synthesized image by replacing a representative image in the synthesized image with another representative image.

15. In a computer system comprising a display and a user interface data entry device, a process for finding an image in a set of images, comprising:

displaying on the display a window for entering a concept-structured query comprising a concept-structured layout of an image to be found in the set of images;

a user inputting a concept-structured query into the window for entering the concept-structured layout using the user interface data entry device; and

outputting on the display a ranked list of images of the set of images based on similarity of the images with the concept-structured layout.

16. The system of claim 15 wherein a user inputs the concept-structured query by typing keywords in the window for entering the concept-structured layout at a location where the concept of the keyword is sought.

17. The system of claim 16 further comprising associating ellipses with each of the keywords to specify the scope and location of the concept associated with the keyword in an image.

18. The system of claim 17 wherein the user can edit the direction and size of the ellipses to alter the scope and location of the concept associated with the keyword in an image.

19. The system of claim 15, further comprising:

synthesizing an image from representative images that represent the keywords entered; and

using the synthesized image to find an image in the set of images that is similar to the concept-structured layout.

20. The system of claim 19, further comprising using the synthesized image to find an image similar to the concept-structured layout in a new set of images.