Concept-structured image search

• US 8,706,740 B2
• Filed: 02/06/2013
• Issued: 04/22/2014
• Est. Priority Date: 09/23/2009
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hua, Xian-Sheng, Wang, Jingdong, Xu, Hao
• Primary Examiner(s): Vital, Pierre
• Assistant Examiner(s): LODHI, ANDALIB FT
• Application Number:
  US13/760,680
• Publication Number:
  US 20130151510A1
• Time in Patent Office: 440 Days
• Field of Search: 707/748, 707/754, 707/769
• US Class Current: 707/748
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

The concept-structured image search technique described herein pertains to a technique for enabling a user to indicate their semantic intention and then retrieve and rank images from a database or other image set according to this intention. The concept-structured image search technique described herein includes a new interface for image search. With this interface, a user can freely type several key textual words in arbitrary positions on a blank image, and also describe a region for each keyword that indicates its influence scope, which is called concept structure herein. The concept-structured image search technique will return and rank images that are in accordance with the concept structure indicated by the user. One embodiment of the technique can be used to create a synthesized image without actually using the synthesized image to perform a search of an image set.

Claims

1. In a computer system comprising a display and a user interface data entry device, a process for finding an image in a set of images, comprising:

displaying on the display a window for entering a concept-structured query comprising a concept-structured layout of an image to be found in the set of images;

receiving a concept-structured query via the window for entering the concept-structured layout by placing at least one keyword representative of an image concept in at least one location in the window using the user interface data entry device;

creating a synthesized image by placing, for each keyword having a specified location in the window, an image representative of the concept of the keyword at the specified location of each keyword in the window;

searching the images of the image set using the synthesized image; and

outputting on the display a ranked list of images of the set of images based on similarity of the images with the synthesized image concept structured layout.

2. The process system-of claim 1 wherein a user inputs the concept structured query by typing at least one keyword in the window for entering the concept-structured layout at a location where an image representative of the concept of the keyword is sought.

3. The process of claim 2 further comprising associating ellipses with each of the keywords to specify the scope and location of the concept associated with the keyword in an image.

4. The process of claim 3 wherein the user can edit the direction and size of the ellipses to alter the scope and location of the concept associated with the keyword in an image.

5. The process of claim 1, further comprising using the synthesized image to find an image similar to the concept-structured layout in a new set of images.

6. A method for finding an image in a set of images, comprising:

displaying on a display a window for entering a concept-structured query comprising a concept structured layout of an image to be found in the set of images;

receiving a concept-structured query via the window for entering the concept-structured layout by placing an image concept in at least one location in the window using a user interface data entry device;

finding images in the images set representative of each of the concepts in the user-entered concept-structured layout;

creating a synthesized image by placing, for each image concept having a specified location in the window, an image representative of the image concept at the specified location in the window;

searching the images of the image set using the synthesized image; and

outputting on the display a ranked list of images of the set of images based on similarity of the images with the synthesized image obtained from concept-structured layout.

7. The method of claim 6 wherein a user inputs the concept-structured query by typing one or more keywords in the window for entering the concept-structured layout at a location where the image concept of the keyword is sought.

8. The method of claim 7 further comprising associating ellipses with each of the keywords to specify the scope and location of the concept associated with the keyword in an image.

9. The method of claim 8 wherein the user can edit the direction and size of the ellipses to alter the scope and location of the concept associated with the keyword in an image.

10. The method of claim 6, further comprising using the synthesized image to find an image similar to the concept-structured layout in a new set of images.

11. The method of claim 6 further comprising the user manually

selecting an image to replace a representative image in the synthesized image.

12. The method of claim 6 further comprising a user moving a representative image in the synthesized image to a new location in the window by dragging the representative image with the input device.

13. A method for synthesizing a new image, comprising:

displaying on a display a window for entering a concept-structured query in the form of a concept-structured layout of an image to be synthesized from a set of images;

receiving a concept-structured layout via a window by placing one or more image concepts in locations on the window using a user interface data entry device;

using the input concept-structured layout to find a group of representative images in the image set for each concept in each location in the window;

determining the best representative image from each group of representative images;

finding the image center for each best representative image;

aligning each located image center with the location of a corresponding concept in the window; and

seamlessly stitching the aligned best representative images to output a synthesized image on the display.

14. The method of claim 13 wherein a user inputs the concept structured query by typing keywords in the window for entering the concept-structured layout at each location of each placed image concept.

15. The method of claim 14 further comprising associating ellipses with each of the keywords to specify the scope and location of the concept associated with the keyword in an image, and wherein the user can edit the direction and size of the ellipses to alter the scope and location of the concept associated with the keyword in an image.

16. The process method of claim 13 wherein the output of the synthesized image on the display displays sub-regions of the best representative images separated by a contour and associated input keywords.

17. The method of claim 13 further comprising the user manually selecting an image to replace a representative image in the synthesized image.

18. The method of claim 13, further comprising:

using the synthesized image to find an image in a set of images that is similar to the concept-structured layout.