Interactively ranking image search results using color layout relevance

• US 8,406,573 B2
• Filed: 12/22/2008
• Issued: 03/26/2013
• Est. Priority Date: 12/22/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Jingdong, Li, Shipeng, Hua, Xian-Sheng, Zhao, Yinghai
• Primary Examiner(s): Johns, Andrew W
• Assistant Examiner(s): Harandi, Siamak
• Application Number:
  US12/341,953
• Publication Number:
  US 20100158412A1
• Time in Patent Office: 1,555 Days
• Field of Search: None
• US Class Current: 382/305
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

This disclosure describes various exemplary user interfaces, methods, and computer program products for the interactively ranking image search results refinement method using a color layout. The method includes receiving a text query for an image search, presenting image search results in a structured presentation based on the text query and information from an interest color layout. The process creates image search results that may be selected by the user based on color selection palettes or color layout specification schemes. Then the process ranks the image search results by sorting the results according to similarity scores between color layouts from the image search results and the interest color layout from a user based on the color selection palettes and the color layout specification schemes.

Claims

1. A method for interactively ranking image search results by using a color layout relevance, implemented at least in part by a computing device, the method comprising:

extracting color layouts for images after the images are crawled from the internet;

receiving a text query for an image search;

evaluating a similarity between an interest color layout and a color layout of any image in an image search results, wherein the similarity is based on a color appearance compatibility, a spatial distribution consistency, and a contextual layout similarity, wherein the color appearance compatibility, the spatial distribution consistency, and the contextual layout similarity are based on a color similarity and a layout component-wise similarity; and

reordering the image search results based on evaluated similarities between color layouts of images and the interest color layout from the user according to a selection of a color selection palette and a color layout specification scheme.

2. The method of claim 1, wherein the color selection palettes comprises a standard color palette, a query dependent color palette, and a color palette using colors from image search results.

3. The method of claim 1, wherein the color layout specification scheme comprises at least one of:

a free-scribbling to allow a user to freely draw strokes to indicate an interest layout;

a template layout to allow the user to select an interest layout from a set of templates, which are common for text queries; and

query-dependent layouts to allow the user to select an interest layout from a set of query-dependent templates, which are mined from image search results corresponding to the text query; and

a layout from a sample image to allow the user to select a sample image from the search results and to draw on the sample image to indicate the interest color layout.

4. The method of claim 1, wherein the color appearance compatibility comprises a degree that the colors in the image corresponding to the interest layout region are similar to the colors in the interest color layout.

5. The method of claim 1, wherein the spatial distribution consistency comprises that the color appearing in the region corresponding to the interest color region should be less similar than the other interest colors.

6. The method of claim 1, wherein the contextual layout similarity comprises assigning different weights to a propagated color, the weights are set according to a distance of the grid containing the propagated color from seeded grids.

7. A computing device comprising a processor;

a memory storing computer-readable instructions executed on the processor, the computer-readable instructions comprising instructions for;

receiving a text query for an image search;

extracting a color layout;

measuring a similarity based on a color appearance compatibility, a spatial distribution consistency, a contextual layout similarity, a color similarity, and a layout component-wise similarity;

providing interactive options for the image search results with color selection palettes and color layout specification schemes;

reordering the image search results using a color layout ranking; and

presenting ranked image search results in a structured presentation based on the text query and information from the interest color layout.

8. The computing device of claim 7, wherein the color appearance compatibility comprises a degree that the colors in the image corresponding to the interest layout region are similar to the colors in the interest color layout.

9. The computing device of claim 7, wherein the spatial distribution consistency comprises that the color appearing in the region corresponding to the interest color region should be less similar than the other interest colors.

10. The computing device of claim 7, wherein the contextual layout similarity comprises assigning different weights to a propagated color, the weights are set according to a distance of the grid containing the propagated color from seeded grids.

11. The computing device of claim 7, wherein the color selection palettes to indicate interest colors comprises:

a standard color palette, wherein colors are organized without using information from the image search results;

a query-dependent color palette, wherein colors that appear most-frequently in the image search results; and

a color palette generated from selecting images from the image search results.

12. The computing device of claim 7, wherein the color specification scheme comprises:

a free-scribbling to allow a user to freely draw strokes to indicate an interest layout;

a template layout to allow the user to select an interest layout from a set of templates, which are common for text queries;

query-dependent layouts to allow the user to select an interest layout from a set of query-dependent templates, which are mined from image search results corresponding to the text query; and

a layout from an example image to allow the user to select a sample image from the search results and to draw on the example image to indicate the interest color layout.

13. The computing device of claim 7, further comprising a checkbox feature for turning on a color layout ranking.