Object retrieval using visual query context

• US 8,560,517 B2
• Filed: 07/05/2011
• Issued: 10/15/2013
• Est. Priority Date: 07/05/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Yang, Linjun, Geng, Bo, Hua, Xian-Sheng, Cai, Yang
• Primary Examiner(s): LE, DEBBIE M
• Application Number:
  US13/176,279
• Publication Number:
  US 20130013578A1
• Time in Patent Office: 833 Days
• Field of Search: 707/748, 707/708, 707/765, 382/170, 382/305
• US Class Current: 707/708
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour

Abstract

Some implementations provide techniques and arrangements to perform image retrieval. For example, some implementations identify an object of interest and a visual context in a first image. In some implementations, a second image that includes a second object of interest and a second visual context may be compared to the object of interest and the visual content, respectively, to determine whether the second image matches the first image.

Claims

1. A method comprising:

under control of one or more processors specifically configured with executable instructions,

receiving, at a search engine, a query associated with a query image, the query image including a plurality of pixels;

categorizing a first portion of the plurality of pixels as foreground pixels;

categorizing a second portion of the plurality of pixels as background pixels;

determining saliency scores for each of the plurality of pixels based on the foreground pixels and the background pixels, the saliency scores comprising contrast-based saliency scores;

normalizing the saliency scores to create normalized saliency scores;

transforming the normalized saliency scores to create a prior probability distribution;

determining a search intent based on the prior probability distribution using a dual-sigmoid approximation; and

identifying, from a set of images, one or more images that match the query image based on the search intent.

2. The method of claim 1, wherein the query includes a region of interest that includes at least a portion of the image.

3. The method of claim 2, wherein the region of interest is specified by a bounding box having a geometric shape.

4. The method of claim 1, wherein identifying, from the set of images, the one or more images that match the query image comprises:

determining query visual words to represent the query image;

comparing the query visual words with visual words associated with each image of the set of images; and

determining that a particular image of the set of images matches the query image when a number of query visual words that match particular visual words associated with the particular image satisfies a threshold.

5. The method of claim 1, further comprising providing the one or more images that match the query image to a client device that originated the query.

6. The method of claim 1, wherein the dual-sigmoid approximation includes a first sigmoid approximation that is used for a first dimension of the query image and a second sigmoid approximation that is used for a second dimension of the query image.

7. Computer-readable storage media including instructions executable by one or more processors to perform operations comprising:

receiving data identifying a region of interest associated with a first image, the first image including a plurality of pixels;

determining contrast-based saliency scores for each of the plurality of pixels;

normalizing the contrast-based saliency scores to create normalized saliency scores;

determining transforming the normalized saliency scores to create a prior probability distribution;

determining a search intent based on a dual-sigmoid approximation in which a first sigmoid approximation is used for a first dimension of the region of interest and a second sigmoid approximation is used for a second dimension of the region of interest; and

performing a first comparison between the first image and a second image based on the search intent.

8. The computer-readable storage media of claim 7, the operations further comprising providing an indication that the second image matches the first image based on the first comparison.

9. The computer-readable storage media of claim 7, the operations further comprising:

performing a second comparison between the first image and a third image based on the region of interest.

10. The computer-readable storage media as recited in claim 9, the operations further comprising providing a second indication that the third image matches the first image based on the second comparison.

11. The computer-readable storage media of claim 7, wherein performing the first comparison between the object of interest in the first image and the second object of interest in the second image comprises:

determining first visual words associated with the object of interest in the first image;

determining second visual words associated with the second object of interest in the second image; and

comparing the first visual words with the second visual words.

12. The computer-readable storage media of claim 7, wherein performing the second comparison between the visual context in the first image and the second visual context in the second image comprises:

determining third visual words associated with the visual context in the first image;

determining fourth visual words associated with the second visual context in the second image; and

comparing the third visual words with the fourth visual words.

20. The computer-readable storage media of claim 7, wherein:

the region of interest includes an object of interest; and

the data is included in a query to identify other images that include the object of interest.

13. A computing device comprising:

one or more processors;

computer-readable storage media accessible to the one or more processors;

a communication interface to receive a query associated with a first image, the query including data identifying a region of interest that includes a portion of the first image;

a saliency detection module to determine contrast-based saliency scores for the first image, normalize the saliency scores to create normalized saliency scores, and determine a prior probability distribution based on the normalized saliency scores;

a search intent detection module to determine a search intent associated with the query based on the prior probability distribution, the search intent determined via a dual-sigmoid approximation, the dual sigmoid function including a first sigmoid function associated with a first dimension of the region of interest and a second sigmoid function associated with a second dimension of the region of interest;

a visual words identification module to select visual words based on the search intent; and

a context-based object retrieval module to perform a search of a plurality of images based on the visual words to determine whether at least one image, of the plurality of images matches the first image.

14. The computing device of claim 13, wherein the region of interest is specified by a geometric shape.

15. The computing device of claim 14, wherein the geometric shape is one of a square, trapezoid, a rhombus, and a circle.

16. The computing device of claim 13, wherein the region of interest is specified by a free-form shape drawn by a user.

17. The computing device of claim 13, wherein the context-based search module is operable to compare the visual words with second visual words associated with a particular image of the plurality of images.

18. The computing device of claim 17, wherein the context-based search module is operable to determine that the particular image matches the first image in response to determining that a threshold number of the visual words match the second visual words.

19. The computing device of claim 13, wherein:

the region of interest includes an object of interest; and

the query includes a request to identify additional images that include the object of interest.