IMAGE OBJECT RETRIEVAL BASED ON AGGREGATION OF VISUAL ANNOTATIONS

• US 20120158716A1
• Filed: 12/16/2010
• Published: 06/21/2012
• Est. Priority Date: 12/16/2010
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Zwol, Roelof Van, Olivares, Ximena
• Granted Patent: US 8,527,564 B2
• Time in Patent Office: Days
• US Class Current: 707/728
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content

Abstract

An approach for responding to a text-based query for a digital image is provided. A request that identifies one or more keywords is received. A number of annotated digital images are selected based on a previously determined optimum quantity of annotated digital images. Composite data is gathered from each annotated digital image and a set of candidate digital images is selected based on the composite data. The set of candidate images are the digital images, of a set of digital images, which have a visual appearance that is most similar to the composite data. A response is generated that identifies those digital images which are most responsive to the one or more keywords. Alternatively, a partitioned response is generated which identifies dissimilar sets of digital images.

Claims

1. A computer-implemented method comprising steps of:

receiving a query, wherein the query identifies a set of one or more query terms;

selecting, from an image corpus, a set of three or more sample images that are associated with annotations that match one or more of the query terms;

for each particular sample image in the set of sample images, selecting, from the image corpus, a separate set of candidate images that are visually similar to that particular sample image, and associating that set of candidate images with the particular sample image;

determining that a first set of candidate images, associated with a first sample image of the set of sample images, and a second set of candidate images, associated with a second sample image of the set of sample images, both contain at least a threshold number of candidate images in common;

determining that a third set of candidate images, associated with a third sample image of the set of sample images, contains less than the threshold number of candidate images in common with the first and second sets of candidate images; and

in response to determining that the third set of candidate images contains less than the threshold number of candidate images in common with the first and second sets of candidate images, generating an aggregated set of images based on the first and second sets of candidate images;

wherein the steps are performed by at least one computing device.

2. The method of claim 1, further comprising generating a response to the query, wherein the response includes at least a subset of the images in the aggregated set of images.

3. The method of claim 1, further comprising associating a rank with each image contained in the first, second, and third sets of candidate images, wherein the determination that the third set of candidate images contain less than the threshold number of candidate images in common with the first and second sets of candidate images is based, at least in part, on the rank.

4. The method of claim 1, wherein the aggregated set of images is generated based on the first, second, and third sets of candidate images, and wherein the method further comprises:

generating a partitioned response to the query, wherein the partitioned response identifies the third set of candidate images as distinguishable from the first and second sets of candidate images.

5. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 1.

6. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 2.

7. A computer-implemented method comprising steps of:

receiving a set of one or more query terms;

selecting, from an image corpus, a set of sample images that are associated with annotations that match one or more of the query terms;

based on the sample images in the set of sample images, generating composite data that represents at least one visual characteristic that the sample images in the set of sample images have in common;

determining a set of candidate images that possess visual characteristics that are similar to the at least one visual characteristics represented by the composite data; and

returning at least a subset of candidate images in the set of candidate images as results of a query that contained the set of query terms;

wherein the steps are performed by at least one computing device.

8. The method of claim 7, wherein the results are generated without determining whether any image in the image corpus is visually similar to any one of the sample images in the set of sample images.

9. The method of claim 7, wherein the number of sample images in the set of sample images to be selected is determined prior to receiving the set of one or more query terms.

10. The method of claim 7, wherein generating composite data further comprises associating the at least one visual characteristic with a value, wherein the value represents a number of occurrences of the at least one visual characteristic within the set of sample images.

11. The method of claim 10, wherein the method further comprises storing the composite data in association with the set of one or more query terms.

12. The method of claim 10, further comprising:

filtering the composite data to remove at least one visual characteristic associated with a value less which is less than a threshold value.

13. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 7.

14. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 9.

15. A computer-implemented method comprising steps of:

for each particular sample image of a first set of sample images, selecting, from an image corpus, a separate set of candidate images that are visually similar to that particular sample image of the first set of sample images, thereby producing a first set of sets of candidate images;

aggregating the sets of candidate images in the first set of sets of candidate images to produce a first aggregated list of images;

determining a first precision score that is based on a measure of relevance of images in the first aggregated list of images to a specified set of query terms;

for each particular sample image of a second set of sample images, selecting, from the image corpus, a separate set of candidate images that are visually similar to that particular sample image of the second set of sample images, thereby producing a second set of sets of candidate images;

aggregating the sets of candidate images in the second set of sets of candidate images to produce a second aggregated list of images;

determining a second precision score that is based on a measure of relevance of images in the second aggregated list of images to the specified set of query terms;

wherein a quantity of sample images in the second set of sample images is greater than a quantity of sample images in the first set of sample images; and

based at least in part on the first precision score and the second precision score, determining a quantity of sample images to be used in performing future searches for images in the image corpus;

wherein the steps are performed by at least one computing device.

16. The method of claim 15, wherein the measure of relevance is based, at least in part, on dividing a quantity of relevant images in a set of images by a quantity of at least a subset of images in a set of images.

17. The method of claim 15, wherein the method further comprises:

prior to determining the first and second precision scores, ranking the first and second aggregated list of images to produce a first and second ranked list of images, wherein each measure of relevance for the first and second precision scores is based on each respective first and second ranked list of images.

18. The method of claim 15, wherein the quantity of sample images to be used in performing future searches is selected based, at least in part, on determining whether a difference value between the first precision score and the second precision score is below a threshold difference value.

19. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 15.

20. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 16.