System and method for managing search results including graphics

• US 9,092,458 B1
• Filed: 09/13/2012
• Issued: 07/28/2015
• Est. Priority Date: 03/08/2005
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Perona, Pietro, Goncalves, Luis, Di Bernardo, Enrico
• Primary Examiner(s): Burke, Jeffrey A
• Application Number:
  US13/615,381
• Time in Patent Office: 1,048 Days
• Field of Search: 707/726, 707/741, 707/748
• US Class Current: 1/1
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

A system and method of sorting graphics files with a graphic server and image search engine is disclosed. The method preferably comprises the steps of: receiving a plurality of search results where each of the search results comprising one or more associated graphics; using a general purpose computer to identifying one or more groups of said graphics that depict or otherwise possess similar visual features; and returning the plurality of search results to a user in accordance with said identified groups. The preferred embodiment effectively clusters graphics files, particularly image files, based on upon shared scale-invariant features in order to enable the user to readily identify collections of images relevant to the user while skip past collections of less relevant images.

Claims

1. A computer implemented method comprising:

receiving, at a computer system comprising hardware, a user search query consisting only of text;

receiving, at the computer system, ranked image search results from a first image search corresponding to the user search query, the user search query consisting only of text, the ranked search results comprising a first plurality of images;

classifying, by the computer system, a first subset of the first plurality of images as a first group based at least in part on a first subject matter;

classifying, by the computer system, a second subset of the first plurality of images as a second group based at least in part on a second subject matter;

providing, by the computer system, for display on a user computing device a first visual representation corresponding to the first subset of images, the first visual representation comprising a first thumbnail image representative of the first subject matter, and a second visual representation corresponding to the second subset of images, the second visual representation comprising a second thumbnail image representative of the second subject matter;

receiving, by the computer system, a selection of the first thumbnail image, the second thumbnail image being unselected;

automatically identifying, by the computer system, a first keyword based at least in part on the user selection of the first thumbnail image, the first keyword not included in the user search query, wherein the identification of the first keyword is based at least in part on identifying words used in webpages associated with the first subset of images represented by the selected first thumbnail image, wherein the identified words appear in greater frequency in webpages associated with the first subset of images represented by the selected first thumbnail image than a frequency of appearance of the identified words in webpages associated with the second subset of images represented by the unselected second thumbnail image;

executing, by the computer system, a first refined image search using a first search query comprising the automatically identified first keyword, wherein the first search query comprising the automatically identified first keyword is different than the user search query, wherein the first refined image search is performed using a first textual search query comprising the automatically identified first keyword;

receiving, by the computer system, second ranked image search results from the first refined image search, the second ranked search results comprising a second plurality of images;

generating, by the computer system, a similarity metric for images included in the second ranked image search results based at least in part on how many visual features are common to two or more images in the second ranked image search results and a quality with which the visual features give rise to patterns that can be matched or aligned;

classifying, by the computer system, a subset of the second plurality of images as a third group using the similarity metric for images included in the second plurality of images;

providing for display, by the computer system, on the user computing device;

a visual representation corresponding to the third group of images, the visual representation corresponding to the third group of images comprising a third thumbnail image, and

a visual representation corresponding to a fourth group of images, the visual representation corresponding to the fourth group of images comprising a fourth thumbnail image;

receiving, by the computer system, a selection of the third thumbnail image, the fourth thumbnail image being unselected;

automatically identifying, by the computer system, a second keyword based at least in part on the user selection of the third thumbnail image, the second keyword not included in the user search query, wherein the identification of the second keyword is based at least in part on identifying words used in webpages associated with the third group of images represented by the selected third representative thumbnail image, wherein the identified words appear in webpages associated with the third group of images, represented by the selected third thumbnail image, in greater frequency than a frequency of appearance of the identified words in webpages associated with the fourth group of images represented by the unselected fourth thumbnail image;

executing, by the computer system, a second refined image search using a second search query comprising the automatically identified second keyword, wherein the second search query comprising the automatically identified second keyword is different than the user search query, wherein the second refined image search is performed using a second textual search query comprising the automatically identified second keyword; and

providing, by the computer system, for display on the user computing device search results, comprising a fifth group of images, from the second refined image search.

2. The method of claim 1,

wherein generating the similarity metric for images included in the second ranked image search results further comprises generating the similarity metric between combinations of images in the second plurality of images based at least in part on the identified common visual features.

3. The method of claim 2, wherein the visual features comprise scale-invariant features.

4. The method of claim 1, wherein the first ranked image search results comprise hyperlinks to resources.

5. The method of claim 1, wherein the method is performed using a graphic server operably coupled to a search engine, and the graphics server is coupled to the user computing device over a network.

6. The method of claim 1, the method further comprising providing for display on the user computing device, in association with the third thumbnail image, corresponding website source information and dimension information.

7. The method of claim 1, wherein a first image in the second plurality of images is included in both the third group and the fourth group at least partly in response to determining that the first image exceeds a similarity threshold with respect to other images in the third group and other images in the fourth group.

8. The method of claim 1, the method further comprising:

wherein the third thumbnail image corresponding to the third group of images and the fourth thumbnail image corresponding to the fourth group of images are displayed at the same time as separate groupings.

9. A system comprising:

one or more computing devices;

a non-transitory computer storage medium storing instructions that when executed by the one or more computing devices are configured to cause the system to perform operations comprising;

receiving a user search query consisting only of text;

receiving ranked image search results from a first image search corresponding to the user search query, the ranked search results comprising a first plurality of images;

classifying a first subset of the first plurality of images as a first group based at least in part on a first subject matter;

classifying a second subset of the first plurality of images as a second group based at least in part on a second subject matter;

providing for display on a user computing device a first visual representation corresponding to the first subset of images, the first visual representation comprising a first thumbnail image representative of the first subject matter, and a second visual representation corresponding to the second subset of images, the second visual representation comprising a second thumbnail image representative of the second subject matter;

receiving a selection of the first thumbnail image, the second thumbnail image being unselected;

automatically identifying a first keyword based at least in part on the user selection of the first thumbnail image, the first keyword not included in the user search query, wherein the identification of the first keyword is based at least in part on identifying words used in webpages associated with the first subset of images represented by the selected first thumbnail image, wherein the identified words appear in greater frequency in webpages associated with the first subset of images represented by the selected first thumbnail image than a frequency of appearance of the identified words in webpages associated with the second subset of images represented by the unselected second thumbnail image;

executing a first refined image search using a first search query comprising the automatically identified first keyword, wherein the first search query comprising the automatically identified first keyword is different than the user search query, wherein the first refined image search is performed using a first textual search query comprising the automatically identified first keyword;

receiving second ranked image search results from the first refined image search, the second ranked search results comprising a second plurality of images;

generating a similarity metric for images included in the second ranked image search results;

classifying a subset of the second plurality of images as a third group using the similarity metric for images included in the second plurality of images;

providing for display on the user computing device;

a visual representation corresponding to the third group of images, the visual representation corresponding to the third group of images comprising a third thumbnail image, and

a visual representation corresponding to a fourth group of images, the visual representation corresponding to the fourth group of images comprising a fourth thumbnail image;

receiving a selection of the third thumbnail image;

automatically identifying a second keyword based at least in part on the user selection of the third thumbnail image, the second keyword not included in the user search query;

executing a second refined image search using a second search query comprising the automatically identified second keyword, wherein the second search query comprising the automatically identified second keyword is different than the user search query, and wherein the second refined image search is performed using a second textual search query comprising the automatically identified second keyword; and

providing for display on the user computing device search results, comprising a fifth group of images, from the second refined image search.

10. The system of claim 9, further comprising:

a feature extractor configured to extract one or more visual features from one or more images; and

a classifier adapted to;

identify visual features common to two or more images using the extracted visual features;

generate a similarity metric between combinations of images based at least in part on the identified common visual features.

11. The system of claim 9, the operations further comprising providing for display, in association with the third thumbnail image, corresponding website source information and dimension information.

12. The system of claim 9, wherein a first image in the second plurality of images is included in both the third group and the fourth group at least partly in response to determining that the first image exceeds a similarity threshold with respect to other images in the third group and other images in the fourth group.

13. A computer implemented method comprising:

receiving, at a computer system comprising hardware, a user search query consisting only of text;

receiving, at the computer system, ranked image search results from a first image search corresponding to the user search query, the ranked search results comprising a first plurality of images;

classifying, by the computer system, a first subset of the first plurality of images as a first group based at least in part on a first subject matter;

classifying, by the computer system, a second subset of the first plurality of images as a second group based at least in part on a second subject matter;

providing, by the computer system, for display on a user computing device a first visual representation corresponding to the first subset of images, the first visual representation comprising a first thumbnail image representative of the first subject matter, and a second visual representation corresponding to the second subset of images, the second visual representation comprising a second thumbnail image representative of the second subject matter;

receiving, by the computer system, a selection of the first thumbnail image, the second thumbnail image being unselected;

automatically identifying, by the computer system, a first keyword based at least in part on the user selection of the first thumbnail image, the first keyword not included in the user search query, wherein the identification of the first keyword is based at least in part on identifying words used in webpages associated with the first subset of images represented by the selected first thumbnail image, wherein the identified words appear in greater frequency in webpages associated with the first subset of images represented by the selected first thumbnail image than a frequency of appearance of the identified words in webpages associated with the second subset of images represented by the unselected second thumbnail image;

executing a first refined image search using a first search query comprising the automatically identified first keyword, wherein the first search query comprising the automatically identified first keyword is different than the user search query, wherein the first refined image search is performed using a first textual search query comprising the automatically identified first keyword;

receiving, by the computer system, second ranked image search results from the first refined image search, the second ranked search results comprising a second plurality of images;

generating, by the computer system, a similarity metric for images included in the second ranked image search results;

classifying, by the computer system, a subset of the second plurality of images as a third group using the similarity metric for images included in the second plurality of images;

providing, by the computer system, for display on a user computing device;

a visual representation corresponding to the third group of images, the visual representation corresponding to the third group of images comprising a third thumbnail image, and

a visual representation corresponding to a fourth group of images, the visual representation corresponding to the fourth group of images comprising a fourth thumbnail image;

receiving, by the computer system, a selection of the third thumbnail image;

automatically identifying, by the computer system a second keyword based at least in part on the user selection of the third thumbnail image, the second keyword not included in the user search query;

executing a second refined image search using a search query comprising the automatically identified second keyword, wherein the search query comprising the automatically identified second keyword is different than the user search query, wherein the second refined image search is performed using a second textual search query comprising the automatically identified second keyword; and

providing, by the computer system, for display on the user computing device search results, comprising a fifth group of images, from the second refined image search.

14. The method of claim 13, the method further comprising;

extracting one or more visual features from images in the second plurality of images;

identifying visual features common to two or more images in the second plurality of images;

wherein generating the similarity metric for images included in the second ranked image search results further comprises generating the similarity metric between combinations of images in the second plurality of images based at least in part on the identified common visual features.

15. The method of claim 14, wherein the visual features comprise scale-invariant features.

16. The method of claim 13, wherein the first ranked image search results comprise hyperlinks to resources.

17. The method of claim 13, wherein the method is performed using a graphic server operably coupled to a search engine, and the graphics server is coupled to the user computing device over a network.

18. The method of claim 13, the method further comprising providing for display on the user computing device, in association with the third thumbnail image, corresponding website source information and dimension information.

19. The method of claim 13, wherein a first image in the second plurality of images is included in both the third group and the fourth group at least partly in response to determining that the first image exceeds a similarity threshold with respect to other images in the third group and other images in the fourth group.