Requesting search results by user interface gesture combining display objects

• US 9,195,720 B2
• Filed: 03/14/2013
• Issued: 11/24/2015
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Penha, Henrique Dias, Brophy, Mark, Inwood, Mathew, Koser, Mikkel Crone, Jenkins, Thomas, Skory, Adam, Bringert, Bjorn E., Barra, Hugo B., Stewart, Andrew Anderson, Hamilton, Robert W.
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Tran, Loc
• Application Number:
  US13/829,520
• Publication Number:
  US 20140280049A1
• Time in Patent Office: 985 Days
• Field of Search: 707/722, 707/736, 707/758
• US Class Current: 1/1
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/9032 Query formulation

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for receiving, from a user device, data indicating a user performed a user input gesture combining a first display object in a plurality of display objects with a second display object in the plurality of display objects; identifying attributes that are associated with both the first display object and the second display object; and performing a search based on the attributes.

Claims

1. A method comprising:

receiving, from a user device, data indicating a user performed a user input gesture combining a first display object in a plurality of non-textual display objects displayed on the user device with a second display object in the plurality of non-textual display objects;

determining first attributes for the first display object and second attributes for the second display object;

identifying attributes included in both the first attributes and the second attributes;

automatically sending, to a web search engine, a search request comprising one or more search terms from each of the identified attributes in response to identifying the attributes included in both the first attributes and the second attributes;

receiving web search results from the web search engine in response to the search request; and

providing the web search results for display on the user device.

2. The method of claim 1, where the data indicating the user performed the user input gesture comprises (i) data representing a user dragging the first display object to overlap the second display object, and (ii) data representing the user releasing the first display object while the first display object overlaps the second display object.

3. The method of claim 1, further comprising:

retrieving first data for the first display object and second data for the second display object, where the first data includes indexed resources for a first entity and the second data includes indexed resources for a second entity, and the first data and the second data each represent an association between the respective entity and the respective display object;

where identifying the attributes included in both the first attributes and the second attributes comprises identifying attributes that are associated with both the first entity and the second entity.

4. The method of claim 1, where the first display object and the second display object are a first image and a second image, respectively, and where identifying the attributes included in both the first attributes and the second attributes comprises:

identifying visual similarities between the first image and the second image, where the visual similarities comprise similarities in color, texture, or shape; and

identifying attributes that represent the visual similarities.

5. The method of claim 1, where identifying the attributes included in both the first attributes and the second attributes comprises:

identifying a first entity associated with the first display object, wherein the first entity is a data element that represents a first person, place, thing, or concept;

identifying a second entity associated with the second display object, wherein the second entity is a data element that represents a second person, place, thing, or concept; and

identifying attributes that are associated with both the first person, place, thing, or concept and the second person, place, thing, or concept.

6. The method of claim 1, comprising:

prior to identifying the attributes included in both the first attributes and the second attributes, receiving one or more additional user input gestures, where each additional user input gesture combines a corresponding display object with the first display object or the second display object, and where identifying the attributes included in both the first attributes and the second attributes comprises identifying attributes included in both the first attributes and the second attributes and that are associated with each of the one or more corresponding display objects.

7. The method of claim 1, further comprising:

determining a threshold score that is a measure of how the first object is related to the second object;

determining that one or more of the search results satisfy the threshold score;

sending, to the user device, response data representing the determination, where the user device displays a visual indication of the response data.

22. The method of claim 1, comprising:

identifying a first entity associated with the first display object; and

identifying a second entity associated with the second display object, wherein determining the first attributes for the first display object and the second attributes for the second display object comprises;

determining the first attributes for the first entity; and

determining the second attributes for the second entity.

8. A system comprising:

a processor; and

a non-transitory computer-readable medium coupled to the processor and having instructions stored thereon, which, when executed by the processor, cause the processor to perform operations comprising;

receiving, from a user device, data indicating a user performed a user input gesture combining a first display object in a plurality of non-textual display objects displayed on the user device with a second display object in the plurality of non-textual display objects;

determining first attributes for the first display object and second attributes for the second display object;

identifying attributes included in both the first attributes and the second attributes;

automatically sending, to a web search engine, a search request comprising one or more search terms from each of the identified attributes in response to identifying the attributes included in both the first attributes and the second attributes;

receiving web search results from the web search engine in response to the search request; and

providing the web search results for display on the user device.

9. The system of claim 8, where the data indicating the user performed the user input gesture comprises (i) data representing a user dragging the first display object to overlap the second display object, and (ii) data representing the user releasing the first display object while the first display object overlaps the second display object.

10. The system of claim 8, the operations further comprising:

retrieving first data for the first display object and second data for the second display object, where the first data includes indexed resources for a first entity and the second data includes indexed resources for a second entity, and the first data and the second data each represent an association between the respective entity and the respective display object;

where identifying the attributes included in both the first attributes and the second attributes comprises identifying attributes that are associated with both the first entity and the second entity.

11. The system of claim 8, where the first display object and the second display object are a first image and a second image, respectively, and where identifying the attributes included in both the first attributes and the second attributes comprises:

identifying visual similarities between the first image and the second image, where the visual similarities comprise similarities in color, texture, or shape; and

identifying attributes that represent the visual similarities.

12. The system of claim 8, where identifying the attributes included in both the first attributes and the second attributes comprises:

identifying a first entity associated with the first display object, wherein the first entity is a data element that represents a first person, place, thing, or concept;

identifying a second entity associated with the second display object, wherein the second entity is a data element that represents a second person, place, thing, or concept; and

identifying attributes that are associated with both the first person, place, thing, or concept and the second person, place, thing, or concept.

13. The system of claim 8, the operations further comprising:

prior to identifying the attributes included in both the first attributes and the second attributes, receiving one or more additional user input gestures, where each additional user input gesture combines a corresponding display object with the first display object or the second display object, and where identifying the attributes included in both the first attributes and the second attributes comprises identifying attributes included in both the first attributes and the second attributes and that are associated with each of the one or more corresponding display objects.

14. The system of claim 8, the operations further comprising:

determining a threshold score that is a measure of how the first object is related to the second object;

determining that one or more of the search results satisfy the threshold score;

sending, to the user device, response data representing the determination, where the user device displays a visual indication of the response data.

23. The system of claim 8, the operations further comprising:

identifying a first entity associated with the first display object; and

identifying a second entity associated with the second display object, wherein determining the first attributes for the first display object and the second attributes for the second display object comprises;

determining the first attributes for the first entity; and

determining the second attributes for the second entity.

15. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform operations comprising:

receiving, from a user device, data indicating a user performed a user input gesture combining a first display object in a plurality of non-textual display objects displayed on the user device with a second display object in the plurality of non-textual display objects;

determining first attributes for the first display object and second attributes for the second display object;

identifying attributes included in both the first attributes and the second attributes;

automatically sending, to a web search engine, a search request comprising one or more search terms from each of the identified attributes in response to identifying the attributes included in both the first attributes and the second attributes;

receiving web search results from the web search engine in response to the search request; and

providing the web search results for display on the user device.

16. The non-transitory computer-readable medium of claim 15, where the data indicating the user performed the user input gesture comprises (i) data representing a user dragging the first display object to overlap the second display object, and (ii) data representing the user releasing the first display object while the first display object overlaps the second display object.

17. The non-transitory computer-readable medium of claim 15, the operations further comprising:

retrieving first data for the first display object and second data for the second display object, where the first data includes indexed resources for a first entity and the second data includes indexed resources for a second entity, and the first data and the second data each represent an association between the respective entity and the respective display object;

where identifying the attributes included in both the first attributes and the second attributes comprises identifying attributes that are associated with both the first entity and the second entity.

18. The non-transitory computer-readable medium of claim 15, where the first display object and the second display object are a first image and a second image, respectively, and where identifying the attributes included in both the first attributes and the second attributes comprises:

identifying visual similarities between the first image and the second image, where the visual similarities comprise similarities in color, texture, or shape; and

identifying attributes that represent the visual similarities.

19. The non-transitory computer-readable medium of claim 15, where identifying the attributes included in both the first attributes and the second attributes comprises:

identifying a first entity associated with the first display object, wherein the first entity is a data element that represents a first person, place, thing, or concept;

identifying a second entity associated with the second display object, wherein the second entity is a data element that represents a second person, place, thing, or concept; and

identifying attributes that are associated with both the first person, place, thing, or concept and the second person, place, thing, or concept.

20. The non-transitory computer-readable medium of claim 15, the operations further comprising:

prior to identifying the attributes included in both the first attributes and the second attributes, receiving one or more additional user input gestures, where each additional user input gesture combines a corresponding display object with the first display object or the second display object, and where identifying the attributes included in both the first attributes and the second attributes comprises identifying attributes included in both the first attributes and the second attributes and that are associated with each of the one or more corresponding display objects.

21. The non-transitory computer-readable medium of claim 15, the operations further comprising:

determining a threshold score that is a measure of how the first object is related to the second object;

determining that one or more of the search results satisfy the threshold score;

sending, to the user device, response data representing the determination, where the user device displays a visual indication of the response data.

24. The non-transitory computer-readable medium of claim 15, the operations further comprising:

identifying a first entity associated with the first display object; and

identifying a second entity associated with the second display object, wherein determining the first attributes for the first display object and the second attributes for the second display object comprises;

determining the first attributes for the first entity; and

determining the second attributes for the second entity.