Object relation builder

• US 9,881,226 B1
• Filed: 09/24/2015
• Issued: 01/30/2018
• Est. Priority Date: 09/24/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Rybakov, Oleg, Benitez, Matias Omar Gregorio, Dirac, Leo Parker, Joseph, Rejith George, Mohan, Vijai, Thirumalai, Srikanth
• Primary Examiner(s): Chang, Jon
• Application Number:
  US14/864,605
• Time in Patent Office: 859 Days
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Recommendations can be generated even in situations where sufficient user information is unavailable for providing personalized recommendations. Instead of generating recommendations for an item based on item type or category, a relation graph can be consulted that enables other items to be recommended that are related to the item in some way, which may be independent of the type or category of item. For example, images of models, celebrities, or everyday people wearing items of clothing, jewelry, handbags, shoes, and other such items can be received and analyzed to recognize those items and cause them to be linked in the relation graph. When generating recommendations or selecting advertisements, the relation graph can be consulted to recommend products that other people have obtained with the item from any of a number of sources, such that the recommendations may be more valuable to the user.

Claims

1. A computer-implemented method, comprising:

obtaining, via one or more computing systems, a plurality of images, at least a subset of the plurality of images each including representations of multiple items;

segmenting, via the one or more computing systems, each image of the subset into one or more item portions;

comparing, via the one or more computing systems, the item portions against an electronic catalog to identify items represented in the item portions;

determining, via the one or more computing systems, based at least in part upon image data from the item portions, visually similar items for the identified items;

generating, via the one or more computing systems, a relation graph including a plurality of items and links between those items, the plurality of items including the identified items and the visually similar items, the links connecting items having been recognized together in one or more images of the subset;

associating, via the one or more computing systems and for each link of the relation graph, a relation score, the relation score for a link being based at least in part upon a number of times the items corresponding to the link were recognized together in an image of the plurality of images;

receiving, via the one or more computing systems, a request for content relating to an item of interest;

determining, via the one or more computing systems, from the relation graph, a set of related items linked to the item of interest;

generating, via the one or more computing systems, with primary content for the item of interest, supplemental content for at least a subset of the set of related items linked to the item of interest; and

causing, via the one or more computing systems, the primary content for the item of interest and the supplemental content for the at least subset of the set of related items linked to the item of interest to be displayed on a computing device.

2. The computer-implemented method of claim 1, further comprising:

selecting the subset of the set of related items based at least in part upon the relation scores for the corresponding links between the set of related items and the item of interest.

3. The computer-implemented method of claim 1, further comprising:

identifying a person represented in an image, of the plurality of images, including the item of interest; and

determining the relation score for an item represented in the image and the item of interest based at least in part upon an identity of the person.

4. The computer-implemented method of claim 1, further comprising:

determining a geographic location associated with the request for content; and

selecting the subset of the set of related items based at least in part upon the geographic location, the relation graph including metadata indicating one or more geographic locations for which items were recognized together in one or more images of the plurality of images.

5. A computer-implemented method, comprising:

obtaining, via at least one processor of a computing device, a plurality of images each including a representation of one or more items;

recognizing, via the at least one computer processor, items represented together in one or more images of the plurality of images;

generating, via the at least one processor, a relation graph including information for the items, the relation graph including links between the items represented together in one or more images of the plurality of images;

generating, via the at least one processor, a relation score for each of the links;

associating, via at least one processor, for each link of the relation graph, the relation score for a link being based at least in part upon a number of times the items corresponding to the link were recognized together in the one or more of the plurality of images;

determining related content associated with an item of interest based at least in part on the relation graph, the related content corresponding to one or more or the items linked to the item of interest in the relation graph; and

displaying, on the computing device, a primary content for the item of interest and a supplemental content for at least a subset of a set of related items linked to the item of interest.

6. The computer-implemented method of claim 5, further comprising:

segmenting each of the images into at least one item portion; and

comparing the at least one item portion against an electronic catalog in order to attempt to recognize an item in each of the item portions.

7. The computer-implemented method of claim 6, further comprising:

analyzing each of the images using a connected components analysis in order to identify the at least one item portion.

8. The computer-implemented method of claim 5, further comprising:

determining a foreground portion and a background portion of each image; and

analyzing the foreground portion of each image with an object recognition algorithm to attempt to recognize at least one item in each of the images.

9. The computer-implemented method of claim 5, further comprising:

determining one or more visually similar items for at least a subset of the one or more items; and

including information for the visually similar items in the relation graph, the visually similar items for a specified item each being linked to other items linked to the specified item.

10. The computer-implemented method of claim 9, further comprising:

generating a set of histogram descriptors for an item of the one or more items; and

comparing the set of histogram descriptors against histogram descriptors stored in an electronic catalog in order to identify the visually similar items.

11. The computer-implemented method of claim 5, further comprising:

receiving an additional plurality of images;

recognizing, via the at least one processor, items represented together in one or more images of the additional plurality of images; and

updating relation scores in the relation graph for items represented together in the one or more images of the additional plurality of images.

12. The computer-implemented method of claim 11, further comprising:

assigning a decay value to the relation scores, whereby a specified relation score of the relation scores decreases over time unless a subsequent occurrence of the items together in an image is determined.

13. The computer-implemented method of claim 5, wherein the relation score for a link between two items is based at least in part upon at least one of a co-location score, a calculated visual similarity, or a number of times the two items were represented together in the one or more images.

14. The computer-implemented method of claim 5, further comprising:

recognizing the items represented together in the one or more images of the plurality of images based further upon metadata included with at least a subset of the one or more images, at least a portion of the metadata identifying at least a subset of the items represented in the image.

15. A system, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the system to;

receive a request for content relating to an item of interest, the content including a plurality of images each including representations of multiple items;

generating a relation graph based at least in part on the plurality of images, the relation graph including information for the items;

determine, using the relation graph, a set of items linked to the item of interest, links between pairs of items of the relation graph indicating that the pairs of items were recognized together in at least one image of the plurality of images, each link having a link score representative of at least a number of times a respective pair of items were recognized together;

select, from the set of items and based at least in part upon respective relation scores, a subset of the set of items;

provide, with primary content for the item of interest, related content for the subset of the set of items; and

displaying, on a display of the system, the primary content along with the related content for the subset of the set of items.

16. The system of claim 15, wherein the instructions when executed further cause the system to:

determine a geographic location associated with the request; and

select the subset of the set of items based at least in part upon geographic data associated with the items of the subset, the geographic data indicating one or more geographic locations where the items were recognized in an image with the item of interest.

17. The system of claim 15, wherein the related content includes at least one of recommendation content or advertising.

18. The system of claim 15, wherein each respective relation score for a link between a pair of items is determined based at least in part upon an identity of a person recognized in an image in which the pair of items is represented together.

19. The system of claim 15, wherein each respective relation score for a link between a pair of items is determined based at least in part upon a type of source of an image in which the pair of items is represented together.