Digital Image Search Training using Aggregated Digital Images

• US 20200134056A1
• Filed: 10/31/2018
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Sarkar, Mausoom, Gupta, Hiresh, Sinha, Abhishek
• Granted Patent: US 10,831,818 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/53 Querying
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

Digital image search training techniques and machine-learning architectures are described. In one example, a query digital image is received by service provider system, which is then used to select at least one positive sample digital image, e.g., having a same product ID. A plurality of negative sample digital images is also selected by the service provider system based on the query digital image, e.g., having different product IDs. The at least one positive sample digital image and the plurality of negative samples are then aggregated by the service provider system into a single aggregated digital image. At least one neural network is then trained by the service provider system using a loss function based on a feature comparison between the query digital image and samples from the aggregated digital image in a single pass.

Claims

1. In a digital medium image search environment, a method implemented by at least one computing device, the method comprising:

receiving, by the at least one computing device, a query digital image;

selecting, by the at least one computing device based on the query digital image, at least one positive sample digital image and a plurality of negative sample digital images;

generating, by the at least one computing device, an aggregated digital image by aggregating the at least one positive sample digital image with the plurality of negative sample digital images into a single digital image; and

training, by the at least one computing device, at least one neural network using a loss function as part of machine learning based on a feature comparison between the query digital image and;

the at least one positive sample digital image; and

the plurality of negative sample digital images of the aggregated digital image, respectively.

2. The method as described in claim 1, wherein the training of the at least one neural network based on the feature comparison between the query digital image and the at least one positive sample digital image and the plurality of negative sample digital images of the aggregated digital image, respectively, is performed in a single pass.

3. The method as described in claim 1, wherein the training of the at least one neural network includes generating a feature vector based on the query digital image and generating a three-dimensional feature volume from the at least one positive sample digital image and the plurality of negative sample digital images in the aggregated digital image.

4. The method as described in claim 3, wherein the training includes generating a feature map by convolving the feature vector and the three-dimensional feature volume, the feature map containing cosine distances from the feature vector of the query digital image to:

a feature vector of the at least one positive sample digital image from the three-dimensional feature volume; and

respective feature vectors of the plurality of negative sample digital images from the three-dimensional feature volume.

5. The method as described in claim 4, wherein the loss function includes a low mask and a high mask that specify respective thresholds within which the cosine distances are used to determine loss values between the cosine distances and the low mask or high mask, respectively.

6. The method as described in claim 1, further comprising generating a bounding box corresponding to at least a portion of an object depicted in the input digital image and wherein the generating of the search result is based on the bounding box.

7. The method as described in claim 6, wherein the generating of the bounding box includes passing the query digital image through a single shot detector (SSD) based deep neural network.

8. The method as described in claim 6, further comprising expanding a context of the bounding box around the portion of the object by expanding a margin of the generated bounding box as including an additional portion of the query digital image.

9. The method as described in claim 1, wherein the generating of the search result includes generating a feature vector for the input digital image, comparing the feature vector to feature vectors generated for respective digital images of a plurality of digital images, and the search result is based on the comparing.

10. The method as described in claim 1, wherein the selecting of the at least one positive sample digital image and the plurality of negative sample digital images based on the query digital image is based on respective product identifiers.

11. The method as described in claim 1, wherein the loss function is a double margin loss.

12. In a digital medium image search training environment, a system comprising:

a positive sample selection module implemented at least partially in hardware of a computing device to select at least one positive sample digital image from a plurality of digital images based on a query digital image;

a negative sample selection module implemented at least partially in hardware of the computing device to select a plurality of negative sample digital images from the plurality of digital images based on the query digital image;

an aggregation module implemented at least partially in hardware of the computing device to generate a single aggregated digital image by aggregating the at least one positive sample digital image with the plurality of negative sample digital images; and

a training module implemented at least partially in hardware of the computing device to train a neural network using a loss function as part of machine learning to generate a search result from a search of the plurality of digital images, the neural network trained through use of the positive sample digital image and the plurality of negative sample digital images within the single aggregated digital image in a single pass.

13. The system as described in claim 12, further comprising a search module implemented at least partially in hardware of the computing device to generate the search result in response to a search request that includes an input digital image based on the trained neural network.

14. The system as described in claim 12, wherein the positive sample selection module is configured to select at least one positive sample digital image based on a product identifier that corresponds to a product identifier of the query digital image.

15. The system as described in claim 12, wherein the negative sample selection module is configured to select the plurality of negative sample digital images as having a product identifier that does not correspond to a product identifier of the query digital image.

16. The system as described in claim 12, wherein the training module is configured to generate a feature vector based on the query digital image and generate a three-dimensional feature volume from the at least one positive sample digital image and the plurality of negative sample digital images in the aggregated digital image.

17. The system as described in claim 16, wherein the training module is configured to generate a feature map by convolving the feature vector and the three-dimensional feature volume, the feature map containing cosine distances from the query digital image to the at least one positive sample digital image and the plurality of negative sample digital images in the aggregated digital image, respectively.

18. Ina digital medium image search training environment, a system comprising:

means for receiving a search request that includes an input digital image; and

means for generating a search result from a plurality of digital images by at least one neural network based on the input digital image of the search request, the at least one neural network trained based on a feature comparison between a query digital image and an aggregated digital image that includes at least one positive sample digital image along with the plurality of negative sample digital images arranged in a grid that are selected based on the query digital image.

19. The system as described in claim 18, wherein the at least one neural network is trained based on a feature comparison between the query digital image and the at least one positive sample digital image and the plurality of negative sample digital images of the aggregated digital image.

20. The system as described in claim 19, wherein the training is performed for the at least one positive sample digital image and the plurality of negative sample digital images in a single pass using the aggregated digital image.