MACHINE LEARNING IMAGE PROCESSING

• US 20180012110A1
• Filed: 04/03/2017
• Published: 01/11/2018
• Est. Priority Date: 07/06/2016
• Status: Active Grant

• Current Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Original Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Inventors: SOUCHE, Christian, YANG, Junmin, NARESSI, Alexandre
• Granted Patent: US 10,210,178 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A machine learning image processing system performs natural language processing (NLP) and auto-tagging for an image matching process. The system facilitates an interactive process, e.g., through a mobile application, to obtain an image and supplemental user input from a user to execute an image search. The supplemental user input may be provided from a user as speech or text, and NLP is performed on the supplemental user input to determine user intent and additional search attributes for the image search. Using the user intent and the additional search attributes, the system performs image matching on stored images that are tagged with attributes through an auto-tagging process.

Claims

1. A machine learning image processing system comprising:

a data repository storing images and tags for each image, wherein the tags for each image describe attributes of an object in the image;

a network interface to connect the machine learning image processing system to at least one network;

at least one processor to execute machine readable instructions stored on at least one non-transitory computer readable medium;

at least one data storage to store a plurality of image attribute machine learning classifiers,

wherein the plurality of image attribute machine learning classifiers comprise convolutional neural networks trained to identify the attributes;

wherein the machine readable instructions comprise machine readable instructions for an auto-tagging subsystem, and the at least one processor is to execute the machine readable instructions for the auto-tagging subsystem to;

apply each image stored in the data repository to the plurality of image attribute machine learning classifiers;

determine predictions for a plurality of image attribute categories from outputs of the plurality of image attribute machine learning classifiers;

determine the attributes of the object in each image stored in the data repository from the predictions; and

tag each image stored in the data repository with the determined attributes for the object in the image.

wherein the machine readable instructions comprise machine readable instructions for an image matching subsystem, and the at least one processor is to execute the machine readable instructions for the image matching subsystem to;

receive, via the network interface, a target image from a mobile application connected to the machine learning image processing system via the at least one network;

receive, via the network interface, supplemental user input associated with the target image from the mobile application connected to the image processing computer via the at least one network;

apply the target image to the plurality of image attribute machine learning classifiers;

determine predictions for the plurality of image attribute categories from applying the target image to the plurality of image attribute machine learning classifiers; and

determine target image attributes for an object in the target image from the predictions for the target image determined by the plurality of image attribute machine learning classifiers;

apply the supplemental user input to a natural language processing model to determine at least one supplemental image search attribute;

identify a matching subset of the images stored in the data repository that match the target image based on image search attributes determined from the target image attributes and the at least one supplemental image search attribute; and

transmit, via the network interface, the matching subset of images to the mobile application for display by the mobile application.

2. The machine learning image processing system of claim 1, wherein to determine the image search attributes determined from the target image attributes and the at least one supplemental image search attribute, the at least one processor is to:

determine object attribute search criteria from applying the supplemental user input to the natural language processing model;

determine whether the object attribute search criteria comprise a modification to the target image attributes or an additional image attribute;

in response to determining the object attribute search criteria comprises the modification to the target image attributes;

create the at least one supplemental image search attribute by modifying at least one of the target image attributes according to the object attribute search criteria; and

determine the image search attributes from the modified at least one of the target image attribute and the target image attributes that are not modified;

in response to determining the object attribute search criteria comprises the additional image attribute, determine the image search attributes from the additional image attribute and the target image attributes.

3. The machine learning image processing system of claim 1, wherein to identify a matching subset of the images stored in the data repository that match the target image based on image search attributes determined from the target image attributes and the at least one supplemental image search attribute the at least one processor is to:

execute a similarity matching between the target image and the images stored in the data repository to identify a first subset of matching images; and

determine, from the first subset of matching images, the matching subset of the images according to the at least one supplemental image search attribute.

4. The machine learning image processing system of claim 1, wherein to determine, from the first subset of matching images, the matching subset of the images according to the at least one supplemental image search attribute, the at least one processor is to:

search the first subset of matching images for a second subset of images that have the at least one supplemental image search attribute to identify the matching subset of the images stored in the data repository.

5. The machine learning image processing system of claim 1, wherein the at least one processor is to execute the machine readable instructions for the image matching subsystem to:

determine similarities between the target image attributes and attributes of images stored in the data repository; and

identify an initial subset of matching images from the data repository based on the similarities, and

wherein to identify a matching subset of the images comprises identifying the matching subset of the images based on the initial subset of matching images and the at least one supplemental image search attribute.

6. The machine learning image processing system of claim 5, determine similarities between the target image attributes and attributes of images stored in the data repository, the at least one processor is to:

determine a hamming distance between the target image attributes and the attributes of the images stored in the data repository; and

select at least one of the images stored in the data repository having a smallest hamming distance relative to hamming distances determined for the comparisons of other ones of the images stored in the data repository.

7. The machine learning image processing system of claim 1, wherein to determine predictions for a plurality of image attribute categories, the at least one processor is to:

determine the predictions from an output of a softmax layer of each of the plurality of image attribute machine learning classifiers.

8. The machine learning image processing system of claim 7, wherein a confidence value is determined from the output of the softmax layer, the confidence value indicating an accuracy of a prediction by one of the plurality of image attribute machine learning classifiers for each of a plurality of classes.

9. The machine learning image processing system of claim 7, wherein the tagged attributes are determined according to the confidence values.

10. The machine learning image processing system of claim 1, wherein the at least one processor is to:

generate the plurality of image attribute machine learning classifiers from training sets for each class of a plurality of classes of objects.

11. The machine learning image processing system of claim 1, wherein the supplemental user input comprises speech or text provided by a user.

12. A visual recommendation system comprising:

a data repository storing images, wherein the stored images include meta data comprised of tags describing attributes of the stored images, and wherein the tags are determined from applying the stored images to a plurality of image attribute machine learning classifiers classifying the stored images in classes for the tags;

at least one processor to;

receive a digital image of an object of interest to a user;

apply the digital image to a first machine learning classifier to identify the object;

apply an image of the object to a second machine learning classifier to determine attributes of the image of the object;

determine an initial subset of images stored in the data repository that are visually similar to the object based on a comparison of the attributes of the image of the object and attributes of the images;

receive supplemental user input associated with the initial subset of images and the object;

apply the supplemental user input to a natural language processing model to determine at least one supplemental image search attribute;

determine object search criteria from the at least one supplemental image search attribute and at least one of the attributes of the image of the object and the attributes of the initial subset of images;

search the tags in the meta data of the stored images according to the object search criteria to identify a matching subset of the images stored in the data repository; and

transmit visual recommendations for the object to a device via a network, wherein the visual recommendations comprise the matching subset of images.

13. The visual recommendation system of claim 12, wherein the stored images comprise images of products, and the visual recommendations comprise images of a subset of the products associated with the object that have the at least one supplemental image search attribute.

14. The visual recommendation system of claim 13, wherein the tags comprise categories of attributes associated with the products.

15. The visual recommendation system of claim 12, wherein the digital image of the object is captured by a camera of the device, and is transmitted to the visual recommendation system from the device.

16. The visual recommendation system of claim 12, wherein the at least one supplemental image search attribute comprises a modification to one of the attributes of the image of the object.

17. The visual recommendation system of claim 12, wherein the at least one supplemental image search attribute comprises an additional image attribute further describing the object of interest to the user.

18. A mobile device comprising:

a camera;

a display;

a microphone;

at least one processor; and

a non-transitory computer readable storing machine readable instructions for a mobile application, wherein the at least one processor is to execute the machine readable instructions to;

cause the camera to capture an image of an object;

transmit, via a network interface, the image of the object to a machine learning image processing system,

wherein the machine learning image processing system stores images and meta data comprised of tags describing attributes of the stored images, and wherein the tags are determined from applying the stored images to a plurality of image attribute machine learning classifiers classifying the stored images in classes for the tags;

receive an initial subset of the stored images from the machine learning image processing system that are visually similar to the object, wherein the machine learning image processing system determines the initial subset of the stored images based on a comparison of attributes of the image of the object and the attributes of the stored images;

display the initial subset of the stored images on the display;

receive, via the microphone, speech describing supplemental user input in response to displaying the initial subset of the stored images;

transmit the speech or text determined from the speech, via the network interface, to the machine learning image processing system,

wherein the machine learning image processing system applies the speech or the text to a natural language processing model to determine object search criteria, and identifies a matching subset of the stored images from the object search criteria and at least one of the attributes of the image of the object and the attributes of the initial subset of the stored images;

receive the matching subset of the stored images, via the network interface, from the machine learning image processing system; and

display the matching subset of the stored images on the display.

19. The mobile device of claim 18, wherein the matching subset of the stored images comprise visual product recommendations associated with the object and the supplemental user input.

20. The mobile device of claim 18, wherein the supplemental user input describes attributes of an object desired for purchase by the user providing the supplemental user input.