TOPIC ASSOCIATION AND TAGGING FOR DENSE IMAGES

• US 20180267996A1
• Filed: 03/20/2017
• Published: 09/20/2018
• Est. Priority Date: 03/20/2017
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: LIN, ZHE, SHEN, XIAOHUI, ZHANG, JIANMING, JIN, HAILIN, LI, YINGWEI
• Granted Patent: US 10,496,699 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/21355 nonlinear criteria, e.g. embedding a manifold in a Euclidean space
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

A framework is provided for associating dense images with topics. The framework is trained utilizing images, each having multiple regions, multiple visual characteristics and multiple keyword tags associated therewith. For each region of each image, visual features are computed from the visual characteristics utilizing a convolutional neural network, and an image feature vector is generated from the visual features. The keyword tags are utilized to generate a weighted word vector for each image by calculating a weighted average of word vector representations representing keyword tags associated with the image. The image feature vector and the weighted word vector are aligned in a common embedding space and a heat map is computed for the image. Once trained, the framework can be utilized to automatically tag images and rank the relevance of images with respect to queried keywords based upon associated heat maps.

Claims

1. A computer system comprising:

one or more processors; and

one or more computer storage media storing computer-useable instructions that, when used by the one or more processors, cause the one or more processors to;

receive a plurality of images, each image of the plurality of images being associated with a plurality of tags and each image of the plurality of images being comprised of a plurality of regions, each region of each image comprising less than an entirety of the image it comprises;

for each region of each image of the plurality of images, generate an image feature vector from one or more visual features;

for each image of the plurality of images, generate a weighted word vector from the associated plurality of tags; and

for each image, compute a heat map corresponding thereto by aligning the image feature vector for each region of a given image and the weighted word feature vector into a common embedding space utilizing cosine similarity loss, wherein a plurality of regions of the heat map corresponds to the plurality of regions of the given image and wherein at least one region of the plurality of regions of the heat map corresponds to each of the plurality of tags.

2. The computing system of claim 1, wherein for each image of the plurality of images, the one or more processors are further caused to compute the one or more visual features.

3. The computing system of claim 2, wherein the one or more visual features are computed utilizing a convolutional neural network.

4. The computing system of claim 1, wherein for each image of the plurality of images, the one or more processor are further caused to:

generate a word vector representation for each of the associated plurality of tags;

calculate a weighted average of the generated word vector representations to generate the weighted word vector; and

normalize the weighted word vector in the common embedding space.

5. The computing system of claim 4, wherein the word vector representation for each of the associated plurality of tags is generated utilizing Pointwise Mutual Information.

6. The computing system of claim 4, wherein the weighted average is calculated, at least in part, uniformly for each of the plurality of tags.

7. The computing system of claim 4, wherein the weighted average is calculated, at least in part, utilizing inverse document frequency.

8. The computing system of claim 4, wherein each of the associated plurality of tags is a user-provided tag, and wherein the weighted average is calculated, at least in part, utilizing a tag order in which a first tag of the plurality of associated tags is assigned a greater weight than a second tag of the plurality of associated tags when the first tag is provided by the user before the second tag.

10. The computing system of claim 1, wherein the image feature vectors for each of the plurality of regions collectively comprise a vector map, and wherein the embedded image feature vectors for each of the plurality of regions collectively comprise an image embedding map.

11. The computing system of claim 10, wherein the one or more processors further are caused to:

pool a local feature vector from the vector map;

concatenate the local feature vector with an embedded image feature vector for one of the plurality of regions to form a topic-guided feature vector;

generate a second embedded image feature vector from the topic-guided feature vector; and

compute a second heat map corresponding to the image by aligning the second image embedding vector with the weighted word vector into the common embedding space utilizing cosine similarity loss.

9. (canceled)

12. A computer-implemented method for tagging images, the method comprising:

receiving an image associated with a plurality of user-provided tags, the image being comprised of a plurality of regions, each region comprising less than the entire image;

generating an embedded image feature vector for each of the plurality of regions;

generate an image-specific weighted word vector from the plurality of user-provided tags; and

computing a first heat map corresponding to the image by aligning the embedded image vector for each region of the image and the weighted word vector into a common embedding space using cosine similarity loss, wherein a plurality of regions of the heat map corresponds to the plurality of regions of the image, and wherein at least one region of the heat map corresponds to each of the plurality of user-provided tags.

13. The method of claim 12, wherein generating the embedded image feature vector for each for the plurality of regions comprises:

computing one or more visual features associated each of the plurality of regions,

generating an image feature vector associated with each of the plurality of regions from the associated one or more visual features; and

generating the embedded image feature vector for each of the plurality of regions from the associated image feature vector utilizing a convolutional neural network.

14. The method of claim 12, wherein generating the image-specific weighted word vector from the plurality of user-provided tags comprises:

generating a word vector representation for each of the associated plurality of user-provided tags; and

calculating a weighted average of the generated word vector representations to generate the image-specific weighted word vector.

15. The method of claim 14, wherein the word vector representation for each of the associated plurality of user-provided tags is generated utilizing Pointwise Mutual Information.

16. The method of claim 14, wherein the weighted average is calculated, at least in part, utilizing inverse document frequency.

17. The method of claim 14, wherein the weighted average is 18ovided tags is assigned a greater weight than a second tag of the plurality of user-provided tags when the first tag is provided by a user before the second tag.

18. The method of claim 13, wherein the image feature vectors for each of the plurality of regions collectively comprise a vector map and wherein the image embedding vectors for each of the plurality of regions collectively comprise an image embedding map.

19. The method of claim 18, further comprising:

pooling a local feature vector from a particular region of the heat map utilizing the vector map, the particular region of the heat map being associated with one of the plurality of user-provided tags;

concatenating the local feature vector with the an image embedding vector for one of the plurality of regions to form a topic-guided feature vector generating a second image embedding vector from the topic-guided feature vector; and

computing a second heat map corresponding to the image by aligning the second image embedding vector with the soft topic feature vector into the common embedding space utilizing cosine similarity loss.

20. A computing system comprising:

means for generating an image embedding vector for each of the plurality of regions of an image utilizing a convolutional neural network;

means for generating a weighted word vector for the image by calculating a weighted average of a plurality of word vector representations, each of the plurality of word vector representations being generated for a different one of a plurality of tags associated with the image; and

means for computing a heat map corresponding to the image by aligning the image embedding vector for each region of the image and the soft topic feature vector into a common embedding space utilizing cosine similarity loss, wherein a plurality of regions of the heat map corresponds to the plurality of regions of the image, and wherein at least one region of the heat map corresponds to each of the plurality of tags associated with the image.