Structured Knowledge Modeling and Extraction from Images

• US 20170132526A1
• Filed: 12/22/2015
• Published: 05/11/2017
• Est. Priority Date: 11/11/2015
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Cohen, Scott D., Chang, Walter Wei-Tuh, Price, Brian L., Elhoseiny, Mohamed Hamdy Mahmoud Abdelbaky
• Granted Patent: US 11,514,244 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

Techniques and systems are described to model and extract knowledge from images. A digital medium environment is configured to learn and use a model to compute a descriptive summarization of an input image automatically and without user intervention. Training data is obtained to train a model using machine learning in order to generate a structured image representation that serves as the descriptive summarization of an input image. The images and associated text are processed to extract structured semantic knowledge from the text, which is then associated with the images. The structured semantic knowledge is processed along with corresponding images to train a model using machine learning such that the model describes a relationship between text features within the structured semantic knowledge. Once the model is learned, the model is usable to process input images to generate a structured image representation of the image.

Claims

1. In a digital medium environment to learn a model that is usable to explicitly correlate image features of an input image with text features automatically and without user intervention, a method implemented by at least one computing device comprising:

obtaining training data by the at least one computing device, the training data including images and associated text;

extracting text features as structured semantic knowledge from the associated text of the training data using natural language processing by the at least one computing device;

training a model using the structured semantic knowledge and the images as part of machine learning by the at least one computing device, the model once trained is configured to form a structured image representation to explicitly correlate the image features of the input image with at least one of the extracted text features.

2. The method as described in claim 1, further comprising generating the structured image representation as a descriptive summarization of the input image by associating structured tags with the input image using the trained model, the associating including computing probabilities that the extracted text features and the image features of the input image describe a same concept.

3. The method as described in claim 1, wherein the associated text is free form and unstructured.

4. The method as described in claim 3, wherein the associated text is a caption or metadata of a respective said image.

5. The method as described in claim 1, wherein the structured semantic knowledge is in a form of <

subject, attribute>

or a <

subject, predicate, object>

tuple.

6. The method as described in claim 1, wherein the extracting includes:

extracting a plurality of the text features of the structured semenatic knowledge using of a plurality of different tuple extraction techniques; and

based on at least partial consensus in the plurality of the text features of the structured semantic knowledge extracted using the plurality of different tuple extraction techniques to;

;

determine whether to correct at least one of the text features of the structured semantic knowledge;

remove the at least one of the text features of the structured semantic knowledge, or

identify a degree of confidence in the extracting of respective said text features of the structured semantic knowledge by the at least one computing device.

7. The method as described in claim 1, wherein the extracting includes:

jointly extracting the structured semantic knowledge for a plurality of individual said text features of associated with a single respective said image; and

based on at least partial consensus from to the plurality of individual said text features of the text associated with the single respective said image to;

determine whether to correct at least one of the plurality of individual text features of the structured semantic knowledge;

remove the at least one of the plurality of individual said text features of the structured semantic knowledge;

or

identify a degree of confidence in the extracting of respective said text feature of the structured semantic knowledge by the at least one computing device.

8. The method as described in claim 1, wherein the extracting includes localizing at least part of the structured semantic knowledge as corresponding to respective subjects, predicates, or objects, in respective said images of the training data.

9. The method as described in claim 1, wherein the training includes adapting the text features or the image features of the image information, one to another, such that similar structured knowledge concepts have nearby and related representations within a vector space as part of the structured semantic knowledge.

10. The method as described in claim 1, wherein the model explicitly correlates the image features of the input image with the text features such that at least one of the image features is explicitly correlated with a first one of the text features but not a second one of the text features.

11. The method as described in claim 1, wherein the structured image representation of the input image explicitly correlates structured semantic knowledge obtained from the trained model with the input image and is performed without using text otherwise associated with the input image.

12. In a digital medium environment to use a model to explicitly correlate text features with image features of an input image automatically and without user intervention, a system by at least one computing device comprising:

an extractor module to extract structured semantic knowledge from text associated with images in training data using natural language processing, the structured semantic knowledge describing the text features;

a model training module to train a model using the structured semantic knowledge and image features of the images in the training data as part of machine learning by the at least one computing device, the model configured for determining probabilities to predict how well image features of the input image correlate to the extracted text features.

13. The system as described in claim 12, wherein the associated text is free form and unstructured.

14. The system as described in claim 12, wherein the structured semantic knowledge is in a form of <

subject, attribute>

or a <

subject, predicate, object>

tuple.

15. The system as described in claim 12, wherein the extractor module is configured to localize at least part of the structured semantic knowledge as corresponding to respective objects in respective said images.

16. The system as described in claim 12, further comprising a structured image representation use module configured to use the structured image representation to locate the input image as part of an image search as part of a determination of how well the structured image representation corresponds to a search query of the image search.

17. The system as described in claim 12, further comprising a structured image representation use module configured to generate a caption for the image based on the structured image representation.

18. The system as described in claim 12, further comprising a structured image representation use module configured to deduce, based on the structured image representation, scene properties of the input image.

19. In a digital medium environment to use a model as part of an image search to locate an input image automatically and without user intervention, a method implemented by at least one computing device comprising:

extracting structured semantic knowledge from text associated with images in training data using natural language processing by the at least one computing device, the structured semantic knowledge describing text features;

training a model using the structured semantic knowledge and the images from the training data as part of machine learning by the at least one computing device;

forming a structured image representation of the input image, by the at least one computing device, that explicitly correlates at least part of the text features of the training data with image features of the input image; and

locating the input image as part of an image search by the at least one computing device, using the structured image representation.

20. The method of claim 19, wherein the text is a caption.