Matching text to images

• US 8,503,769 B2
• Filed: 12/28/2010
• Issued: 08/06/2013
• Est. Priority Date: 12/28/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lin, Dahua, Kannan, Anitha, Ke, Qifa, Baker, Simon
• Primary Examiner(s): CHAWAN, SHEELA C
• Application Number:
  US12/979,375
• Publication Number:
  US 20120163707A1
• Time in Patent Office: 952 Days
• Field of Search: 382/159, 382/217, 382/176, 382/310, 382/229, 382/174, 382/178, 382/180, 382/171, 382/226, 382/177, 382/224, 715/202, 715/210, 715/209, 715/205, 715/234, 715/854, 715/757, 704/270, 345/419, 345/473, 345/420, 707/E17.008, 707/E17.013, 707/999.01, 707/E17.009, 707/956, 707/E17.01, 707/E17.037, 707/999.107, 707/999.104, 707/E17.005, 707/999.005, 707/E17.108, 707/785, 707/918, 707/E17.012, 707/999.2, 707/704
• US Class Current: 382/159
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables

Abstract

Text in web pages or other text documents may be classified based on the images or other objects within the webpage. A system for identifying and classifying text related to an object may identify one or more web pages containing the image or similar images, determine topics from the text of the document, and develop a set of training phrases for a classifier. The classifier may be trained and then used to analyze the text in the documents. The training set may include both positive examples and negative examples of text taken from the set of documents. A positive example may include captions or other elements directly associated with the object, while negative examples may include text taken from the documents, but from a large distance from the object. In some cases, the system may iterate on the classification process to refine the results.

Claims

1. A method performed on at least one computer processor, said method comprising:

receiving an image to classify, said image being located within a text document;

identifying a plurality of said text documents comprising said image;

identifying a training set of examples from at least one of said text documents, said training set of examples comprising a subset of text within said text documents;

training a classifier using said training set;

classifying said text within said text document using said classifier to identify a group of text associated with said image; and

for each of said plurality of said text documents, analyzing text in said text documents to determine a list of topics within said text document, said topics being used to select at least a portion of said training set.

2. The method of claim 1, said text documents comprising at least one Hyper Text Markup Language (HTML) document.

3. The method of claim 1 further comprising:

processing at least one of said text documents to identify a document object model and selecting at least one member of said training set using said document object model, said document object model defining a set of nodes based on objects within said text document.

4. The method of claim 3, said at least one member being selected that is more than a predefined number of nodes from said image.

5. The method of claim 4, said at least one member being a negative example within said training set.

6. The method of claim 3 further comprising:

identifying a text object that is less than a predefined number of nodes from said image.

7. The method of claim 6, said at least one member being a positive example within said training set.

8. The method of claim 7, said text object being a caption for said image.

9. The method of claim 1 further comprising:

iterating on said method by performing said training said classifier, and said classifying at least two times.

10. The method of claim 9, said iterating further comprising performing said identifying a training set of examples.

11. A system comprising:

a computer processor;

an object classifier operable on said processor, said object classifier that;

receives a set of text documents and identifies a common object to classify, said common object being comprised in each of said text documents;

identifies a training set of examples from at least one of said text documents, said training set of examples comprising a subset of text within said text documents;

trains a classifier using said training set; and

classifies said text within said text document using said classifier to identify a group of text associated with said object; and

a document object model generator that processes at least one of said text documents to identify a document object model, said document object model defining a set of nodes based on objects within said text document, said document object model being used to select at least one member of said training set.

12. The system of claim 11, said training set comprising at least one positive example and at least one negative example.

13. The system of claim 12, said at least one positive example comprising a text object related to said object.

14. The system of claim 11, said object classifier that further:

identifies a plurality of captions for said object from said set of text documents;

compares said plurality of captions to identify a first caption being an uninteresting caption and removing said first caption from consideration as a positive example for said training set.

15. The system of claim 11, at least some of said text documents being Hyper Text Markup Language (HTML) documents.

16. The system of claim 11, said classifier being a binary classifier that returns a binary result indicating either related or not related.

17. A method performed on at least one computer processor, said method comprising:

receiving an image to classify, said image being located within a first web page;

identifying a plurality of web pages comprising said image by transmitting a search request to a search system and returning said plurality of said web pages, the first web page included in said plurality of web pages;

identifying a training set of examples from at least one of said web pages,

said training set of examples comprising a subset of text within said plurality of web pages, said training set comprising at least one positive example and at least one negative example;

training a classifier using said training set, said classifier being a binary classifier that returns a binary result indicating either related or not related;

classifying said text within said plurality of web pages using said classifier to identify a group of text associated with said image;

creating a document object model; and

using said document object model to select at least one member of said training set.

18. The method of claim 17 further comprising:

using said document object model to select said at least one negative example.

19. The method of claim 18 further comprising:

selecting said at least one positive example as a caption for said image.

20. The method of claim 17 further comprising:

for each of said plurality of web pages, analyzing text in said each web page to determine a list of topics within said each web page, said topics being used to select at least a portion of said training set.