Paraphrasing the web by search-based data collection

• US 20080172378A1
• Filed: 03/16/2007
• Published: 07/17/2008
• Est. Priority Date: 01/11/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dolan, William B.
• Granted Patent: US 8,244,521 B2
• Time in Patent Office: Days
• US Class Current: 707/5
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

String-oriented web queries are utilized as a tool to examine the fabric of how words, phrases and/or n-grams alternate in a language. This fabric is exploited in order to build up a matrix of semantically equivalent pieces of language. In one embodiment, the Distributional Hypothesis is utilized, along with strategies for confirming synonymy, to systematically build up a picture of what words/phrases can be legitimately substituted for one another.

Claims

1. A method for supporting a language processing application, the method comprising:

obtaining a target item of text;

utilizing an index search procedure to identify a distributionally similar item of text;

determining, based on the results of one or more index queries, whether the distributionally similar item of text is semantically equivalent to the target item of text; and

if the distributionally similar item is semantically equivalent to the target item of text, then utilizing the distributionally similar item as a substitute for the target item of text within the language processing application.

2. The method of claim 1, wherein obtaining a target item of text comprises identifying n-gram components of a target phrase.

3. The method of claim 1, wherein utilizing an index search procedure comprises utilizing an index query engine to execute queries.

4. The method of claim 1, wherein utilizing an index search procedure comprises utilizing an index query engine to execute a query with a wildcard substituted for the target item of text.

5. The method of claim 1, wherein utilizing an index search procedure comprises:

executing a query that incorporates the target item of text, wherein the query leads to a collection of query results;

based on the query results, identifying left and/or right contexts associated with the target item of text; and

executing a query that incorporates at least one of the left and right contexts.

6. The method of claim 5, wherein executing a query that incorporates at least one of the left and right contexts further comprises executing a query that does not include the target item of text.

7. The method of claim 5, wherein executing a query that incorporates at least one of the left and right contexts further comprises executing a query that includes at least one wildcard in place of the target item of text.

8. The method of claim 1, wherein utilizing an index search procedure to identify a distributionally similar item of text comprises utilizing an index query engine to perform at least one world-wide-web query.

9. The method of claim 1, wherein determining comprises utilizing an index query engine to perform at least one world-wide-web query.

10. The method of claim 1, wherein determining (208) comprises determining based on right and/or left contexts as reflected in a search result set produced in response to the processing of a query that incorporates the distributionally similar item of text.

11. The method of claim 10, wherein determining comprises determining by comparing right and/or left contexts associated with the target item of texts with right and/or left contexts associated with the distributionally similar item of text.

12. The method of claim 1, wherein determining comprises executing at least one index query configured to test a heuristic rule.

13. The method of claim 12, executing at least one index query configured to test a heuristic rule comprises executing at least one index query to determine whether their is a coordination pattern.

14. The method of claim 12, executing at least one index query configured to test a heuristic rule comprises executing at least one index query to determine whether their is a coordination pattern indicative of a morphological alteration.

15. The method of claim 12, executing at least one index query configured to test a heuristic rule comprises executing at least one index query to determine whether their is a negative coordination pattern.

16. The method of claim 12, executing at least one index query configured to test a heuristic rule comprises executing at least one index query to determine whether their is an explicit synonym relationship.

17. The method of claim 12, executing at least one index query configured to test a heuristic rule comprises executing at least one index query to determine whether their is significant co-occurrence.

18. A language processing system, comprising:

an index query engine;

a processing component configured to determine, based on the results of one or more queries executed by the index query engine, whether an item of text is semantically equivalent to the target item of text.

19. The system of claim 18, wherein the processing component is further configured to determine, based on the results of one or more queries executed by the search engine, whether a target item of text is distributionally similar to the item of text.

20. A method for supporting a language processing application, the method comprising:

obtaining a target item of text;

determining, based on the results of one or more index queries, whether the target item of text is semantically equivalent to another item of text; and

if said another item is semantically equivalent to the target item of text, then utilizing said another item as a substitute for the target item of text within the language processing application.