Content data indexing and result ranking

• US 7,987,189 B2
• Filed: 08/20/2007
• Issued: 07/26/2011
• Est. Priority Date: 07/01/2002
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Jayanti, Harish, Anderson, Christopher Walter
• Primary Examiner(s): Wassum; Luke S
• Application Number:
  US11/841,332
• Publication Number:
  US 20070282831A1
• Time in Patent Office: 1,436 Days
• Field of Search: None
• US Class Current: 707/741
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/917 Text
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure

Abstract

A full text indexing system is provided for processing content associated with data applications such as encyclopedia and dictionary applications. A build process collects data from various sources, processes the data into constituent parts, including alternative word sets, and stores the constituent parts in structured database tables. A run-time process is used to query the database tables and the results in order to provide effective matches in an efficient manner. Run-time processing is optimized by preprocessing all steps that are query-independent during the build process. A double word table representing all possible word pair combinations for each index entry and an alternative word table are used to further optimize runtime processing.

Claims

1. In a computing system having access to multiple content entities, each content entity including searchable content, a method for building a searchable content index for searching and retrieving content entities in an efficient manner that returns results of content entities expected to be found, the method comprising:

identifying searchable data within each of a plurality of content entities;

dividing text portions of the searchable data within each of the plurality of content entities into words and tokens, and storing each of the words and tokens in a first table;

removing from the first table each duplicate word and token;

applying an alternative word set to the first table after each duplicate word and token has been removed, wherein applying the alternative word set to the table includes adding to the first table alternative words associated with one or more of the words or tokens in the first table;

identifying all possible combinations of the words from the plurality of content entities; and

creating a second table, wherein the second table is a double word table that includes only all possible unique two word combinations of words from the plurality of content entities.

2. A method as recited in claim 1, wherein identifying the searchable data comprises:

using a content management system to identify a data source.

3. A method as recited in claim 2, wherein the data source is an encyclopedic data source.

4. A method as recited in claim 1, wherein each token comprises at least two words representing words commonly found together.

5. A method as recited in claim 1, wherein dividing text portions of the searchable data within each of the plurality of content entities into words and tokens is performed by a parsing module incorporating a natural language parser.

6. A method as recited in claim 1, wherein applying an alternative word set to the table after each duplicate word and token has been removed comprises:

identifying alternative words for words and tokens in the table.

7. A method as recited in claim 6, wherein the identified alternative words include synonyms of the words and tokens in the table.

8. A method as recited in claim 6, wherein the identified alternative words include common misspellings of the words and tokens in the table.

9. A method as recited in claim 6, wherein the identified alternative words include common related phrases of the words and tokens in the table.

10. A method as recited in claim 1, wherein applying the alternative word set to the table includes:

within the table, creating an alt word table; and

identifying one or more associations between different content entities and, separate from the alt word table, creating in the table one or more associations between the different content entities.

11. In a computing system having access to multiple content entities, each content entity including searchable content, a method for performing a run-time search of the searchable content to identify a ranked list of relevant content entities, the method comprising:

receiving, from a user of a client application, a query that includes one or more target search terms, the one or more target search terms being provided in a natural word format;

translating the query received from the user into a database query conducive to a known architecture of a database associated with searchable content, wherein the database includes at least a double word table, the double word table including only all possible unique two word combinations of words from the searchable content;

querying the database by comparing the database query to the database, thereby generating a list of content entities that are potential matches;

ranking the content entities in the list in descending order, based on a calculated likelihood that a particular entity is a target of the query from the user;

removing from the ranking any content entities that are duplicates; and

returning, to the client application, a list of content entities with the highest ranking.

12. A method as recited in claim 11, wherein the client application is an encyclopedia application.

13. A method as recited in claim 11, wherein the client application is a dictionary application.

14. A method as recited in claim 11, wherein the database includes a search index table associated with searchable content.

15. A method as recited in claim 14, wherein the search index table includes:

a search content table listing all content entities that have been indexed; and

a search content word table listing all words for each entry in the search content table, in a predefined order and without duplicates.

16. A method as recited in claim 15, wherein the search index table further includes:

a search word table listing all unique words and excluding all stop words.

17. A method as recited in claim 16, wherein the search word table includes an alt word table listing words similar to one or more words or tokens in target search terms.

18. A method as recited in claim 11, wherein returning, to the client application, a list of content entities with the highest ranking comprises:

returning to the client a list of only those content entities with a ranking above a predetermined threshold value.

19. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, cause a computing system to perform a method for performing a run-time search of the searchable content to identify a ranked list of relevant content entities, the method comprising:

receiving, from a user of a client application, a query that includes one or more target search terms, the one or more target search terms being provided in a natural word format;

translating the query received from the user into a database query conducive to a known architecture of a database associated with searchable content, wherein the database includes at least a double word table, the double word table including only all possible unique two word combinations of words from the searchable content;

querying the database by comparing the database query to the database, thereby generating a list of content entities that are potential matches;

ranking the content entities in the list in descending order, based on a calculated likelihood that a particular entity is a target of the query from the user;

removing from the ranking any content entities that are duplicates; and

returning, to the client application, a list of content entities with the highest ranking.