Information retrieval system

• US 20040044952A1
• Filed: 08/15/2003
• Published: 03/04/2004
• Est. Priority Date: 10/17/2000
• Status: Active Grant

• Current Assignee: Telstra Corporation Limited (Telstra Group Ltd.)
• Original Assignee: Telstra Corporation Limited (Telstra Group Ltd.)
• Inventors: Jiang, Jason, Raskutti, Bhavani Laxman, Starkie, Bradford Craig
• Granted Patent: US 8,793,261 B2
• Time in Patent Office: Days
• US Class Current: 715/500
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence

Abstract

An information retrieval system including a natural language parser (3) for parsing documents of a document space (1) to identify key terms of each document based on linguistic structure, and for parsing a search query to determine the search term, a feature extractor (4) for determining an importance score for terms of the document space (1) based on distribution of the terms in the document space (1), an index term generator (5) for generating index terms using the key terms identified by the parser (3) and the extractor (4) and having an importance score above a threshold level, and a query clarifier (16) for selecting from the index terms, on the basis of the search term, index terms for selecting at least one document from the document space (1). A speech recognition engine (12) is used to generate the query, and a bi-gram language module (6) generates grammar rules for the speech recognition engine (12) using the index terms.

Claims

1. A method of generating index terms for documents, including:

parsing the documents to identify key terms of each document based on sentence structure;

determining an importance score for terms of the documents based on distribution of said terms in said documents; and

retaining said key terms having an importance score above a predetermined threshold as said index terms.

2. A method as claimed in claim 1, wherein a natural language parser executes said parsing, and said key terms are linguistically important terms of the document.

3. A method as claimed in claim 2, wherein said parser generates key-centred phrase structure frames for sentences of said document, and generates at least one frame relation graph that is parsed to determine the frames representative of the sentences of said document, said frames including said key terms.

4. A method as claimed in claim 1, wherein a feature extractor executes said novelty score determining, and said distribution is represented by the distribution of n-grams over said documents.

5. A method as claimed in claim 4, wherein said feature extractor assigns a novelty score to each n-gram of each document based on the probability of the occurrence of the n-gram in the document and the probability of occurrence elsewhere in said documents.

6. A method as claimed in claim 5, wherein the novelty score is determined on the basis of

7. A method as claimed in claim 6, wherein the feature extractor assigns a weight to each word of said document based on the novelty score for component n-grams of said word.

8. A method as claimed in claim 7, wherein the feature extractor generates said importance score for terms on the basis of the weights for the words of the document.

9. A method as claimed in claim 1, including generating grammar rules, for a speech recognition engine, using said index terms.

10. A method as claimed in claim 9, including generating a list of phrases including said index terms and said importance scores for said terms, and inputting said list as a bi-gram array with said importance scores representing probabilities, to generate said grammar rules for said speech recognition engine.

16. A method as claimed in any one of claims 11 to 15, wherein said index terms are generated by a method as claimed in any one of claims 1 to 10.

17. A method as claimed in claim 16, wherein said parsing is executed by a natural language parser.

18. A method as claimed in claim 16, wherein said search query is spoken and converted into text data by a speech recognition engine.

19. A method as claimed in claim 18, including generating grammar rules for said speech recognition engine using said index terms.

20. A method as claimed in claim 18, including generating a list of phrases including said index terms and said importance scores for said terms, and inputting said list as a bi-gram array with said importance scores representing probabilities, to generate said grammar rules for said speech recognition engine.

21. An extraction system having components for executing a method as claimed in any one of claims 1 to 10.

23. An information retrieval system having an extraction system as claimed in claim 21 and a query response system as claimed in claim 22.

24. Information retrieval software having code for executing the steps of a method as claimed in any one of claims 1 to 20.

11. An information retrieval method, including:

parsing a search query to determine at least one search term;

matching said at least one search term against index terms of documents to determine matching documents;

ranking said matching documents according to fitness values of the index terms, of said matching documents, matched to said search terms; and

presenting said matching documents according to said ranking.

12. An information retrieval method as claimed in claim 11, wherein said matching includes selecting index terms of said documents that include said search term, and generating a fitness value for each selected index term on the basis of an importance score of words in a longest-common substring of the search term and the selected index term.

13. An information retrieval method as claimed in claim 12, wherein said matching includes using the selected index terms having a predetermined characteristic to determine said matching documents.

14. An information retrieval method as claimed in claim 13, wherein said predetermined characteristic is having said fitness value above a predetermined threshold.

15. An information retrieval method as claimed in claim 11, wherein said search term is determined from said query on the basis of weight values determined for terms of said query, said weight values representing respective linguistic and interpretation characteristics of said terms of said query.

22. A query response system having components for executing a method as claimed in any one of claims 11 to 15.

25. An information retrieval system, including:

a natural language parser for parsing documents of a document space to identify key terms of each document based on linguistic structure, and for parsing a search query to determine at least one search term;

a feature extractor for determining an importance score for terms of the document space based on distribution of said terms in said document space;

an index term generator for generating index terms using said key terms identified by said parser and said extractor and having an importance score above a threshold level; and

a query clarifier for selecting from said index terms, on the basis of said at least one search term, index terms for selecting at least one document from said document space.

26. An information retrieval system as claimed in claim 25, including a speech recognition engine for generating said query.

27. An information retrieval system as claimed in claim 26, including generating grammar for said speech recognition engine using said index terms.

28. An information retrieval system as claimed in claim 27, including generating a list of phrases including said index terms and said importance scores for said terms, and inputting said list as a bi-gram array with said importance scores representing probabilities, to generate said grammar for said speech recognition engine.

29. An information retrieval system as claimed in claim 25, wherein said query clarifier selects said index terms having a fitness value above a threshold level, said fitness value being generated on the basis of said importance score for terms in a longest common substring of the search term and the index term.