Field weighting in text searching

• US 20050210006A1
• Filed: 03/18/2004
• Published: 09/22/2005
• Est. Priority Date: 03/18/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Taylor, Michael, Robertson, Stephen, Zaragoza, Hugo, Petriuc, Mihai, Larimore, Stefan Isbein
• Granted Patent: US 7,584,221 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/38 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

A field-weighted search combines statistical information for each term across document fields in a suitably weighted fashion. Both field-specific term frequencies and field and document lengths are considered to obtain a field-weighted document weight for each query term. Each field-weighted document weight can then be combined in order to generate a field-weighted document score that is responsive to the overall query.

Claims

1. A method of determining a field-weighted score for a document having multiple fields relative to a query having a plurality of query terms, the method comprising:

replicating each field of the document in accordance with a field weight corresponding to the field to produce an individual field set corresponding to each field in the document;

combining each field set for the document into a virtual document;

indexing the virtual document to produce a virtual document statistics; and

computing the field-weighted score from the virtual document index based on the query.

2. The method of claim 1 wherein the query is associated with a search and the field-weighted score represents a level of relevance of the document to the query.

3. The method of claim 1 wherein each field weight is represented by an integer value and the replicating operation comprises:

generating each field set to include a number of copies of a field of the document, wherein the number of copies equals the integer value.

4. The method of claim 1 wherein the replicating operation comprises:

concatenating copies of one of the fields into a field set.

5. The method of claim 1 wherein the combining operation comprises:

concatenating each field set into the virtual document.

6. The method of claim 1 wherein the computing operation comprises:

computing a field-weighted document weight for each query term in the query from the virtual document statistics.

7. The method of claim 1 wherein the computing operation comprises:

computing a field-weighted document weight for each query term in the query from the virtual document statistics; and

computing the field-weighted score based on the field-weighted document weight for each query term.

8. The method of claim 1 further comprising:

ranking the field-weighted score with field-weighted scores of other documents.

9. A computer program product encoding a computer program for executing on a computer system a computer process for determining a field-weighted score for a document having multiple fields relative to a query having a plurality of query terms, the computer process comprising:

replicating each field of the document in accordance with a field weight corresponding to the field to produce an individual field set corresponding to each field in the document;

combining each field set for the document into a virtual document;

indexing the virtual document to produce a virtual document statistics; and

computing the field-weighted score from the virtual document index based on the query.

10. The computer program product of claim 9 wherein the query is associated with a search and the field-weighted score represents a level of relevance of the document to the query.

11. The computer program product of claim 9 wherein each field weight is represented by an integer value and the replicating operation comprises:

generating each field set to include a number of copies of a field of the document, wherein the number of copies equals the integer value.

12. The computer program product of claim 9 wherein the replicating operation comprises:

concatenating copies of one of the fields into a field set.

13. The computer program product of claim 9 wherein the combining operation comprises:

concatenating each field set into the virtual document.

14. The computer program product of claim 9 wherein the computing operation comprises:

computing a field-weighted document weight for each query term in the query from the virtual document statistics.

15. The computer program product of claim 9 wherein the computing operation comprises:

computing a field-weighted document weight for each query term in the query from the virtual document statistics; and

computing the field-weighted score based on the field-weighted document weight for each query term.

16. The computer program product of claim 9 further comprising:

ranking the field-weighted score with field-weighted scores of other documents.

17. A method of determining a field-weighted score for a document having multiple fields relative to a query having a plurality of query terms, the method comprising:

determining a field-specific term frequency for each field in the document for each query term;

weighting each field-specific term frequency according to a field weight designated for the corresponding field to compute a field-weighted term frequency for each query term;

computing a field-weighted document weight for each query term based on the field-weighted term frequency for each query term; and

computing the field-weighted score as a function of the field-weighted document weight of all query terms.

18. The method of claim 17 wherein the query is associated with a search and the field-weighted score represents a level of relevance of the document to the query.

19. The method of claim 17 further comprising:

computing a field-weighted document length based on a field weight for each field and a field length for each field, wherein the operation of computing a field-weighted document weight comprises computing a field-weighted document weight for each query term based on the field-weight term frequency for each query term and the field-weighted document length.

20. The method of claim 17 further comprising:

computing a field-weighted document length based on a field weight for each field and a field length for each field by summing one or more weighted field lengths of the fields in the document, each weighted field length being a field length weighted by a corresponding field weight.

21. The method of claim 17 further comprising:

computing a field-weighted document length based on a field weight for each field and a field length for each field by summing one or more weighted field lengths of the fields in the document, each weighted field length being a field length weighted by a corresponding field weight, wherein the operation of computing a field-weighted document weight comprises computing a field-weighted document weight for each query term based on the field-weight term frequency for each query term and the field-weighted document length.

22. The method of claim 17 wherein the determining operation comprises:

determining the field-specific term frequency for each field from document statistics associated with the document, the document statistics including a field-weighted term frequency for at least one query term in the document.

23. The method of claim 17 wherein the determining operation comprises:

determining the field length for each field from document statistics associated with the document.

24. The method of claim 17 wherein the operation of computing a field-weighted document weight comprises:

summing one or more weighted field-specific term frequencies of the fields in the document.

25. The method of claim 17 further comprising:

ranking the field-weighted score with field-weighted scores of other documents.

26. A computer program product encoding a computer program for executing on a computer system a computer process for determining a field-weighted score for a document having multiple fields relative to a query having a plurality of terms, the computer process comprising:

determining a field-specific term frequency for each field in the document for each query term;

weighting each field-specific term frequency according to a field weight designated for the corresponding field to compute a field-weighted term frequency for each query term;

computing a field-weighted document weight for each query term based on the field-weighted term frequency for each query term; and

computing the field-weighted score as a function of the field-weighted document weight of all query terms.

27. The computer program product of claim 26 wherein the query is associated with a search and the field-weighted score represents a level of relevance of the document to the query.

28. The computer program product of claim 26 wherein the computer process further comprises:

computing a field-weighted document length based on a field weight for each field and a field length for each field, wherein the operation of computing a field-weighted document weight comprises computing a field-weighted document weight for each query term based on the field-weight term frequency for each query term and the field-weighted document length.

29. The computer program product of claim 26 wherein the computer process further comprises:

computing a field-weighted document length based on a field weight for each field and a field length for each field by summing one or more weighted field lengths of the fields in the document, each weighted field length being a field length weighted by a corresponding field weight.

30. The computer program product of claim 26 wherein the computer process further comprises:

computing a field-weighted document length based on a field weight for each field and a field length for each field by summing one or more weighted field lengths of the fields in the document, each weighted field length being a field length weighted by a corresponding field weight, wherein the operation of computing a field-weighted document weight comprises computing a field-weighted document weight for each query term based on the field-weight term frequency for each query term and the field-weighted document length.

31. The computer program product of claim 26 wherein the determining operation comprises:

determining the field-specific term frequency for each field from document statistics associated with the document, the document statistics including a field-weighted term frequency for at least one query term in the document.

32. The computer program product of claim 26 wherein the determining operation comprises:

determining the field length for each field from document statistics associated with the document.

33. The computer program product of claim 26 wherein the operation of computing a field-weighted document weight comprises:

summing one or more weighted field-specific term frequencies of the fields in the document.

34. The computer program product of claim 26 further comprising:

ranking the field-weighted score with field-weighted scores of other documents.

35. A system for determining a field-weighted score for a document having multiple fields relative to a query having a plurality of terms, the system comprising:

a field-weighted term frequency calculator that determines a field-specific term frequency for each field in the document for each query term and weights each field-specific term frequency according to a field weight identified for the corresponding field to compute a field-weighted term frequency for each query term;

a field-weighted document weight calculator that computes a field-weighted document weight for each query term based on the field-specific term frequency for each query term; and

a document score calculator that computes the field-weighted score as a function of the field-weighted document weight of all query terms.

36. The system of claim 35 wherein the query is associated with a search and the field-weighted score represents a level of relevance of the document to the query.

37. The system of claim 35 further comprising:

a field-weighted document length calculator that computes a field-weighted document length based on a field weight for each field and a field length for each field, wherein the field-weighted document weight calculator computes a field-weighted document weight for each query term based on the field-weight term frequency for each query term and the field-weighted document length.

38. A method of determining a field-weighted score for a document having multiple fields relative to a query having a plurality of query terms, the method comprising:

computing a field-weighted term frequency for each query term based on field weights designated for individual fields in the document;

computing a field-weighted document weight for each query term based on the field-weighted term frequency for each field in the document; and

computing the field-weight score as a function of the field-weighted document weights of the query terms.

39. The method of claim 38 further comprising:

computing a field-weighted document length based on a field weight for each field and a field length for each field, wherein the operation of computing a field-weighted document weight comprises computing a field-weighted document weight for each query term based on the field-weight term frequency for each query term and the field-weighted document length.

40. The method of claim 38 wherein computing a field-weighted document weight comprises:

computing the field-weighted document weight using a field-weighted free parameter of a BM25 document weighting function, the field-weighted free parameter being based on a corresponding optimized free parameter computed in a non-field-weighted configuration.

41. The method of claim 38 wherein computing a field-weighted document weight comprises:

computing the field-weighted document weight using a field-weighted free parameter of a BM25 document weighting function, the field-weighted free parameter being based on an average term frequency over all terms in a non-field-weighted configuration, an average term frequency over all terms in a field-weighted configuration, and a corresponding optimized free parameter computed in the non-field-weighted configuration.

42. The method of claim 38 wherein computing a field-weighted document weight comprises:

computing the field-weighted document weight using a factor reflecting a dependence on a number of the fields in the document in which a query term occurs.

43. The method of claim 38 wherein computing a field-weighted score comprises:

computing the field-weighted score using a factor reflecting a dependence on which field in the document includes the most query terms.

44. A computer program product encoding a computer program for executing on a computer system a computer process for determining a field-weighted score for a document having multiple fields relative to a query having a plurality of query terms, the computer process comprising:

computing a field-weighted term frequency for each query term based on field weights designated for individual fields in the document;

computing a field-weighted document weight for each query term based on the field-weighted term frequency for each field in the document; and

computing the field-weight score as a function of the field-weighted document weights of the query terms.

45. The computer program product of claim 44 wherein the computer process further comprises:

computing a field-weighted document length based on a field weight for each field and a field length for each field, wherein the operation of computing a field-weighted document weight comprises computing a field-weighted document weight for each query term based on the field-weight term frequency for each query term and the field-weighted document length.

46. The computer program product of claim 44 wherein computing a field-weighted document weight comprises:

computing the field-weighted document weight using a field-weighted free parameter of a BM25 document weighting function, the field-weighted free parameter being based on a corresponding optimized free parameter computed in a non-field-weighted configuration.

47. The computer program product of claim 44 wherein computing a field-weighted document weight comprises:

computing the field-weighted document weight using a field-weighted free parameter of a BM25 document weighting function, the field-weighted free parameter being based on an average term frequency over all terms in a non-field-weighted configuration, an average term frequency over all terms in a field-weighted configuration, and a corresponding optimized free parameter computed in the non-field-weighted configuration.

48. The computer program product of claim 44 wherein computing a field-weighted document weight comprises:

computing the field-weighted document weight using a factor reflecting a dependence on a number of the fields in the document in which a query term occurs.

49. The computer program product of claim 44 wherein computing a field-weighted score comprises:

computing the field-weighted score using a factor reflecting a dependence on which field in the document includes the most query terms.

50. A system for determining a field-weighted score for a document having multiple fields relative to a query having a plurality of query terms, the system comprising:

a field-weighted term frequency calculator that computes a field-weighted term frequency for each query term based on field weights designated for individual fields in the document;

a field-weighted document weight calculator that computes a field-weighted document weight for each query term based on the field-weighted term frequency for each field in the document; and

a search engine that computes the field-weighted score as a function of the field-weighted document weights of the query terms.

51. The system of claim 50 further comprising:

a field-weighted document length calculator that computes a field-weighted document length based on a field weight for each field and a field length for each field, wherein the field-weighted document weight calculator computes a field-weighted document weight for each query term based on the field-weight term frequency for each query term and the field-weighted document length.