DOCUMENT ANALYSIS AND ASSOCIATION SYSTEM AND METHOD

• US 20120278341A1
• Filed: 09/24/2010
• Published: 11/01/2012
• Est. Priority Date: 09/26/2009
• Status: Active Grant

• Current Assignee: Sajari Pty Ltd. (Algolia, Inc.)
• Original Assignee: Sajari Pty Ltd. (Algolia, Inc.)
• Inventors: ogilvy, Hamish, Prime, Owen James, Burns, Phillip Anthony
• Granted Patent: US 8,666,994 B2
• Time in Patent Office: Days
• US Class Current: 707/749
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06Q 10/00 Administration; Management

Abstract

Methods and systems for indexing a plurality of documents, each document comprising a text portion, the method parses the text portion of each of the plurality of documents to form a plurality of respective local document indexes each associated with a respective document, and stores the local document index in a database, Each local document index has a plurality of local text terms and a local weighting associated with each text term From the plurality of local document indexes, forming a global document index associated with each global text term. The global weighting is determined with respect to a parameter associated with a reference global text term. Also, methods and systems for analyzing a text portion, retrieving documents from a database relevant to the text portion and for refining the results of a search are disclosed.

Claims

1. A method for indexing a plurality of documents, each document comprising a text portion, the method comprising:

a) parsing the text portion of each of the plurality of documents to form a plurality of respective local document indexes each associated with a respective document, and storing the local document index in a database, wherein each local document index comprises a plurality of local text terms contained in the respective document and a local weighting associated with each text term;

b) from the plurality of local document indexes, forming a global document index comprising a plurality of global text terms contained in the plurality of documents, and a global weighting associated with each global text term;

wherein the global weighting associated with each of the global text terms is determined with respect to a parameter associated with a reference global text term.

2. A method as claimed in claim 1 wherein the global weighting associated with each of the global text terms is further determined with respect to the number of documents in which each global text term appears across all the plurality of documents.

3. A method as claimed in claim 1 or claim 2 wherein the global weighting associated with each of the global text terms is determined with respect to the number of documents in which the reference text term appears.

4. A method as claimed in any one of claims 1 to 3 wherein the global weighting associated with each of the global text terms is further determined with respect to user interactions.

5. A method as claimed in claim 4 wherein the weighting associated with each text term comprises a combination of a plurality of weightings, each associated with each global text term.

6. A method as claimed in claim 5 wherein one or the plurality of weightings are selected from the group of:

the number of times the term appears in a single document;

the number of times the term appears in all the plurality of documents;

the position of the text term in a document;

the capitalisation of the term;

punctuation surrounding the term;

words in the text portion adjacent to the term;

word rarity;

word sequence;

combinations of text terms;

or the number of words in each text term;

or a user-defined weighting.

7. A method as claimed in either claim 5 or claim 6 wherein the weighting may be a positive weighting or a negative weighting, or selected from a graduated scale of weightings ranging from positive to negative.

8. A method as claimed in claim 7 wherein where one or more of the plurality of weightings is a negative weighting for a selected global text term, the selected global text term is assigned a zero weighting.

9. A method as claimed in claim 6 wherein the user-defined weighting is derived from self-learning system comprising a plurality of user-defined weightings for a selected global text term.

10. A system for indexing a plurality of documents, each document comprising a text portion, the system comprising:

a parsing module for parsing the text portion of each of the plurality of document to form a plurality of respective local document indexes each associated with a respective document, wherein each local document index comprises a plurality of local text terms contained in the respective document and a local weighting associated with each text term;

a database adapted for storing each of the local document indexes in a memory;

a processor for analysing the plurality of local document indexes and forming a global document index from the plurality of local document indexes, the global document index comprising a plurality of global text terms contained in the plurality of documents, and a global weighting associated with each global text term;

wherein the global weighting associated with each of the global text terms is determined with respect to a parameter associated with a reference global text term; and

wherein the global document index is stored in the database and related to each of the local document indexes.

11. A method for analysing a text portion and retrieving documents relevant to the text portion, to the method comprising:

a) receiving an input comprising an input text portion;

b) identify at least one text term in the text portion;

c) assigning at least one weight associated with the at least one text term;

d) forming an input local index of the at least one text term and at least one associated local term weight, wherein the at least one associated local term weight is determined with reference to a global term index stored in a database, the global term index comprising a plurality of global text terms and associated global text term weights, and being formed from a plurality of reference documents, wherein a representation of each of the reference documents is stored in the database;

e) querying the database to identify one or more of the reference documents of relevance with respect to the input text portion; and

f) outputting a representation of the identified relevant reference documents.

12. A method as claimed in claim 11 wherein the representation of each of the plurality of reference documents stored in the database comprises either the reference document or a link thereto, and the representation further comprises a respective local reference term index for each reference document.

13. A method as claimed in claim 11 wherein the representation of each of the plurality of reference documents stored in a database comprises a representative text string derived from the text portion of each reference document and a respective local reference term index.

14. A method as claimed in any one of claims 11 to 13 wherein a plurality of text terms are identified in the text portion, each of the plurality of text terms being assigned at least one associated local term weight determined with reference to the global term index.

15. A method as claimed in any one of claims 11 to 14 wherein step (11.b) comprises parsing of the text portion to identify the at least one or plurality of text terms.

16. A method as claimed in any one of claims 11 to 15 wherein in step (11.e) the relevant reference documents are determined from a comparison of the input local index with each of the plurality of reference local indexes associated with each respective reference document.

17. A method as claimed in any one of claims 12 to 15 wherein in step (11.e) the relevant reference documents are determined from an intersection of the at least one or plurality of text terms of the input text portion with one or more terms in the local reference term index associated with each reference document.

18. A method as claimed in either claim 11 or claim 14 wherein the at least one or plurality of text terms comprise single word terms within the input text portion.

19. A method as claimed in any one of claim 11, 14 or 18 wherein the at least one or plurality of text terms comprise double-word terms within the input text portion.

20. A method as claimed in either claim 11, 14, 18 or 19 wherein the at least one or plurality of text terms comprise triple-word terms within the input text portion.

21. A method as claimed in any one of claim 11, 14, 18, 19 or 20 wherein the local weights are assigned to each of the at least one or plurality of terms in accordance one or more parameters selected from the group of:

word rarity;

punctuation;

capitalisation;

word sequence;

conibinations of terms;

or the number of words in each term.

22. A method as claimed in claim 11, wherein the representation of the identified relevant reference documents comprises a representative text string derived from the text of each of the identified relevant reference documents.

23. A method as claimed in claim 22 wherein the representative text string from each document comprises a selected number of text words before and/or after one or more selected relevant text terms with significant weights.

24. A method as claimed in claim 11 further comprising the steps of:

g) displaying the relevant reference documents on a user interface, the user interface comprising input means for receiving user input with respect to each of the displayed reference documents;

h) accepting user-input with respect to one or more of the displayed documents;

i) re-forming the input local term index on the basis of the user input;

j) on the basis of the re-formed input local term index, querying the database to identify one or more relevant reference documents of enhanced relevance to the input text portion; and

k) outputting a representation of the further identified reference documents of enhanced relevance.

25. A method as claimed in claim 24 wherein in step (24.i), re-forming the input local term index comprises:

i.1) re-assigning the input local text term weights of the input text terms which also appear in each of the reference documents for which user-determined input is received; and

wherein step (24.j) comprises;

j.1) on the basis of the re-assigned input local text term weights, querying the database to identify one or more relevant reference documents of enhanced relevance to the input text portion.

26. A method as claimed in claim 24 wherein in step (24.g) the user input means is a means for assigning positive and negative relevance weights with respect to each displayed reference document.

27. A method as claimed in claim 24 further comprising repeating steps (24.h) to (24.k) thereby identifying and outputting one or more relevant further documents of increased enhanced relevance in respect of the relevant text terms in the text portion.

28. A method as claimed in claim 24 further comprising repeating steps (24.h) to (24.k) on the reference documents with enhanced relevance to identify and output reference documents with additional enhancement of relevance.

29. A method as claimed in claim 24 wherein the additional relevance information comprises either a positive indication of relevance of a particular document or a negative indication of relevance of a particular document.

30. A method as claimed in claim 29 wherein for each reference document for which a positive indication of relevance is received, the associated weighting of each of the input text terms in the input local index which also appear in the local text term index of the positively-identified reference document is increased by a predetermined amount.

31. A method as claimed in claim 29 wherein for each reference document for which a negative indication of relevance is received, the associated weighting of each of the input text terms in the input local index which also appear in the local text term index of the negatively-identified reference document is decreased by a predetermined amount.

32. A method as claimed in either claim 29 or 30 wherein the predetermined amount may be a multiplier applied to the index term weight.

33. A method as claimed in claim 32 wherein the multiplier may be zero such that a selected term has no relevance to subsequent interactions.

34. A method as claimed in claim 29 wherein where a selected text term appears in one or documents which receive a positive indication of relevance, and the selected text term also appears in one or documents which receive a negative indication of relevance, the associated weighting of the selected text term in the input local index is updated based on a combination of the positive and negative indications.

35. A method as claimed in claim 34 wherein where the selected text term appears in one or documents which receive a positive indication of relevance, and the selected text term also appears in an equal number of documents which receive a negative indication of relevance, the associated weighting of the selected text term in the input local index is unchanged.

36. A method as claimed in claim 30 wherein in step (24.i)), re-forming the input local term index comprises:

i.2) forming an augmented input local term index on the basis of text terms in the local term index of documents receiving a positive indication of relevance; and

wherein step (24.j) comprises;

j.2) on the basis of the input local text term weights in the augmented input local text term index, querying the database to identify one or more relevant reference documents of enhanced relevance to the input text portion.

37. A method as claimed in claim 36 wherein for each reference document for which a positive indication of relevance is received, terms in the positively identified reference document which do not appear in the input local term index are added thereto to form the augmented local text term index together with associated local index text term weights determined.

38. A method as claimed in any one, of claims 11 to 37 wherein the text portion is a text string comprising a plurality of text words.

39. A method as claimed in any one of claims 11 to 37 wherein the text portion is a text document.

40. A method as claimed in any one of claims 11 to 37 wherein each of the reference documents is assigned a document relevance score representative of the relevance to the input text portion.

41. A method as claimed in any one of claims 11 to 37 wherein the reference documents are text documents representative of a document selected from one or more of the group of:

a book;

an article;

a text record;

a certificate;

an agreement;

a contract;

a manuscript;

a paper;

a scientific paper;

a resume;

a patent specification;

an employment advertisement;

a legal transcript;

a legal document;

or a news report.

42. A method as claimed in any one of claims 11 to 37 wherein the text portion is selected from one or more of the group of:

a text string comprising one or more words;

a text document;

a book;

an article;

a text record;

a resume;

a certificate;

an agreement;

a contract;

a manuscript;

a paper;

a scientific paper;

a patent specification;

a resume;

a curriculum vitae;

a legal transcript;

a legal document;

or a news report.

43. A method for refining the results of a search, the search results comprising a representation of a selected plurality of reference documents, such reference documents displayed being of relevance to an input text portion comprising one or more search terms, the selected plurality of reference documents comprising a subset of a plurality of documents in a database, the method comprising the steps of:

a) forming a local term index from the search terms, the local term index comprising one or more text terms, each local text term associated with a local text term weight;

b) receiving and displaying the search results on a user interface, the user interface comprising input means for receiving user input with respect to one or more of the plurality of the displayed reference documents;

c) accepting user input on one or more of the displayed reference documents;

d) re-forming the local term index on the basis of the user input;

e) on the basis of the re-formed input local term index, querying the database to identify one or more documents of enhanced relevance to the input text portion; and

f) outputting a representation of the further identified reference documents of enhanced relevance.

44. A method as claimed in claim 43 wherein in step (43.d), re-forming the input local term index comprises:

d.1) re-assigning the input local text term weights of the input text terms which also appear in each of the reference documents for which user-determined input is received; and

wherein step (43.e) comprises;

e.1) on the basis of the re-assigned input local text term weights, querying the database to identify one or more relevant reference documents of enhanced relevance to the input text portion.

45. A method as claimed in claim 43 wherein in step (43.a) the local text term weights for each of the local text terms are equal.

46. A method as claimed in claim 43 wherein in step (43.a) the local text term weights for each of the local text terms are derived from a global text term index, the global text term index comprising a plurality of text terms associated with global text term weights, wherein the global text term weights are derived from text term analysis of a plurality of documents.

47. A method as claimed in claim 43 wherein in step (43.d), re-forming the input local term index comprises:

d.2) forming an augmented input local term index on the basis of text terms in the local term index of documents receiving a positive indication of relevance; and

wherein step (43.e) comprises;

e.2) on the basis of the input local text term weights in the augmented input local text term index, querying the database to identify one or more relevant reference documents of enhanced relevance to the input text portion.

48. A method as claimed in claim 47 wherein for each reference document for which a positive indication of relevance is received, new terms in the positively identified reference document which do not appear in the local term index are added thereto to form the augmented local text term index and associated local index text term weights for the new terms are determined.

49. A method as claimed in either claim 47 or claim 48 wherein for each reference document for which a negative indication of relevance is received, terms in the negatively identified reference documents which do not appear in the local term index are subtracted therefrom to form the augmented local text term index.

50. A system for refining the results of a search, the search results comprising a representation of a selected plurality of documents of relevance to one or more search terms, the selected plurality of documents comprising a subset of a plurality of documents in a database, the system comprising:

means for forming a local term index from the search terms, the local term index comprising one or more text terms, each local text term associated with a local text term weight;

means for receiving and displaying the search results on a user interface, the user interface comprising input means for receiving user input with respect to each of the displayed reference documents;

user input means for accepting user input on one or more of the displayed documents;

processing means for analysing the user input re-forming the input local term index on the basis of the user input;

query means for querying the database on the basis of the re-formed input local term index to identify one or more documents of enhanced relevance to the input text portion; and

output means for outputting a representation of the further identified reference documents of enhanced relevance.

51. A system for analysing an input text portion and retrieving documents relevant to the text portion, the system comprising:

input means for receiving an input comprising an input text portion;

identification means to identify at least one text term in the text portion;

assignment means for assigning at least one weight associated with the at least one text term;

indexing means for forming an input local term index of the at least one text term and at least one associated local term weight, wherein the at least one associated local text term weights is determined with reference to a global term index stored in a database, the global term index comprising a plurality of global text terms and associated global text term weights, and being formed from a plurality of reference documents, wherein a representation of each of the reference documents is stored in the database;

query means for querying the database to identify one or more relevant reference documents with respect to the input text portion; and

output means for outputting a representation of the identified relevant reference documents.

52. A system as claimed in claim 51 wherein the representation of each of the plurality of reference documents stored in the database comprises either the reference document or a link thereto, and the representation further comprises a respective local reference term index for each reference document.

53. A system as claimed in claim 51 wherein the representation of each of the plurality of reference documents stored in a database comprises a representative text string derived from the text portion of each reference document and a respective local reference term index.

54. A system as claimed in any one of claims 51 to 53 wherein a plurality of text terms are identified in the text portion, each of the plurality of text terms being assigned at least one associated local term weight determined with reference to the global term index.

55. A system as claimed in claim 51 further comprising:

display means for displaying the relevant reference documents on a user interface, the user interface comprising input means for receiving user input with respect to each of the displayed reference documents;

user input means accepting user-input on one or more of the displayed documents;

processing means for analysing the user input and re-forming the input local text term index;

query means for querying the database on the basis of the re-formed input local text term index to identify one or more relevant reference documents of enhanced relevance to the input text portion; and

means for outputting a representation of the further identified reference documents of enhanced relevance.

56. A method as claimed in claim 55 wherein the re-forming of the input local term index comprises:

re-assigning the input local text term weights of the input text terms which also appear in each of the reference documents for which user-determined input is received; and

the querying of the database on the basis of the re-formed input local text term index comprises;

on the basis of the re-assigned input local text term weights, querying the database to identify one or more relevant reference documents of enhanced relevance to the input text portion.

57. A method as claimed in claim 55 wherein the re-forming of the input local term index comprises:

forming an augmented input local term index on the basis of text terms in the local term indexes of documents receiving a positive indication of relevance; and

the querying of the database on the basis of the re-formed input local text term index comprises;

on the basis of the input local text term weights in the augmented input local text term index, querying the database to identify one or more relevant reference documents of enhanced relevance to the input text portion.

58. A computer readable medium comprising a program for analysing a text portion and retrieving documents relevant to the text portion, said program controlling the operation of a data processing apparatus on which the program executes to perform the steps of:

a) receiving an input comprising an input text portion;

b) identify at least one text term in the text portion;

c) assigning at least one weight associated with the at least one text term;

d) forming an input local index of the at least one text term and at least one associated local term weight, wherein the at least one associated local term weight is determined with reference to a global term index stored in a database, the global term index comprising a plurality of global text terms and associated global text term weights, and being formed from a plurality of reference documents, wherein a representation of each of the reference documents is stored in the database

e) querying the database to identify one or more of the reference documents of relevance with respect to the input text portion; and

f) outputting a representation of the identified relevant reference documents.

59. A computer readable medium comprising a program according to claim 58, wherein the program executes to perform the further steps of:

g) displaying the relevant reference documents on a user interface, the user interface comprising input means for receiving user input with respect to each of the displayed reference documents;

h) accepting user-input with respect to one or more of the displayed documents;

i) re-forming the input local term index on the basis of the user input;

j) on the basis of the re-formed input local term index, querying the database to identify one or more relevant reference documents of enhanced relevance to the input text portion; and

k) outputting a representation of the further identified reference documents of enhanced relevance.

60. A computer readable medium comprising a program for refining the results of a search, the search results comprising a representation of a selected plurality of reference documents, such reference documents displayed being of relevance to an input text portion comprising one or more search terms, the selected plurality of documents comprising a subset of a plurality of documents in a database, said program controlling the operation of a data processing apparatus on which the program executes to perform the steps of:

a) forming a local term index from the search terms, the local term index comprising one or more text terms, each local text term associated with a local text term weight;

b) receiving and displaying the search results on a user interface, the user interface comprising input means for receiving user input with respect to one or more of the plurality of the displayed reference documents;

c) accepting user input on one or more of the displayed documents;

d) re-forming the input local term index on the basis of the user input;

e) on the basis of the re-formed input local term index, querying the database to identify one or more documents of enhanced relevance to the input text portion; and

f) outputting a representation of the further identified reference documents of enhanced relevance.