Electronic document repository management and access system

• US 20040093323A1
• Filed: 11/07/2002
• Published: 05/13/2004
• Est. Priority Date: 11/07/2002
• Status: Active Grant

• Current Assignee: Thomson Reuters Enterprise Centre GmbH (The Woodbridge Co. Ltd.)
• Original Assignee: Thomson Global Resources AG
• Inventors: Bluhm, Mark, Getting, Bruce, Walz, Shirley, Hayft, Mark
• Granted Patent: US 7,085,755 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A system and a method providing for the distribution and management of a large corpus of value added electronic documents while providing customized services to a plurality of diverse end users.

Claims

1. A system for maintaining a large aggregation of electronically stored documents and making them available to users that submit inquiry messages, comprising:

a. at least one data collection for storing a documents in electronic form, each document having a unique identifier;

b. an intake component for receiving new documents to be added to the at least one data collection;

c. an enrichment component associated with the intake component for processing a received document to enrich the document;

d. a user interface component for receiving at least one user inquiry message seeking information from the data collection;

e. a search component for processing the at least one user inquiry message to identify documents in the data collection that are responsive and retrieving an identifier for documents that are responsive; and

f. a delivery component responsive to a user document request for delivering a requested document.

2. The system of claim 1 wherein the system further comprises a persistence service for storing retrieved identifiers for later access without the search component reprocessing the at least one user inquiry message.

3. The system of claim 1 wherein the system further comprises a table of contents service for maintaining at least one table of contents with a plurality of nodes, one or more of said nodes identifying one or more documents associated with said one or more nodes.

4. The system of claim 3 wherein the table of contents service supports two or more tables of contents, each table of contents being adapted to a particular user type.

5. The system of claim 3 wherein the at least one table of contents references documents that are in two or more data collections.

6. The system of claim 4 wherein one of the two or more tables of contents references a node in another of the two or more tables of contents to define a recursive structure.

7. The system of claim 1 wherein documents in the at least one data collection are partitioned into at least one collection subset and the system has an index service that maintains an index of keywords appearing at least once in the collection subset, with an association between the keywords in the index and the location of their appearance in the collection subset.

8. The system of claim 1 wherein documents in the at least one data collection are partitioned into at least one collection subset and wherein an intake component for receiving new documents ensures that each additional document has a unique identifier and is assigned to at least one collection subset.

9. The system of claim 1 wherein the at least one data collection has a least one document set that is an aggregation of the documents in one or more collection subsets.

10. The system of claim 1 wherein the system further comprises a security service component that receives user identification information and inquiry messages and validates the identification information and inquiry messages against user subscription information.

11. The system of claim 1 wherein the enrichment component processes a document to cause association of the document with at least one of the following:

a. additional editorial material prepared by a human agent;

b. additional editorial material prepared by an automated agent;

c. a link providing a pointer to another document in the data collection;

d. citation based metadata to legal or bibliographic documents;

or

e. an entry associated with the document appearing in a metadata file.

12. The system of claim 1 wherein the intake component for receiving new documents prioritizes the received documents and processes out of a normal order based on time of receipt documents bearing a time sensitive indicator.

13. The system of claim 1 wherein the intake component for receiving new documents checks the uniqueness of an assigned document identifier before a new document with such document identifier is made available in the at least one data collection.

14. The system of claim 1, wherein the intake component for receiving new documents checks a new document for a predetermined intake format before such new document is made available in the at least one data collection.

15. The system of claim 1 wherein the aggregation of documents comprises at least 20 terabytes of information.

16. A system for processing inquiry messages seeking documents comprising:

a. one or more user interfaces for receiving inquiry messages, each of said one or more user interfaces being adapted to a resource application running on the system;

b. one or more data collections for storing documents for delivery to a user in response to an inquiry message;

c. one or more metadata information files for holding metadata for facilitating searches for documents stored in the one or more data collections; and

d. a new document intake component for processing documents that are to be added to the one or more data collections, the interface having metadata extractors to develop metadata from a new document and to store at least a portion of the metadata in the metadata information files substantially simultaneously with storing the new document as ready for user access in the one or more data collections.

17. The system of claim 16 wherein the system comprises a table of contents service for maintaining at least one table of contents with a plurality of nodes, one or more of said nodes identifying one or more documents associated with said one or more nodes.

18. The system of claim 17 wherein the table of contents service supports two or more tables of contents, each table of contents being adapted to a particular user type.

19. The system of claim 17 wherein the at least one table of contents references documents that are in two or more data collections.

20. The system of claim 18 wherein one of the two or more tables of contents references a node in another of the two or more tables of contents to define a recursive structure.

21. The system of claim 16 wherein documents in at least one data collection are partitioned into at least one collection subset and the system has an index service that maintains an index of keywords appearing at least once in the collection subset, with an association between the keywords in the index and the location of their appearance in the collection subset.

22. The system of claim 16 wherein documents in at least one data collection are partitioned into at least one collection subset and wherein the means for receiving new documents ensures that each additional document has a unique identifier and is assigned to at least one collection subset.

23. The system of claim 16 wherein the at least one data collection has a least one document set that is an aggregation of the documents in one or more collection subsets.

24. The system of claim 16 wherein the system further comprises a security service component that receives user identification information and inquiry messages and validates the identification information and inquiry messages against user subscription information.

25. The system of claim 16 wherein the metadata extractors process a document to cause association of the document with at least one of the following:

a. additional editorial material prepared by a human agent;

b. additional editorial material prepared by an automated agent;

c. a link providing a pointer to another document in the data collection;

d. citation based metadata to legal or bibliographic documents;

or

e. an entry associated with the document appearing in a metadata file.

26. The system of claim 16 wherein the intake component prioritizes documents and processes out of a normal order based on time of receipt documents bearing a time sensitive indicator.

27. The system of claim 16 wherein the intake component checks the uniqueness of an assigned document identifier before a new document with such document identifier is made available in any data collection.

28. The system of claim 16, wherein the intake component checks a new document for a predetermined intake format before such new document is made available in any data collection.

29. The system of claim 16 wherein the one or more data collections comprise at least 20 terabytes of information.

30. A method for delivering to users that submit inquiry messages inquiry results and documents selected from a large aggregation of electronically stored documents, comprising:

a. providing access to at least one user interface associated with a resource application running on a computer system for eliciting from a user an inquiry message in electronic form seeking documents stored in electronic form in a data collection, each document having a unique identifier;

b. providing for the inquiry message to be delivered to a search component shared by multiple resource applications for processing the inquiry message to identify documents in the data collection that are responsive and retrieving an identifier for documents that are responsive;

c. providing to the user in response to the inquiry message a search result message identifying one or more documents;

d. in response to a user message that selects a document from the search result message, delivering the selected document to the user in a predetermined format based on the at least one resource application associated with the at least one user interface; and

e. associating with the selected document a point-in-time attribute, to permit detection of an updated version of the selected document.

31. A computer data signal embodied in a transmission medium for facilitating delivery to users that submit inquiry messages inquiry results and documents selected from a large aggregation of electronically stored documents, comprising:

a code component for presenting at least one user interface associated with a resource application running on a computer system for eliciting from a user an inquiry message in electronic form seeking documents stored in electronic form in a data collection, each document having a unique identifier;

a code component providing for the inquiry message to be delivered to a search component shared by multiple resource applications for processing the inquiry message to identify documents in the data collection that are responsive and retrieving an identifier for documents that are responsive;

a code component for providing a search result message identifying one or more documents;

a code component responsive to document selection data, for delivering the selected document to the user in a predetermined format based on the at least one resource application associated with the at least one user interface and associating with the selected document a point-in-time attribute, to permit detection of an updated version of the selected document.

32. A system for maintaining a large aggregation of electronically stored documents and making them available to users that submit inquiry messages, comprising:

a. at least one data collection for storing documents in electronic form, each document having a unique identifier;

b. an intake component for receiving new documents to be added to the at least one data collection and associating with a new document a duplicate comparison signature comprising the document'"'"'s top n unique inverse document frequency (idf) terms and a document length scalar;

c. a user interface component for receiving at least one user inquiry message seeking information from the data collection;

d. a search component for processing the at least one user inquiry message to identify documents in the data collection that are responsive and retrieving an identifier for documents that are responsive;

e. a duplicates detection component for performing comparisons among the duplicate comparison signatures of the documents that are responsive and identifying duplicates when comparison signatures are exactly the same or within a predetermined similarity threshold; and

f. a delivery component responsive to a user document request for displaying a requested document.

33. The system of claim 32 wherein the duplicate comparison signature comprises the top n unique idf terms and their relative positions in the document and a document length scalar and n is from four to thirty.

34. The system of claim 32 wherein the duplicate comparison signature comprises the top n unique idf terms and their relative positions in the document and a document length scalar based on tokens, wherein the predetermined similarity threshold with respect to the length scalar is the range 0 to 256 tokens.

35. The system of claim 32 wherein the system employs client-server architecture and the duplicates detection component is on the server side and communicates the identification of duplicates to the client side.

36. The system of claim 32 wherein the system employs client-server architecture and the identification of duplicates to the client side is used to remove the duplicate documents from a search results list communicated to a user.

37. The system of claim 36 wherein on the client side the search results list communicated to the user contains an indication of those documents for which a duplicate was found.

38. The system of claim 36 wherein on the client side the search results list communicated to the user contains an indication of those documents for which a duplicate was found and a set of duplicate documents is made available for a user to access.

39. A system for retailing information sought by user inquiry messages comprising:

a. one or more user interfaces for receiving inquiry messages, each of said one or more user interfaces being adapted to a resource application running on the system;

b. one or more data collections for storing documents of interest for the resource application for delivery to a user in response to an inquiry message;

c. one or more metadata information files for holding metadata for facilitating searches initiated by the inquiry messages for documents stored in the one or more data collections;

d. a new document intake component for processing documents that are to be added to the one or more data collections, the interface having metadata extractors to develop metadata from a new document and to store at least a portion of the metadata in the metadata information files substantially simultaneously with storing the new document as ready for user access in the one or more data collections; and

e. security and billing services shared by two or more resource applications for controlling access to the one or more data collections in response to inquiry messages and developing information for billing users for access to the one or more data collections.

40. The system of claim 39 wherein the one or more data collections contain one or more of the following types of information:

legal, tax, accounting, medical, scientific, intellectual property, educational course material or news.

41. The system of claim 40 wherein the system further comprises a table of contents service for maintaining for each of the types of information at least one table of contents with a plurality of nodes, one or more of said nodes identifying one or more documents associated with said one or more nodes and at least one document is identified in more than one table of contents.

42. A system for processing inquiry messages seeking documents comprising:

a. one or more user interfaces for receiving inquiry messages, each of said one or more user interfaces being adapted to a resource application running on the system;

b. one or more data collections for storing documents for delivery to a user in response to an inquiry message;

c. one or more metadata information files for holding metadata for facilitating searches for documents stored in the one or more data collections; and

d. a new document intake component for processing documents that are to be added to the one or more data collections, the interface having metadata extractors to develop metadata in the form of Resource Description Framework (RDF) statements from a new document and to store at least a portion of the metadata in the metadata information files substantially simultaneously with storing the new document as ready for user access in the one or more data collections.

43. The system of claim 42 further comprising a table of contents service for maintaining at least one table of contents with a plurality of nodes for each of the types of information and at least one of the metadata extractors uses as vocabulary for the RDF statements, the labels of nodes of the at least one table of contents.

44. A system for maintaining a large aggregation of electronically stored documents and making them available to users that submit inquiry messages, comprising:

a. at least one data collection for storing a documents in electronic form, each document having a unique identifier;

b. an intake component for receiving new documents to be added to the at least one data collection;

c. a timestamp component associated with the intake component for locating or placing in a received document at least one PIT field;

d. a user interface component for receiving at least one user inquiry message seeking information from the data collection;

e. a search component for processing the at least one user inquiry message to identify documents in the data collection that are responsive and retrieving an identifier for documents that are responsive, said processing and retrieving being responsive to the at least one PIT field to deliver a document version responsive to PIT limitations of the inquiry messages; and

f. a delivery component responsive to a user document request for delivering a requested document.

45. The system of claim 44 wherein the at least one PIT field contains time and date information.

46. The system of claim 44 wherein the at least one PIT field contains version information.

47. The system of claim 44 wherein the at least one PIT field differentiates different versions of legislation.

48. A system for processing inquiry messages seeking documents comprising:

a. one or more user interfaces for receiving inquiry messages, each of said one or more user interfaces being adapted to a resource application running on the system;

b. one or more data collections for storing documents for delivery to a user in response to an inquiry message;

c. one or more trail files for holding information about users'"'"' searching processes within a particular resource application and identifiers for documents found in response to inquiry messages; and

d. a trail analysis component for processing trail files to determine common usage patterns for a particular resource application and adjusting parameters of the resource application to present searching options and search results in a manner more consonant with such usage patterns.

49. The system of claim 48 wherein the common usage patterns involve the sequence in which users display documents identified in search results relative to the priority order in which documents are presented as search results and the parameters adjusted affect the priority order in which documents are presented as search results.

50. The system of claim 48 wherein the common usage patterns involve the users review of documents identified in search results as duplicates and the parameters adjusted affect the similarity threshold for a duplicate detection service.

51. A system for processing inquiry messages seeking documents comprising:

a. one or more user interfaces for receiving inquiry messages, each of said one or more user interfaces being adapted to a resource application running on the system;

b. one or more data collections for storing documents for delivery to a user in response to an inquiry message;

c. one or more trail files for holding information about users'"'"' searching processes within a particular resource application and identifiers for documents found in response to inquiry messages; and

d. a trail analysis component for processing trail files to determine common usage patterns and building metadata files accessible to a resource application to embody such usage patterns.