Computerized searchable document repository using separate metadata and content stores and full text indexes

• US 8,688,695 B2
• Filed: 05/26/2011
• Issued: 04/01/2014
• Est. Priority Date: 05/26/2011
• Status: Expired due to Fees

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Mimosa Systems Incorporated (Iron Mountain Incorporated)
• Inventors: Kapoor, Rahul, Botros, Sherif M., Ranade, Sameer H.
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): Black, Linh
• Application Number:
  US13/116,763
• Publication Number:
  US 20120303632A1
• Time in Patent Office: 1,041 Days
• Field of Search: 707/707, 707/726, 707/800, 707/696, 707/711, 707/715, 707739-745, 707/830, 707/831, 707765-768, 707713-719, 386/331
• US Class Current: 707/726
• CPC Class Codes: G06F 16/93 Document management systems

Abstract

A computerized searchable repository stores documents as structured metadata parts and unstructured content parts using single instancing. A full text index used for keyword searching includes a metadata index and a content index. A linking structure includes metadata-to-content (MD to CT) links and content-to-metadata (CT to MD) linking entries, with each MD to CT link linking a metadata part of a document to each content part of the document, and each CT to MD linking entry having one or more CT to MD links collectively linking a content part to the metadata parts of the documents that include the content part. Indexing includes metadata indexing a metadata part, conditionally content indexing a content part, and updating the linking structure. Content indexing is performed only if the content part does not match a content part already stored and indexed. Index entries each associate a key word or key value with corresponding metadata or content parts containing the key word or key value. Updating the linking structure includes generating new MD to CT and CT to MD links between the metadata part and either the new content part or an existing matching content part if present.

Claims

1. A computerized searchable repository for documents, each document having a structured metadata part and one or more unstructured content parts, comprising:

a storage device to store the documents, a full text index and a linking structure, the content parts of the documents being stored in a single-instanced manner to avoid storing duplication of identical content parts, the full text index being usable for keyword searching of the documents and including a metadata index and a content index of the metadata and content parts respectively of the documents, the linking structure including metadata-to-content links and content-to-metadata linking entries, each metadata-to-content link linking a metadata part of a respective document to each content part of the document, each content-to-metadata linking entry having one or more content-to-metadata links collectively linking a respective content part to the metadata parts of a group of documents that each include the content part; and

a processor to perform full text indexing of the documents in the storage device, the full text indexing of each document including metadata indexing a metadata part, conditionally content indexing a content part, and updating the linking structure, the content indexing being performed only if the content part is a new content part not matching any of at least a set of content parts already stored in a content store and indexed in the content index, each of the metadata indexing and content indexing including generating new index entries in the metadata or content index respectively for the metadata or content part respectively, each index entry associating a respective key word or key value with a corresponding one or more metadata or content parts containing the key word or key value, and the updating of the linking structure including generating new metadata-to-content and content-to-metadata links between the metadata part and either the new content part or an existing matching content part if present.

2. A computerized searchable repository according to claim 1, wherein the documents are organized into sets in the storage device, and wherein:

the content index is a single content index including index entries for content parts of all the documents, and the metadata index includes a plurality of metadata indexes each including index entries for metadata parts of only a corresponding set of the documents; and

the linking structure includes a single content-to-metadata linking structure containing the content-to-metadata entries for all the documents, and a plurality of metadata-to-content linking structures each containing the metadata-to-content links for only a corresponding set of the documents.

3. A computerized searchable repository according to claim 2, wherein the sets of documents are selected from local email files, share point files, file system files, hosted email files, hosted document management systems, databases, enterprise resource planning or customer relationship management systems, and instant messaging content.

4. A computerized searchable repository according to claim 2, wherein:

each metadata-to-content linking structure is included in a respective one of the plurality of metadata indexes; and

the content-to-metadata linking structure is separate from the content index so as to be updateable without requiring updating of the content index.

5. A computerized searchable repository according to claim 2, wherein the content-to-metadata linking structure is organized in a set-first manner to support searches limited in scope to one or more identified sets of the documents.

6. A computerized searchable repository according to claim 2, wherein the set of content parts required not to match the new content part includes the content parts of documents from multiple sets.

7. A computerized searchable repository according to claim 1, wherein the metadata part for each document includes information identifying how the document is structured from the content parts of the document.

8. A computerized searchable repository according to claim 7, wherein the information identifies whether a content part is a message body or an attachment and the order in which the attachments occur.

9. A computerized searchable repository according to claim 1, wherein the content part of at least some of the documents includes text and internal meta-data separate from the metadata part of the document, and wherein the content indexing includes examining only the text part of a content part to determine whether the content part is a new content part not matching any of the set of content parts already stored in the content store and indexed in the content index.

10. A computerized searchable repository according to claim 1, wherein the processor is further to perform a mixed-criteria search including metadata criteria as well as content criteria, the mixed-criteria search including selecting between a metadata-first strategy and a content-first strategy depending on an estimate of whether the metadata criteria or the content criteria is more selective.

11. A computerized searchable repository according to claim 1, wherein the processor is further to perform searching using query expansion, to separate out AND queries into separate queries on separate document parts and to perform an intersection on intermediate results internally to generate final search results.

12. A method of operating a computerized searchable repository for documents, each document having a structured metadata part and one or more unstructured content parts, comprising:

storing the documents, a full text index and a linking structure in a storage sub-system of the computerized searchable repository, the content parts of the documents being stored in a single-instanced manner to avoid storing duplication of identical content parts, the full text index being usable for keyword searching of the documents and including a metadata index and a content index of the metadata and content parts respectively of the documents, the linking structure including metadata-to-content links and content-to-metadata linking entries, each metadata-to-content link linking a metadata part of a respective document to each content part of the document, each content-to-metadata linking entry having one or more content-to-metadata links collectively linking a respective content part to the metadata parts of a group of documents that each include the content part; and

operating processing circuitry of the computerized searchable repository to perform full text indexing of the documents in the storage sub-system, the full text indexing of each document including metadata indexing a metadata part, conditionally content indexing a content part, and updating the linking structure, the content indexing being performed only if the content part is a new content part not matching any of at least a set of content parts already stored in a content store and indexed in the content index, each of the metadata indexing and content indexing including generating new index entries in the metadata or content index respectively for the metadata or content part respectively, each index entry associating a respective key word or key value with a corresponding one or more metadata or content parts containing the key word or key value, and the updating of the linking structure including generating new metadata-to-content and content-to-metadata links between the metadata part and either the new content part or an existing matching content part if present.

13. A method according to claim 12, wherein the documents are organized into sets in the storage sub-system, and wherein:

the content index is a single content index including index entries for content parts of all the documents, and the metadata index includes a plurality of metadata indexes each including index entries for metadata parts of only a corresponding set of the documents; and

the linking structure includes a single content-to-metadata linking structure containing the content-to-metadata entries for all the documents, and a plurality of metadata-to-content linking structures each containing the metadata-to-content links for only a corresponding set of the documents.

14. A method according to claim 13, wherein the sets of documents are selected from local email files, share point files, file system files, hosted email files, hosted document management systems, databases, enterprise resource planning or customer relationship management systems, and instant messaging content.

15. A method according to claim 13, wherein:

each metadata-to-content linking structure is included in a respective one of the plurality of metadata indexes; and

the content-to-metadata linking structure is separate from the content index so as to be updateable without requiring updating of the content index.

16. A method according to claim 13, wherein the content-to-metadata linking structure is organized in a set-first manner to support searches limited in scope to one or more identified sets of the documents.

17. A method according to claim 12, wherein the set of content parts required not to match the new content part includes the content parts of documents from multiple sets.

18. A method according to claim 12, wherein the metadata part for each document includes information identifying how the document is structured from the content parts of the document.

19. A method according to claim 18, wherein the information identifies whether a content part is a message body or an attachment and the order in which the attachments occur.

20. A method according to claim 12, wherein the content part of at least some of the documents includes text and internal meta-data separate from the metadata part of the document, and wherein the content indexing includes examining only the text part of a content part to determine whether the content part is a new content part not matching any of the set of content parts already stored in the content store and indexed in the content index.

21. A method according to claim 12, wherein operating the processing circuitry further includes performing a mixed-criteria search including metadata criteria as well as content criteria, the mixed-criteria search including selecting between a metadata-first strategy and a content-first strategy depending on an estimate of whether the metadata criteria or the content criteria is more selective.

22. A method according to claim 12, wherein operating the processing circuitry further includes performing searching using query expansion, separating out AND queries into separate queries on separate document parts and performing an intersection on intermediate results internally to generate final search results.