Methods and systems for automatically generating semantic/concept searches

• US 8,719,257 B2
• Filed: 02/16/2011
• Issued: 05/06/2014
• Est. Priority Date: 02/16/2011
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Rangan, Venkat
• Primary Examiner(s): FILIPCZYK, MARCIN R
• Application Number:
  US13/028,826
• Publication Number:
  US 20120209847A1
• Time in Patent Office: 1,175 Days
• Field of Search: 707/710, 707/715, 707/737, 707/739, 707/743, 707/749, 707/750, 707/718, 707/776, 707999003-999006
• US Class Current: 707/718
• CPC Class Codes: G06F 16/3347 using vector based model

Abstract

In various embodiments, a semantic space associated with a corpus of electronically stored information (ESI) may be created and used for concept searches. Documents (and any other objects in the ESI, in general) may be represented as vectors in the semantic space. Vectors may correspond to identifiers, such as, for example, indexed terms. The semantic space for a corpus of ESI can be used in information filtering, information retrieval, indexing, and relevancy rankings.

Claims

1. A method comprising:

receiving, at one or more computer systems, a plurality of documents;

for each term in a set of terms associated with each document in the plurality of documents, generating a term vector for the term with one or more processors associated with the one or more computer systems based on a set of randomly indexed document vectors associated with documents in the plurality of documents in which the term appears, wherein each of the randomly indexed document vectors comprises a set of values, each of the set of values being a random value;

storing each generated term vector in association with its corresponding term in a storage device associated with the one or more computer systems;

generating, with the one or more processors associated with the one or more computer systems, a document vector for each document in the plurality of documents based on a set of term vectors associated with terms that appear in the document, wherein the set of term vectors is merged into each document vector based on the frequency of each term in each document;

storing each generated document vector in association with its corresponding document in the storage device associated with the one or more computer systems;

generating, with the one or more processors associated with the one or more computer systems, a query term vector for one or more query terms based on term vectors for terms that correspond to the one or more query terms;

generating, with the one or more processors associated with the one or more computer systems, a query based on a set of terms whose term vectors satisfy one or more conditions related to the query term vector; and

executing, with the one or more processors associated with the one or more computer systems, the query to obtain a set of documents in the plurality of documents that are relevant to a concept defined by the one or more query terms.

2. The method of claim 1 wherein generating the term vector for the term based on the set of randomly indexed document vectors associated with documents in the plurality of documents in which the term appears comprises:

determining frequency of the term in a document; and

incorporating, into the term vector, a document vector associated with the document scaled based on the determined frequency of the term in the document.

3. The method of claim 1 wherein generating the document vector for each document in the plurality of documents based on a set of term vectors associated with terms that appear in the document comprises:

determining frequency of a term in the document;

determining frequency of the term in the plurality of documents; and

incorporating, into the document vector, a term vector associated with the term scaled based on the determined frequency of the term in the document and the determined frequency of the term in the plurality of documents.

4. The method of claim 1 wherein storing each generated term vector in association with its corresponding term in the storage device associated with the one or more computer systems comprises generating an object-ordered index.

5. The method of claim 1 wherein storing each generated term vector in association with its corresponding term in the storage device associated with the one or more computer systems comprises generating a vector-ordered index.

6. The method of claim 1 further comprising:

generating a set of document clusters based on the document vectors associated with each document in the plurality of documents; and

identifying a centroid associated with at least one cluster as a concept represented by the set of documents.

7. The method of claim 1 further comprising:

receiving a term vector associated with a selected term;

determining a set of terms wherein each term vector of a term in the set of terms satisfies one or more conditions related to the term vector associated with the selected term; and

generating information indicating that the set of terms are related to the selected term.

8. The method of claim 1 further comprising:

receiving a term vector associated with a selected term;

determining a set of terms wherein a term vector of a term in the set of terms satisfies one or more conditions related to the term vector associated with the selected term;

determining a set of documents based on the set of terms wherein a document vector of a document in the set of document satisfies one or more conditions related to term vectors of the set of terms; and

generating information indicating that the set of documents are related to the selected term.

9. The method of claim 1 further comprising:

receiving a document vector associated with a selected document;

determining a set of documents wherein each document vector of a document in the set of documents satisfies one or more conditions related to the document vector associated with the selected document; and

generating information indicating that the set of documents are related to the selected document.

10. The method of claim 1 further comprising:

receiving a review specification indicative of a review of a subset of documents in the plurality of documents;

determining whether a document vector of a document satisfies one or more conditions related to document vectors associated with document in the subset of documents; and

applying the review specification to the document when the document vector of the document satisfies the one or more conditions related to document vectors associated with document in the subset of documents.

11. A non-transitory computer-readable medium storing a plurality of instructions that cause a computer to perform operations comprising:

generating, for each term in a set of terms associated with each document in a plurality of documents, a term vector for the term based on a set of randomly indexed document vectors associated with documents in the plurality of documents in which the term appears, wherein each of the randomly indexed document vectors comprises a set of values, each of the set of values being a random value;

storing each generated term vector in association with its corresponding term;

generating a document vector for each document in the plurality of documents based on a set of term vectors associated with terms that appear in the document, wherein the set of term vectors is merged into each document vector based on the frequency of each term in each document;

storing each generated document vector in association with its corresponding document;

generating a query term vector for one or more query terms based on term vectors for terms that correspond to the one or more query terms;

generating a query based on a set of terms whose term vectors satisfy one or more conditions related to the query term vector; and

executing the query to obtain a set of documents in the plurality of documents that are relevant to a concept defined by the one or more query terms.

12. The computer-readable medium of claim 11 wherein generating the term vector for the term based on the set of randomly indexed document vectors associated with documents in the plurality of documents in which the term appears comprises:

determining frequency of the term in a document; and

incorporating, into the term vector, a document vector associated with the document scaled based on the determined frequency of the term in the document.

13. The computer-readable medium of claim 11 wherein generating the document vector for each document in the plurality of documents based on a set of term vectors associated with terms that appear in the document comprises:

determining frequency of a term in the document;

determining frequency of the term in the plurality of documents; and

incorporating, into the document vector, a term vector associated with the term scaled based on the determined frequency of the term in the document and the determined frequency of the term in the plurality of documents.

14. The computer-readable medium of claim 11 wherein storing each generated term vector in association with its corresponding term comprises generating an object-ordered index.

15. The computer-readable medium of claim 11 wherein storing each generated term vector in association with its corresponding term comprises generating a vector-ordered index.

16. The computer-readable medium of claim 11, wherein the operations further comprise:

generating a set of document clusters based on the document vectors associated with each document in the plurality of documents; and

identifying a centroid associated with at least one cluster as a concept represented by the set of documents.

17. The computer-readable medium of claim 11, wherein the operations further comprise:

receiving a term vector associated with a selected term;

determining a set of terms wherein each term vector of a term in the set of terms satisfies one or more conditions related to the term vector associated with the selected term; and

generating information indicating that the set of terms are related to the selected term.

18. The computer-readable medium of claim 11, wherein the operations further comprise:

receiving a term vector associated with a selected term;

determining a set of terms wherein a term vector of a term in the set of terms satisfies one or more conditions related to the term vector associated with the selected term;

determining a set of documents based on the set of terms wherein a document vector of a document in the set of document satisfies one or more conditions related to term vectors of the set of terms; and

generating information indicating that the set of documents are related to the selected term.

19. The computer-readable medium of claim 11, wherein the operations further comprise:

receiving a document vector associated with a selected document;

determining a set of documents wherein each document vector of a document in the set of documents satisfies one or more conditions related to the document vector associated with the selected document; and

generating information indicating that the set of documents are related to the selected document.

20. The computer-readable medium of claim 11, wherein the operations further comprise:

receiving a review specification indicative of a review of a subset of documents in the plurality of documents;

determining whether a document vector of a document satisfies one or more conditions related to document vectors associated with document in the subset of documents; and

applying the review specification to the document when the document vector of the document satisfies the one or more conditions related to document vectors associated with document in the subset of documents.