System, method, and apparatus for pairing a short document to another short document from a plurality of short documents

• US 10,083,229 B2
• Filed: 10/09/2009
• Issued: 09/25/2018
• Est. Priority Date: 10/09/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Eyres, Greg, Hacigumus, Vahit Hakan, Lehman, Tobin J., Strong, Jr., H. Raymond
• Primary Examiner(s): HARPER, ELIYAH STONE
• Application Number:
  US12/576,959
• Publication Number:
  US 20110087701A1
• Time in Patent Office: 3,273 Days
• Field of Search: 707999102, 707736
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/93 Document management systems

Abstract

A computer-implemented method for pairing a new document to a document from a plurality of documents. Embodiments include, for each of the new document and the plurality of documents, generating a vector of terms of interest uniquely associated with a document of the new document and the plurality of documents. For each term of interest, an associated element value of the vector is assigned as zero if the term of interest does not occur in the document and one otherwise. The method also includes, for each document from the plurality of documents, determining a similarity between the vectors. The method also includes selecting a document from the plurality of documents as related to the new document if the similarity between the vector for the new document and the vector for the document from the plurality of documents is greater than or equal to a threshold value.

Claims

1. A computer-implemented method for pairing a new document to a document from a plurality of documents in a document repository, comprising:

for each of the new document and the plurality of documents in the document repository, generating a vector uniquely associated with a document of the new document and the plurality of documents, wherein;

the vector comprises a number of elements equal to a number of terms of interest; and

for each term of interest, an associated element value of the vector is assigned as zero if the term of interest does not occur in the document and one if the term does occur in the document;

for each document from the plurality of documents, determining a similarity between the vector for the new document and the vector for the document from the plurality of documents comprising calculating a cosine measurement of similarity between the vector for the new document and the vector for the document from the plurality of documents;

if it is determined that the similarity between the vector for the new document and the vector for a document from the plurality of documents is greater than or equal to a threshold value then;

selecting the document from the plurality of documents;

generating a merged document by merging the new document with the document from the plurality of documents in response to the document from the plurality of documents being selected, wherein the merging comprises combining at least a portion of the new document with at least a portion of the selected document into the merged document;

removing the selected document from the document repository and adding the merged document to the document repository; and

generating a new vector for the merged document; and

if it is determined that the similarity is less than the threshold value then adding the new document to the document repository without merging the new document.

2. The computer-implemented method of claim 1, wherein selecting a document from the plurality of documents as related to the new document further comprises determining that the similarity between the vector for the new document and the vector for the selected document is as great as the similarity between the vector for the new document and the vector for any other document from the plurality of documents.

3. The computer-implemented method of claim 1, further comprising submitting the merged document as a new document for pairing.

4. The computer-implemented method of claim 1, wherein the associated element value for each term of interest is weighted based on a history of the term of interest in previous pairings.

5. The computer-implemented method of claim 1, wherein:

the new document is a service description in an element of a request for service;

documents of the plurality of documents are each a service description of a service offering available in a catalog of service offerings; and

the selected document from the plurality of documents is a service offering determined to pair to the element of the request for service.

6. The computer-implemented method of claim 5, wherein each service description in the catalog of service offerings is compared to the service description in an element of a request for service, and a service description having the highest similarity is paired with the element of the request for service.

7. The computer-implemented method of claim 1, wherein the documents are short documents.

8. The computer-implemented method of claim 7, wherein the short documents consist of 500 words or less.

9. The computer-implemented method of claim 7, wherein the short documents consist of 50 words or less.

14. The computer-implemented method of claim 1, wherein the associated element value for each term of interest is weighted based on a position of the term of interest in the document.

15. The computer-implemented method of claim 1, wherein the associated element value for each term of interest is weighted based on a position of the term of interest in relation to a predetermined term.

16. The computer-implemented method of claim 1, wherein the associated element value for each term of interest is weighted based on a position of the term of interest within a structural hierarchy of the document.

10. A non-transitory computer useable storage medium to store a computer readable program, wherein the computer readable program, when executed on a computer, causes the computer to perform operations for merging a new document with a document from a plurality of documents, the operations comprising:

receiving a new document;

for each of the new document and each document of a plurality of documents in a document repository, generating a vector uniquely associated with each of the documents of the plurality of documents and the new document, wherein;

the vector comprises a number of elements equal to a number of terms of interest; and

for each term of interest, an associated element value of the vector is assigned as zero if the term of interest does not occur in the document and one if the term does occur in the document;

for each document from the plurality of documents, determining a similarity between the vector for the new document and the vector for the document from the plurality of documents;

selecting a document from the plurality of documents as related to the new document in response to;

determining that the similarity between the vector for the new document and the vector for the document from the plurality of documents is greater than or equal to a threshold value, wherein determining the similarity comprises calculating a cosine measurement of similarity between the vector for the new document and the vector for the document from the plurality of documents; and

determining that the similarity between the vector for the new document and the vector for the document from the plurality of documents is greater than or equal to the similarity between the vector for the new document and the vector for any other document from the plurality of documents; and

merging the new document with the selected document if a document from the plurality of documents is selected, wherein the merging comprises combining at least a portion of the new document with at least a portion of the selected document into a merged document;

removing the selected document from the document repository and adding the merged document to the document repository; and

generating a new vector for the merged document; and

if it is determined that the similarity is less than the threshold value then adding the new document to the document repository without merging the new document with a document from the plurality of documents.

11. The computer useable storage medium of claim 10, wherein the new document comprises a plurality of terms arranged in a hierarchy, and further comprising weighting a term in the terms of interest based on a hierarchical level of the term.

12. A system comprising:

a document repository to store a plurality of documents;

a comparison engine comprising;

a vector generator to generate a vector for each of a new document and a document from the document repository, wherein;

the vector comprises a number of elements equal to a number of terms of interest; and

for each term of interest, an associated element value of the vector is assigned as zero if the term of interest does not occur in the document, and one if the term does occur in the document; and

a similarity generator to determine a similarity between the document and the document from the document repository, the similarity based on the vector for the new document and the vector for the document from the document repository, wherein to determine the similarity between the document and the document from the document repository comprises calculating a cosine measurement of similarity between the vector for the new document and the vector for the document from the plurality of documents; and

a pairing engine comprising;

a document receiver to receive the new document to be paired with the documents in the document repository;

a document submitter to submit the new document to the comparison engine and direct the comparison engine to determine a similarity between the new document and each document of the plurality of documents in the document repository; and

a pair indicator to indicate a pairing between the new document and a document in the document repository having the highest similarity with the new document if the similarity between the new document and the document in the document repository having the highest similarity with the new document is greater than or equal to a threshold value; and

a merge engine comprising;

a document receiver to receive a new document to be incorporated with the documents in the document repository;

a document submitter to submit the new document to the comparison engine and direct the comparison engine to determine a similarity between the new document and each document of the plurality of documents in the document repository;

a merged document generator to generate a merged document by combining at least a portion of the new document with at least a portion of a document in the document repository having the highest similarity with the new document if the similarity between the new document and the document in the document repository having the highest similarity with the new document is greater than or equal to a threshold value;

a document remover to remove the document in the document repository having the highest similarity with the new document; and

a merged document adder to add the merged document to the document repository;

wherein the vector generator further generates a new vector for the merged document.

13. The system of claim 12, wherein the merge engine is further configured to:

submit the merged document as a new document to the pairing engine.