DOCUMENT SIMILARITY EVALUATION SYSTEM, DOCUMENT SIMILARITY EVALUATION METHOD, AND COMPUTER PROGRAM

• US 20130191410A1
• Filed: 11/09/2012
• Published: 07/25/2013
• Est. Priority Date: 01/19/2012
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Zhou, Wenqi
• Granted Patent: US 9,235,624 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/24558 Binary matching operations
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 40/194 Calculation of difference between files

Abstract

Disclosed is a document similarity evaluation system or the like which can evaluate a degree of concentration and dispersion of parts with high similarity in at least two kinds of documents. The system includes a segment search unit which finds common segments (CS) in first and second segment strings, counts the number of CS, and identifies an appearance range (AR) within CS; and a similarity index (SI) calculation unit which calculates a first sum that is a sum of the numbers of characters of each segment (NCS) in AR and a second sum that is a sum of NCS of CS and calculates SI between the first and second segment strings by the following equation, SI=F(NTC)/G(NCC)×NS (where, NTC is the first sum, NCC is the second sum, NS is the number of the CS, functions F and G monotonically increase at larger than 0).

Claims

1. A document similarity evaluation system comprising:

a segment search unit which finds common segments in both a first segment string and a second segment string, counts the number of the common segments that are found, and identifies an appearance range within which the common segments appear; and

a similarity index calculation unit which calculates a first sum that is a sum of the numbers of characters of each segment included in the appearance range identified by the segment search unit, calculates a second sum that is a sum of the numbers of characters of each segment identified as the common segments, and calculates the similarity index indicating the similarity between the first segment string and the second segment string by using the following equation,

similarity index=F(NTC)/G(NCC)×

NS

(Where, in the above-mentioned equation,

NTC is the first sum,

NCC is the second sum,

NS is the number of the common segments, and

a function F and a function G are monotonically increasing functions by which a certain integer value is associated with a positive real value).

2. The document similarity evaluation system according to claim 1, wherein the similarity index calculation unit calculates the first sum and the second sum based on a character number information in which each segment included in the appearance range is associated with the number of characters included in the each segment.

3. The document similarity evaluation system according to claim 1, wherein the similarity index calculation unit calculates the similarity index indicating the similarity between the first segment string and the second segment string by using the following equation,

similarity index=H(NTC/NCC)×

NS,

(Where, in the above-mentioned equation,

NTC is the first sum,

NCC is the second sum,

NS is the number of the common segments, and

a function H is a monotonically increasing function by which a certain integer value is associated with a positive real value).

4. The document similarity evaluation system according to claim 1, wherein the similarity index calculation unit calculates the similarity index indicating the similarity between the first segment string and the second segment string by using the following equation,

similarity index=NTC/NCC×

NS

(Where, in the above-mentioned equation,

NTC is the first sum,

NCC is the second sum, and

NS is the number of the common segments).

5. A document similarity evaluation method calculating a similarity index indicating a similarity between a first segment string and a second segment string comprising:

finding common segments in both the first segment string and the second segment string,

counting the number of the common segments that are found;

identifying an appearance range within which the common segments appear;

calculating a first sum that is a sum of the numbers of characters of each segment included in the appearance range;

calculating a second sum that is a sum of the numbers of characters of each segment identified as the common segments; and

calculating the similarity index by the following equation,

similarity index=F(NTC)/G(NCC)×

NS

(Where, in the above-mentioned equation,

NTC is the first sum,

NCC is the second sum,

NS is the number of the common segments, and

a function F and a function G are monotonically increasing functions by which a certain integer value is associated with a positive real value).

6. The document similarity evaluation method according to claim 5, wherein the calculation of the similarity index indicating the similarity is performed by using the following equation,

similarity index=H(NTC/NCC)×

NS

(Where, in the above-mentioned equation,

NTC is the first sum,

NCC is the second sum,

NS is the number of the common segments, and

a function H is a monotonically increasing function by which a certain integer value is associated with a positive real value).

7. The document similarity evaluation method according to claim 5, wherein the calculation of the similarity index indicating the similarity is performed by using the following equation,

similarity index=NTC/NCC×

NS

(Where, in the above-mentioned equation,

NTC is the first sum,

NCC is the second sum, and

NS is the number of the common segments).

8. A non-transitory computer-readable storage medium storing a computer program which causes a computer to realize:

a segment search function to find common segments in both a first segment string and a second segment string, count the number of the common segments that are found, and identify an appearance range within which the common segments appears; and

a similarity index calculation function to calculate a first sum that is a sum of the numbers of characters of each segment included in the appearance range identified in the segment search function, calculate a second sum that is a sum of the numbers of characters of each segment identified as the common segments, and calculate a similarity index indicating the similarity between the first segment string and the second segment string by using the following equation,

similarity index=F(NTC)/G(NCC)×

NS

(Where, in the above-mentioned equation,

NTC is the first sum,

NCC is the second sum,

NS is the number of the common segments, and

a function F and a function G are monotonically increasing functions by which a certain integer value is associated with a positive real value).

9. The non-transitory computer-readable storage medium according to claim 8, wherein the similarity index calculation function calculates the similarity index by using the following equation,

similarity index=H(NTC/NCC)×

NS

(Where, in the above-mentioned equation,

NTC is the first sum,

NCC is the second sum,

NS is the number of the common segments, and

a function H is a monotonically increasing function by which a certain integer value is associated with a positive real value).

10. The non-transitory computer-readable storage medium according to claim 8, wherein the similarity index calculation function calculates the similarity index by using the following equation,

similarity index=NTC/NCC×

NS

(Where, in the above-mentioned equation,

NTC is the first sum,

NCC is the second sum, and

NS is the number of the common segments).