Text processing apparatus, text processing method, and computer-readable recording medium

• US 8,886,519 B2
• Filed: 12/21/2009
• Issued: 11/11/2014
• Est. Priority Date: 12/26/2008
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Tamura, Akihiro, Ishikawa, Kai, Ando, Shinichi
• Primary Examiner(s): SPOONER, LAMONT M
• Application Number:
  US13/142,302
• Publication Number:
  US 20110282653A1
• Time in Patent Office: 1,786 Days
• Field of Search: 704/1, 704/9, 704/10, 704/235, 704/257, 704/277, 707706-708, 715/256
• US Class Current: 704/9
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment

Abstract

A text processing apparatus is provided with a segment determination unit 36 and a descriptive content determination unit 33. The segment determination unit 36 determines, with respect to a homogeneous segment that is similar to segments constituting a first text which is set as an analysis target (analysis target text) and that is included in another first text, whether the content thereof is included in a second text. The descriptive content determination unit 33 determines whether each segment constituting the analysis target text should be described in a corresponding second text, based on the determination result.

Claims

1. A text processing apparatus for performing an analysis process by contrasting a first text set constituted by first texts and a second text set constituted by second texts corresponding to the first texts, the first texts and the second texts corresponding thereto being generated around a same event through mutually different generation processes, the text processing apparatus comprising:

a segment determination unit; and

a descriptive content determination unit,

wherein the segment determination unit determines, with respect to a homogeneous segment that is similar to a plurality of segments constituting a first text which is set as an analysis target and that is included in another first text, whether a content thereof is included in the second texts, and

the descriptive content determination unit determines, based on a result of the determination by the segment determination unit, whether each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target.

2. The text processing apparatus according to claim 1,

wherein the segment determination unit includes a homogeneous segment specification unit and an inclusion determination unit,

the homogeneous segment specification unit, by contrasting each of the plurality of segments constituting the first text which is set as the analysis target and the other first text, specifies a segment similar to any of the plurality of segments constituting the first text which is set as the analysis target, from a plurality of segments constituting the other first text, and sets the specified segment as a homogeneous segment, and

the inclusion determination unit determines whether the content of the homogeneous segment is included in the second text corresponding to the other first text which includes the homogeneous segment.

3. The text processing apparatus according to claim 2,

wherein the descriptive content determination unit derives, based on the result of the determination by the segment determination unit, an extent to which the content of the homogeneous segment is included in the second text corresponding to the other first text which includes the homogeneous segment, further derives, based on the derived extent, a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, and performs the determination using the degree.

4. The text processing apparatus according to claim 3,

wherein the inclusion determination unit, in addition to the determination regarding the content of the homogeneous segment, computes, for each of the plurality of segments constituting the first text which is set as the analysis target and for the homogeneous segment, an inclusion score representing a possibility of a content of the segment being included in the second text corresponding to the first text which includes the segment, and

the descriptive content determination unit further derives the degree using the inclusion score computed by the inclusion determination unit, such that the degree increase the higher the inclusion score.

5. The text processing apparatus according to claim 3,

wherein the descriptive content determination unit derives, as the extent of inclusion in the second text, a percentage at which the content of the homogeneous segment is described in the second text corresponding to the other first text which includes the homogeneous segment, further derives a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, such that the degree increases the higher the percentage, and performs the determination using the derived degree.

6. The text processing apparatus according to claim 5,

wherein the homogeneous segment specification unit, in addition to the specification of the homogeneous segment, computes a similarity score representing an extent of similarity between each of the plurality of segments constituting the first text which is set as the analysis target and the homogeneous segment, and

the descriptive content determination unit further derives the degree such that the degree increases the higher the similarity score computed by the homogeneous segment specification unit.

7. The text processing apparatus according to claim 3,

wherein the descriptive content determination unit derives, as the extent of inclusion in the second text, a frequency at which the content of the homogeneous segment is described in the second text corresponding to the other first text which includes the homogeneous segment, further derives a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, such that the degree increases the higher the frequency, and performs the determination using the derived degree.

8. The text processing apparatus according to claim 1,

wherein the segment determination unit includes an inclusion determination unit and a homogeneous segment specification unit,

the inclusion determination unit determines, for the plurality of segments respectively constituting all of the first texts, whether a content of each segment is included in the second text corresponding to the first text which includes the segment, and

the homogeneous segment specification unit, by contrasting each of the plurality of segments constituting the first text which is set as the analysis target and a segment that is determined by the inclusion determination unit to be included in the second text and that is included in another first text, specifies a segment similar to any of the plurality of segments constituting the first text which is set as the analysis target, and sets the specified segment as a homogeneous segment.

9. The text processing apparatus according to claim 8,

wherein the descriptive content determination unit derives, based on the result of the determination by the segment determination unit, an extent to which the content of the homogeneous segment is included in the second text corresponding to the other first text which includes the homogeneous segment, further derives, based on the derived extent, a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, and performs the determination using the degree.

10. The text processing apparatus according to claim 9,

wherein the inclusion determination unit, in addition to the determination regarding the content of each segment, computes, for the plurality of segments respectively constituting all of the first texts, an inclusion score representing a possibility of the content of each segment being included in the second text corresponding to the first text which includes the segment, and

the descriptive content determination unit further derives the degree using the inclusion score computed by the inclusion determination unit, such that the degree increase the higher the inclusion score.

11. The text processing apparatus according to claim 1, further comprising a text segmentation unit that segments the first texts based on at least one of a sentence, a paragraph, a topic and a phase included therein, and sets the plurality of segments.

12. The text processing apparatus according to claim 1, wherein setting of the plurality of segments constituting the first texts is performed based on at least one of a sentence, a paragraph, a topic and a phase included in the first text.

13. A text processing method for performing an analysis process by contrasting a first text set constituted by first texts and a second text set constituted by second texts corresponding to the first texts, with a computer, the first texts and the second texts corresponding thereto being generated around a same event through mutually different generation processes, the text processing method comprising the steps of:

(a) determining by the computer, with respect to a homogeneous segment that is similar to a plurality of segments constituting a first text which is set as an analysis target and that is included in another first text, whether a content thereof is included in the second texts; and

(b) determining by the computer, based on a result of the determination in the (a) step, whether each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target.

14. The text processing method according to claim 13,

wherein the (a) step includes the steps of:

(l) specifying, by contrasting each of the plurality of segments constituting the first text which is set as the analysis target and the other first text, a segment similar to any of the plurality of segments constituting the first text which is set as the analysis target, from a plurality of segments constituting the other first text, and setting the specified segment as a homogeneous segment, and

(m) determining whether the content of the homogeneous segment is included in the second text corresponding to the other first text which includes the homogeneous segment.

15. The text processing method according to claim 14,

wherein the (b) step comprises deriving, based on the result of the determination in the (a) step, an extent to which the content of the homogeneous segment is included in the second text corresponding to the other first text which includes the homogeneous segment, further deriving, based on the derived extent, a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, and performing the determination using the degree.

16. The text processing method according to claim 15,

wherein the (m) step, in addition to the determination regarding the content of the homogeneous segment, comprises computing, for each of the plurality of segments constituting the first text which is set as the analysis target and for the homogeneous segment, an inclusion score representing a possibility of a content of the segment being included in the second text corresponding to the first text which includes the segment, and

the (b) step comprises further deriving the degree using the inclusion score computed in the (m) step, such that the degree increase the higher the inclusion score.

17. The text processing method according to claim 15,

wherein the (b) step comprises deriving, as the extent of inclusion in the second text, a percentage at which the content of the homogeneous segment is described in the second text corresponding to the other first text which includes the homogeneous segment, further deriving a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, such that the degree increases the higher the percentage, and performing the determination using the derived degree.

18. The text processing method according to claim 17,

wherein the (l) step, in addition to the specification of the homogeneous segment, comprises computing a similarity score representing an extent of similarity between each of the plurality of segments constituting the first text which is set as the analysis target and the homogeneous segment, and

the (b) step comprises further deriving the degree such that the degree increases the higher the similarity score computed in the (l) step.

19. The text processing method according to claim 15,

wherein the (b) step comprises deriving, as the extent of inclusion in the second text, a frequency at which the content of the homogeneous segment is described in the second text corresponding to the other first text which includes the homogeneous segment, further deriving a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, such that the degree increases the higher the frequency, and performing the determination using the derived degree.

20. The text processing method according to claim 13,

wherein the (a) step includes the steps of:

(x) determining, for the plurality of segments respectively constituting all of the first texts, whether a content of each segment is included in the second text corresponding to the first text which includes the segment, and

(y) specifying, by contrasting each of the plurality of segments constituting the first text which is set as the analysis target and a segment that is determined in the (x) step to be included in the second text and that is included in another first text, a segment similar to any of the plurality of segments constituting the first text which is set as the analysis target, and setting the specified segment as a homogeneous segment.

21. The text processing method according to claim 20,

wherein the (b) step comprises deriving, based on the result of the determination in the (a) step, an extent to which the content of the homogeneous segment is included in the second text corresponding to the other first text which includes the homogeneous segment, further deriving, based on the derived extent, a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, and performing the determination using the degree.

22. The text processing method according to claim 21,

wherein the (x) step, in addition to the determination regarding the content of each segment, comprises computing, for the plurality of segments respectively constituting all of the first texts, an inclusion score representing a possibility of the content of each segment being included in the second text corresponding to the first text which includes the segment, and

the (b) step further comprises deriving the degree using the inclusion score computed in the (x) step, such that the degree increase the higher the inclusion score.

23. The text processing method according to claim 13, further comprising the step of:

(c) segmenting the first texts based on at least one of a sentence, a paragraph, a topic and a phase included therein, and setting the plurality of segments, by the computer.

24. The text processing method according to claim 13,

setting of the plurality of segments constituting the first texts is performed based on at least one of a sentence, a paragraph, a topic and a phase included in the first text.

25. A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute an analysis process of contrasting a first text set constituted by first texts and a second text set constituted by second texts corresponding to the first texts, the first texts and the second texts corresponding thereto being generated around a same event through mutually different generation processes, the program recorded on the recording medium includes a command for causing the computer to execute the steps of:

(a) determining, with respect to a homogeneous segment that is similar to a plurality of segments constituting a first text which is set as an analysis target and that is included in another first text, whether a content thereof is included in the second texts; and

(b) determining, based on a result of the determination in the (a) step, whether each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target.

26. The non-transitory computer-readable recording medium according to claim 25,

wherein the computer executes, as the (a) step, the steps of:

(l) specifying, by contrasting each of the plurality of segments constituting the first text which is set as the analysis target and the other first text, a segment similar to any of the plurality of segments constituting the first text which is set as the analysis target, from a plurality of segments constituting the other first text, and setting the specified segment as a homogeneous segment, and

(m) determining whether the content of the homogeneous segment is included in the second text corresponding to the other first text which includes the homogeneous segment.

27. The non-transitory computer-readable recording medium according to claim 26,

wherein the (b) step comprises deriving, based on the result of the determination in the (a) step, an extent to which the content of the homogeneous segment is included in the second text corresponding to the other first text which includes the homogeneous segment, further deriving, based on the derived extent, a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, and performing the determination using the degree.

28. The non-transitory computer-readable recording medium according to claim 27,

wherein the (m) step, in addition to the determination regarding the content of the homogeneous segment, comprises computing, for each of the plurality of segments constituting the first text which is set as the analysis target and for the homogeneous segment, an inclusion score representing a possibility of a content of the segment being included in the second text corresponding to the first text which includes the segment, and

the (b) step comprises further deriving the degree using the inclusion score computed in the (m) step, such that the degree increase the higher the inclusion score.

29. The non-transitory computer-readable recording medium according to claim 27,

wherein the (b) step comprises deriving, as the extent of inclusion in the second text, a percentage at which the content of the homogeneous segment is described in the second text corresponding to the other first text which includes the homogeneous segment, further deriving a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, such that the degree increases the higher the percentage, and performing the determination using the derived degree.

30. The non-transitory computer-readable recording medium according to claim 29,

wherein the (l) step, in addition to the specification of the homogeneous segment, comprises computing a similarity score representing an extent of similarity between each of the plurality of segments constituting the first text which is set as the analysis target and the homogeneous segment, and

the (b) step comprises further deriving the degree such that the degree increases the higher the similarity score computed in the (l) step.

31. The non-transitory computer-readable recording medium according to claim 27,

wherein the (b) step comprises deriving, as the extent of inclusion in the second text, a frequency at which the content of the homogeneous segment is described in the second text corresponding to the other first text which includes the homogeneous segment, further deriving a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, such that the degree increases the higher the frequency, and performing the determination using the derived degree.

32. The non-transitory computer-readable recording medium according to claim 25,

wherein the computer executes, as the (a) step, the steps of:

(x) determining, for the plurality of segments respectively constituting all of the first texts, whether a content of each segment is included in the second text corresponding to the first text which includes the segment, and

(y) specifying, by contrasting each of the plurality of segments constituting the first text which is set as the analysis target and a segment that is determined in the (x) step to be included in the second text and that is included in another first text, a segment similar to any of the plurality of segments constituting the first text which is set as the analysis target, and setting the specified segment as a homogeneous segment.

33. The non-transitory computer-readable recording medium according to claim 32,

wherein the (b) step comprises deriving, based on the result of the determination in the (a) step, an extent to which the content of the homogeneous segment is included in the second text corresponding to the other first text which includes the homogeneous segment, further deriving, based on the derived extent, a degree to which each segment constituting the first text which is set as the analysis target should be described in the second text corresponding to the first text which is set as the analysis target, and performing the determination using the degree.

34. The non-transitory computer-readable recording medium according to claim 33,

wherein the (x) step, in addition to the determination regarding the content of each segment, comprises computing, for the plurality of segments respectively constituting all of the first texts, an inclusion score representing a possibility of the content of each segment being included in the second text corresponding to the first text which includes the segment, and

the (b) step further comprises deriving the degree using the inclusion score computed in the (x) step, such that the degree increase the higher the inclusion score.

35. The non-transitory computer-readable recording medium according to claim 25, causing the computer to execute the step of:

(c) segmenting the first texts based on at least one of a sentence, a paragraph, a topic and a phase included therein, and setting the plurality of segments.

36. The non-transitory computer-readable recording medium according to claim 25,

setting of the plurality of segments constituting the first texts is performed based on at least one of a sentence, a paragraph, a topic and a phase included in the first text.