SYSTEMS AND METHODS FOR INFORMATION EXTRACTION USING CONTEXTUAL PATTERN DISCOVERY

• US 20120303661A1
• Filed: 05/27/2011
• Published: 11/29/2012
• Est. Priority Date: 05/27/2011
• Status: Active Grant

• Current Assignee: Beijing Zitiao Network Technology Company Limited (ByteDance Ltd.)
• Original Assignee: International Business Machines Corporation
• Inventors: Ho, Ching-Tien, Li, Yunyao, Zhu, Huaiyu, Blohm, Sebastian Johannes, Chu, Vivian Yaw-Wen
• Granted Patent: US 8,630,989 B2
• Time in Patent Office: Days
• US Class Current: 707/776
• CPC Class Codes: G06F 40/295 Named entity recognition

Abstract

Described herein are methods, systems, apparatuses and products for automatically discovering patterns in a text corpus. An aspect provides extracting at least one context string related to at least one annotator from the at least one text corpus; analyzing the at least one context string for at least one sequence, the at least one sequence comprised of at least one subsequence; determining at least one sequence signature for each at least one sequence by applying applicable rules to the at least one sequence; and grouping the at least one sequence signature into at least one group.

Claims

1. A system comprising:

at least one processor; and

a memory device operatively connected to the at least one processor;

wherein, responsive to execution of program instructions accessible to the at least one processor and configured to automatically discover at least one pattern in at least one text corpus, the at least one processor is configured to;

extract at least one context string related to at least one annotator from the at least one text corpus;

analyze the at least one context string for at least one sequence, the at least one sequence comprised of at least one subsequence;

determine at least one sequence signature for each at least one sequence by applying applicable rules to the at least one sequence; and

group the at least one sequence signature into at least one group corresponding to at least one sequence signature.

2. The system according to claim 1, wherein the rules are based on rule configuration options and statistics relating the at least one subsequence to the at least one text corpus.

3. The system according to claim 2, wherein the statistics relating the at least one subsequence to the at least one text corpus comprise a number of instances of a subsequence in the at least one text corpus.

4. The system according to claim 2, wherein the statistics relating the at least one subsequence to the at least one text corpus comprise a measure of correlation between a pair of subsequences.

5. The system according to claim 4, wherein the measure of correlation comprises an uncertainty coefficient of the pair of subsequences.

6. The system according to claim 1, wherein at least one configurable grouping option is provided for grouping the at least one sequence signature into at least one group, the at least one configurable grouping option comprising running the applicable rules a specific number of times.

7. The system according to claim 1, wherein extracting at least one context string comprises querying the at least one text corpus using at least one structured query.

8. The system according to claim 7, wherein the at least one structured query comprises at least one annotator and at least one context condition for defining context.

9. The system according to claim 1, wherein at least one configurable sequence analysis option is specified for analyzing the at least one context string for at least one sequence.

10. The system according to claim 9, wherein the at least one configurable sequence analysis option comprises a number of instances of the at least one subsequence of the at least one sequence in the at least one context string.

11. A method comprising:

automatically discovering at least one pattern in at least one text corpus, wherein discovering at least one pattern comprises;

extracting at least one context string related to at least one annotator from the at least one text corpus;

analyzing the at least one context string for at least one sequence, the at least one sequence comprised of at least one subsequence;

determining at least one sequence signature for each at least one sequence by applying applicable rules to the at least one sequence; and

grouping the at least one sequence into at least one group corresponding to at least one sequence signature.

12. The method according to claim 11, wherein the rules are based on rule configuration options and statistics relating the at least one subsequence to the at least one text corpus.

13. The method according to claim 12, wherein the statistics relating the at least one subsequence to the at least one text corpus comprise a number of instances of a subsequence in the at least one text corpus.

14. The method according to claim 12, wherein the statistics relating the at least one subsequence to the at least one text corpus comprise a measure of correlation between a pair of subsequences.

15. The method according to claim 14, wherein the measure of correlation comprises an uncertainty coefficient of the pair of subsequences.

16. The method according to claim 11, wherein at least one configurable grouping option is provided for grouping the at least one sequence signature into at least one group, the at least one configurable grouping option comprising running the applicable rules a specific number of times.

17. The method according to claim 11, wherein:

extracting at least one context string comprises querying the at least one text corpus using at least one structured query; and

the at least one text corpus is organized as a trie data structure.

18. The method according to claim 11, wherein at least one configurable sequence analysis option is specified for analyzing the at least one context string for at least one sequence.

19. The method according to claim 18, wherein the at least one configurable sequence analysis option comprises a number of instances of the at least one subsequence of the at least one sequence in the at least one context string.

20. A computer program product comprising:

a computer readable storage medium having computer readable program code configured to automatically discover at least one pattern in at least one text corpus embodied therewith, the computer readable program code comprising;

computer readable program code configured to extract at least one context string related to at least one annotator from the at least one text corpus;

computer readable program code configured to analyze the at least one context string for at least one sequence, the at least one sequence comprised of at least one subsequence;

computer readable program code configured to determine at least one sequence signature for each at least one sequence by applying applicable rules to the at least one sequence; and

computer readable program code configured to group the at least one sequence into at least one group corresponding to at least one sequence signature.