System method and computer program product for obtaining structured data from text

• US 20040163043A1
• Filed: 02/10/2003
• Published: 08/19/2004
• Est. Priority Date: 02/10/2003
• Status: Active Grant

• Current Assignee: Kaidara
• Original Assignee: Kaidara
• Inventors: Madec, Olivier Camille, Baudin, Catherine Marie, Manago, Michel Vladimir
• Granted Patent: US 7,174,507 B2
• Time in Patent Office: Days
• US Class Current: 715/513
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing

Abstract

A method, system, and computer program product for obtaining structured data from text includes inputting text to be transformed into structured data, defining at least one criterion relating to content of the text, applying a first matching method to the text to identify a value in the text for each of the criterion, and applying a second matching method to the text to identify a value in the text for each of the criterion that the first matching method fails to identify a value for. Each of the criterion is then associated with a value obtained by the first or second matching method to create at least one criterion-value pair, and structured data is created based on the at least one criterion-value pair. The first matching method has a higher reliability than the second matching method.

Claims

1. A method of transforming text into structured data, comprising:

inputting text to be transformed into structured data;

defining at least one criterion relating to content of said text;

applying a first matching method to said text to identify a value in said text for each of said criterion;

applying a second matching method to said text to identify a value in said text for each of said criterion that said first matching method fails to identify a value for;

associating each of said criterion with a value obtained by said first or second matching method to create at least one criterion-value pair; and

creating structured data based on said at least one criterion-value pair, wherein said first matching method has a higher reliability than said second matching method.

2. The method of claim 1, wherein said inputting text comprises using a parsing script to:

decompose said text into a series of words and separators sequenced in the order of the apparition order in the text; and

mark stop words and common words based on a list of words in the parsing script.

3. The method of claim 1, wherein said defining a criterion comprises inputting each of said criterion on the fly in response to a user input.

4. The method of claim 3, further comprising inputting at least one initial value corresponding to any of said criterion in response to a user input, said initial value being used in at least one of said first and second matching methods.

5. The method of claim 1, wherein said defining a criterion comprises importing at least one of said criterion from a domain model.

6. The method of claim 5, further comprising importing an initial value corresponding to any of said criterion from said domain model, said initial value being used in at least one of said first and second matching methods.

7. The method of claim 6, further comprising importing at least one synonym for any value or initial value from said domain model, said synonym being used in at least one of said first and second matching methods.

8. The method of claim 5, further comprising importing a criticality weight corresponding to any of said criterion from said domain model, said criticality weight being used in at least one of said first and second matching methods.

9. The method of claim 5, further comprising importing a relationship corresponding to any two of said criterion, said relationship being used in at least one of said first and second matching methods.

10. The method of claim 1, wherein said applying a first matching method comprises applying a pattern-based matching method to said text to identify a value in said text for each of said criterion.

11. The method of claim 10, wherein said applying a pattern-based matching method comprises applying an extraction rule that which does not require an initial value to said text to identify a value in said text for said criterion.

12. The method of claim 10, wherein said applying a pattern-based matching method comprises applying a constrained extraction rule that require an initial value to said text to identify a value in said text for said criterion.

13. The method of claim 12, further comprising importing said initial value from a domain model.

14. The method of claim 10, wherein said applying a pattern-based matching method comprises applying an action rule to said text to identify a value in said text for said criterion.

15. The method of claim 14, further comprising inputting a user action corresponding to said action rule.

16. The method of claim 1, wherein said applying a second matching method comprises applying an ontology-based matching method requiring initial values to said text to identify a value in said text for each of said criterion.

17. The method of claim 16, further comprising inputting at least one of said initial values as a value returned by said first matching method.

18. The method of claim 16, further comprising importing said initial values from a domain model.

19. The method of claim 16, further comprising defining a tolerance level to control a degree of closeness between said text and said initial values that is necessary to identify a value in said text for said criterion.

20. The method of claim 19, wherein said defining a tolerance level comprises defining at least one of a word similarity threshold and a global similarity threshold.

21. The method of claim 19, wherein said defining a tolerance level comprises defining a proximity window for said ontology-based matching method.

22. The method of claim 1, further comprising detecting a conflict when a word in the input text matches more than one of said criterion.

23. The method of claim 22, further comprising resolving said conflict by assigning a can reuse property to each of said criterion.

24. The method of claim 22, further comprising resolving said conflict by assigning an ignore conflicts property to said method.

25. The method of claim 1, further comprising assigning a confidence level to any of said criterion-value pairs.

26. The method of claim 25, wherein said assigning a confidence level comprises assigning a confidence level based on at least one of said reliability factor of the matching method used to create said any of said criterion-value pairs, a recognition performance of said any of the criterion-value pairs, and a degree of ambiguity in said any of the criterion-value pairs.

27. The method of claim 26, further comprising assigning a matching weight to said any of the criterion-value pairs based on said confidence level of said any of the criterion-value pairs and a length of the value in said any of said criterion-value pairs.

28. The method of claim 27, wherein said creating structured data comprises defining said any of the criterion-value pairs as structured data if said matching weight of said any of the criterion-value pairs is greater than a predetermined weight threshold.

29. The method of claim 28, further comprising generating a clarifying question for any of said criterion-value pairs having a matching less than said weight threshold.

30. The method of claim 28, further comprising generating a validation set for any of said criterion-value pairs having a matching less than said weight threshold.

31. The method of claim 1, further comprising propagating values of said structured data to a domain model.

32. The method of claim 31, wherein said propagating comprises creating an initial domain model with said values.

33. The method of claim 31, wherein said propagating comprises updating a preexisting domain model with said values.

34. The method of claim 31, further comprising synchronizing said first and second matching methods with said domain model.

35. The method of claim 34, wherein said synchronizing comprises preventing text terms used to obtain said values from being used to provide other values.

36. The method of claim 34, wherein said synchronizing comprises deactivating values in said model based on relationships between criterion.

37. The method of claim 34, further comprising reevaluating said text using said synchronized methods.

38. The method of claim 1, wherein said first and second matching methods are different types of matching methods.

39. The method of claim 1, wherein said first and second matching methods are a same type of matching method having different levels of reliability.

40. A computer readable medium containing program instructions for execution on a computer system, which when executed on the computer system, cause the computer system to perform the steps in the method recited in any one of claims 1-39.

80. The method of claim 1, further comprising setting different levels of user interactivity with the text transformation method.

81. The method of claim 7, further comprising attaching said synonyms to values, wherein said synonyms are strict or loose synonyms.

41. A system comprising:

a memory device having embodied therein, data related to transforming text into structured data; and

a processor in communication with said memory device, said processor configured to cause the system to;

input text to be transformed into structured data;

define at least one criterion relating to content of said text;

apply a first matching method to said text to identify a value in said text for each of said criterion;

apply a second matching method to said text to identify a value in said text for each of said criterion that said first matching method fails to identify a value for;

associate each of said criterion with a value obtained by said first or second matching method to create at least one criterion-value pair; and

create structured data based on said at least one criterion-value pair, wherein said first matching method has a higher reliability than said second matching method.

42. The system of claim 41, wherein said processor is configured to input text by using a parsing script to:

decompose said text into a series of words and separators sequenced in the order of the apparition order in the text; and

mark stop words and common words based on a list of words in the parsing script.

43. The system of claim 41, wherein said processor is configured to define a criterion by inputting each of said criterion on the fly in response to a user input.

44. The system of claim 43, wherein said processor is further configured to input at least one initial value corresponding to any of said criterion in response to a user input, said initial value being used in at least one of said first and second matching methods.

45. The system of claim 41, wherein said processor is configured to define a criterion by importing at least one of said criterion from a domain model.

46. The system of claim 45, wherein said processor is further configured to import an initial value corresponding to any of said criterion from said domain model, said initial value being used in at least one of said first and second matching methods.

47. The system of claim 46, wherein said processor is further configured to import at least one synonym for any value or initial value from said domain model, said synonym being used in at least one of said first and second matching methods.

48. The system of claim 45, wherein said processor is further configured to import a criticality weight corresponding to any of said criterion from said domain model, said criticality weight being used in at least one of said first and second matching methods.

49. The system of claim 45, wherein said processor is further configured to import a relationship corresponding to any two of said criterion, said relationship being used in at least one of said first and second matching methods.

50. The system of claim 41, wherein said processor is configured to apply a first matching method by applying a pattern-based matching method to said text to identify a value in said text for each of said criterion.

51. The system of claim 50, wherein said processor is configured to apply a pattern-based matching method by applying an extraction rule that which does not require an initial value to said text to identify a value in said text for said criterion.

52. The system of claim 50, wherein said processor is configured to apply a pattern-based matching method by applying a constrained extraction rule that require an initial value to said text to identify a value in said text for said criterion.

53. The system of claim 52, wherein said processor is further configured to import said initial value from a domain model.

54. The system of claim 50, wherein said processor is configured to apply a pattern-based matching method by applying an action rule to said text to identify a value in said text for said criterion.

55. The system of claim 54, wherein said processor is further configured to input a user action corresponding to said action rule.

56. The system of claim 41, wherein said processor is configured to apply a second matching method by applying an ontology-based matching method requiring initial values to said text to identify a value in said text for each of said criterion.

57. The system of claim 56, wherein said processor is further configured to input at least one of said initial values as a value returned by said first matching method.

58. The system of claim 56, wherein said processor is further configured to import said initial values from a domain model.

59. The system of claim 56, wherein said processor is further configured to define a tolerance level to control a degree of closeness between said text and said initial values that is necessary to identify a value in said text for said criterion.

60. The system of claim 59, wherein said processor is configured to define a tolerance level by defining at least one of a word similarity threshold and a global similarity threshold.

61. The system of claim 59, wherein said processor is configured to define a tolerance level by defining a proximity window for said ontology-based matching method.

62. The system of claim 41, wherein said processor is further configured to detect a conflict when a word in the input text matches more than one of said criterion.

63. The system of claim 62, wherein said processor is further configured to resolve said conflict by assigning a can reuse property to each of said criterion.

64. The system of claim 62, wherein said processor is further configured to resolve said conflict by assigning an ignore conflicts property to said method.

65. The system of claim 41, wherein said processor is further configured to assign a confidence level to any of said criterion-value pairs.

66. The system of claim 65, wherein said processor is configured to assign a confidence level by assigning a confidence level based on at least one of said reliability factor of the matching method used to create said any of said criterion-value pairs, a recognition performance of said any of the criterion-value pairs, and a degree of ambiguity in said any of the criterion-value pairs.

67. The system of claim 66, wherein said processor is further configured to assign a matching weight to said any of the criterion-value pairs based on said confidence level of said any of the criterion-value pairs and a length of the value in said any of said criterion-value pairs.

68. The system of claim 67, wherein said processor is configured to create structured data by defining said any of the criterion-value pairs as structured data if said matching weight of said any of the criterion-value pairs is greater than a predetermined weight threshold.

69. The system of claim 68, wherein said processor is further configured to generate a clarifying question for any of said criterion-value pairs having a matching less than said weight threshold.

70. The system of claim 68, wherein said processor is further configured to generate a validation set for any of said criterion-value pairs having a matching less than said weight threshold.

71. The system of claim 41, wherein said processor is further configured to propagating values of said structured data to a domain model.

72. The system of claim 71, wherein said processor is configured to propagate by creating an initial domain model with said values.

73. The system of claim 71, wherein said processor is configured to propagate by updating a preexisting domain model with said values.

74. The system of claim 71, wherein said processor is further configured to synchronize said first and second matching methods with said domain model.

75. The system of claim 74, wherein said processor is configured to synchronize by preventing text terms used to obtain said values from being used to provide other values.

76. The system of claim 74, wherein said processor is configured to synchronize by deactivating values in said model based on relationships between criterion.

77. The system of claim 74, wherein said processor is further configured to reevaluate said text using said synchronized methods.

78. The system of claim 41, wherein said processor is configured to apply different types of matching methods for said first and second matching methods.

79. The system of claim 41, wherein said processor is configured to apply a same type of matching method for each of said first and second matching methods, each of said same type of matching method having different levels of reliability from each other same type of matching method.

82. A method of transforming text into structured data, comprising:

step for inputting text to be transformed into structured data;

step for defining at least one criterion relating to content of said text;

step for applying a first matching method to said text to identify a value in said text for each of said criterion;

step for applying a second matching method to said text to identify a value in said text for each of said criterion that said first matching method fails to identify a value for;

step for associating each of said criterion with a value obtained by said first or second matching method to create at least one criterion-value pair; and

step for creating structured data based on said at least one criterion-value pair, wherein said first matching method has a higher reliability than said second matching method.

83. A system for transforming text into structured data, comprising:

means for inputting text to be transformed into structured data;

means for defining at least one criterion relating to content of said text;

means for applying a first matching method to said text to identify a value in said text for each of said criterion;

means for applying a second matching method to said text to identify a value in said text for each of said criterion that said first matching method fails to identify a value for;

means for associating each of said criterion with a value obtained by said first or second matching method to create at least one criterion-value pair; and

means for creating structured data based on said at least one criterion-value pair, wherein said first matching method has a higher reliability than said second matching method.