Information extraction apparatus and methods

• US 8,495,042 B2
• Filed: 10/10/2008
• Issued: 07/23/2013
• Est. Priority Date: 10/10/2007
• Status: Expired due to Fees

• Current Assignee: ITI Scotland Limited (Scottish Government)
• Original Assignee: ITI Scotland Limited (Scottish Government)
• Inventors: Symington, Beatrice
• Primary Examiner(s): THAI, HANH B
• Application Number:
  US12/682,648
• Publication Number:
  US 20110099184A1
• Time in Patent Office: 1,747 Days
• Field of Search: 707600-602, 707 45- 46
• US Class Current: 707/702
• CPC Class Codes: G06F 16/34 Browsing; Visualisation therefor

Abstract

Automatic information extraction apparatus for extracting data for review by a human curator from digital representations of documents comprising natural language text, the automatic information extraction apparatus having a plurality of selectable operating modes in which the automatic information extraction apparatus is operable to extract different data for review by a human curator. In the different operating modes, the information extraction apparatus may extract data with a different balance between recall and precision.

Claims

1. A non-transitory information extraction computing apparatus including at least one processor with accessible input/output and at least one data store, said at least one processor being programmed for extracting data for review by a human curator from digital representations of documents comprising natural language text, the information extraction computing apparatus, in use, executing computer program instructions that cause the apparatus to provide a plurality of selectable operating modes in which the automatic information extraction apparatus extracts different data for review by a human curator, and, in at least two of the plurality of selectable operating modes, to extract data with different balances between precision and recall, whereby a balance that favours precision over recall will lead to fewer incorrect instances of data being extracted than a balance that favours recall over precision, but will omit more data which should have been extracted.

2. Automatic information extraction apparatus according to claim 1, wherein the automatic information extraction apparatus comprises a plurality of alternative information extraction modules which are operable to cause different data to be extracted for review.

3. Automatic information extraction apparatus according to claim 2, wherein alternative information extraction modules comprise the same computer program module but different parameter data which specifies the functionality of the computer program module.

4. Automatic information extraction apparatus according to claim 2, wherein the alternative information extraction modules are named entity recognition modules.

5. Automatic information extraction apparatus according to claim 4, wherein the alternative named entity recognition modules comprise the same software modules which implements a named entity recognition algorithm and different parameter data which determines the function of the named entity recognition algorithm.

6. Automatic information extraction apparatus according to claim 1, wherein the automatic information extraction apparatus is operable to identify individual instances of entities in a digital representation of a document.

7. Automatic information extraction apparatus according to claim 6, wherein the automatic information extraction apparatus is operable to ensure that each individual instance of a character string which is identified as an entity is consistently identified as the same entity within the same digital representation of a document.

8. Automatic information extraction apparatus according to claim 6, wherein the automatic information extraction apparatus is operable to ensure that each individual instance of a character string which is identified as a particular type of entity is consistently identified as the same type of entity, respectively, within the same digital representation of a document.

9. Automatic information extraction apparatus according to claim 1, which is operable to apply different algorithms to force consistent extraction of data concerning entities represented by identical character strings within a document in two different information extraction modes.

10. Automatic information extraction apparatus according to claim 1, wherein the automatic information extraction apparatus is operable to receive data concerning the human curator who will review data extracted from a particular digital representation of a document, or preferences associated with the human curator who will review the data extracted from a particular digital representation of a document, and to select an operating mode from amongst the plurality of operating modes for the extraction of data from that particular digital representation of a document.

11. Information extraction and curation apparatus comprising automatic information extraction apparatus according to claim 1 and a curation computer-user interface which is operable to present extracted data to a human curator for review and possible amendment.

12. Information extraction and curation apparatus according to claim 11, operable to analyse the interaction between one or more human curators and the curation computer-user interface in order to allocate an operating mode for subsequent information extraction by a particular human curator, or to modify an information extraction mode for subsequent information extraction, or to measure the performance of an information extraction mode.

13. Information extraction and curation apparatus according to claim 11, operable to measure one or more metrics related to the speed of a curator.

14. Information extraction and curation apparatus according to claim 11, wherein the curation computer-user interface is operable to display a digital representation of a document to a user with identified entities and/or relations highlighted in the display digital representation of a document.

15. A method of extracting data for review by a human curator from digital representations of documents comprising natural language text, comprising providing automatic information extraction apparatus according to claim 1 or information extraction and curation apparatus, selecting an operating mode from amongst the plurality of operating modes and causing the automatic information extraction apparatus to function in the selected operating mode to extract data for review by a human curator.

16. A method of extracting data according to claim 15, wherein the operating mode is selected dependent on the human curator, from a group of human curators, who is to review the extracted data.

17. A method of extracting data according to claim 16, comprising carrying out information extraction in a plurality of operating modes to extract data for review by a plurality of human curators.

18. A method of extraction data according to claim 15, comprising receiving instructions as to the information extraction mode to be employed to extract data from a specific batch of digital representations of documents and then extracting data from the specified batch of digital representations of documents using the specific information extraction module.

24. A computer readable carrier storing computer program instructions which, when executed on computing apparatus, cause the computing apparatus to function as the automatic information extraction apparatus of claim 1, or the information extraction and curation apparatus, or which cause the computing apparatus to carry out the methods.

19. A non-transitory computer readable carrier tangibly embodying computer program instructions which, when executed by a computing apparatus, cause the computing apparatus to carry out a method of optimising an automatic information extraction apparatus that extracts data for review by a human curator from digital representations of documents comprising natural language text, the method comprising the steps of:

(i) extracting data from at least one digital representation of a document comprising natural language text, using an information extraction module;

(ii) providing a computer-user interface which presents the extracted data to a human curator for review and which analyses the interactions between the human curator and the computer-user interface; and

(iii) modifying the information extraction module responsive to the analysed interactions to facilitate an improvement in the subsequent performance of a or the human curator using the information extraction module and computer-user interface to review extracted data, the information extraction module being modified so as to change the balance between precision and recall of the information extraction module, whereby a balance that favours precision over recall will lead to fewer incorrect instances of data being extracted than a balance that favours recall over precision, but will omit more data which should have been extracted.

20. A computer readable carrier according to claim 19, wherein the analysis of the interactions between the human curator and the computer-user interface comprises measuring one or more metrics related to the speed of a curator.

21. A computer readable carrier according to claim 19, wherein the analysis of the interactions between the human curator and the computer-user interface comprises measuring a metric related to the number of searches made by a curator using a search engine which is operable to find, edit or check extracted data.

22. A computer readable carrier according to claim 19, wherein the information extraction module is operable to identify entities and/or relations in a digital representation of a document and the computer-user interface is operable to display at least a part of the digital representation of a document with identified entities and/or relations highlighted therein.

23. A computer readable carrier according to claim 19, wherein the modification to the information extraction module is an optimisation.