Natural question generation from query data using natural language processing system

• US 10,423,649 B2
• Filed: 04/06/2017
• Issued: 09/24/2019
• Est. Priority Date: 04/06/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ko, William S., Kossarian, Malous M., Smith, Douglas A., Zhu, Huaiyu
• Primary Examiner(s): Robinson, Greta L
• Application Number:
  US15/480,780
• Publication Number:
  US 20180293302A1
• Time in Patent Office: 901 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A training query generation system is usable to generate fully formed training questions from prior search queries, some of which may be fully formed search queries and some of which are not fully formed. The system may identify fully formed questions from a query database stored on a storage device. The query database includes a plurality of search query character string, The system further identifies partially formed questions from the query database, creates question templates from the identified fully formed questions, and stores the question templates in a template database. The system then identifies entities with the partially formed questions, classifies each entity, and stores the classified entities in an entity database. Fully formed questions can then be generated using the question templates from the template database and classified entities from the entity database.

Claims

1. A computer-implemented method, comprising:

identifying fully formed questions from a query database stored on a storage device, the query database containing a plurality of search query character strings;

identifying partially formed questions from the query database;

creating question templates from the identified fully formed questions;

storing the question templates in a template database;

identifying entities with the partially formed questions and classifying each entity;

storing the classified entities in an entity database; and

generating fully formed questions using the question templates from the template database and classified entities from the entity database.

2. The computer-implemented method of claim 1, wherein identifying the fully formed questions comprises identifying those search query character strings that end with a question mark.

3. The computer-implemented method of claim 1, wherein identifying the fully formed questions comprises:

parsing each search query character string to extract semantic roles to each portion of each such query character string; and

identifying those search query character strings for which an auxiliary verb syntactic role precedes a subject syntactic role.

4. The computer-implemented method of claim 1, wherein identifying the fully formed questions comprises identifying those search query character strings which begin with an interrogative word.

5. The computer-implemented method of claim 1, wherein, for each identified fully formed question, creating a question template includes extracting semantic roles on the fully formed question, identifying entity types of these roles, and modifying the fully formed question to replace the extracted semantic roles with their corresponding entity type labels.

6. The computer-implemented method of claim 1, wherein identifying entities with the partially formed questions and classifying each entity comprises extracting entities and identifying their types from the partial query strings.

7. The computer-implemented method of claim 1, further comprising

receiving an uniform resource locator (URL) of a web page accessed following submission of a particular search query character string;

accessing a URL-to-topic mappings database containing mappings between URLs and associated web page topics to determine a topic associated with the received URL; and

adding a record to the query database including the particular search query character string and the associated topic.

8. The computer-implemented method of claim 7, further comprising including a timestamp in the record.

9. The computer-implemented method of claim 7, wherein storing the question templates in the template database includes, for each question template, retrieving a timestamp from the query database associated with the record from which the question template was created and writing the timestamp to the template database.

10. A system, comprising:

one or more computing devices configured to execute a query service and a natural language service;

one or more storage devices configured to store a query database, a template database and an entity database;

wherein the query service is configured to;

receive a search query character string;

generate a topic identifier associated with the search query character string; and

add a record to the query database including the search query character string, the topic identifier, and a time stamp;

wherein natural language service is configured to;

retrieve a plurality of records from the query database;

for each record, determine whether the search query character string is a fully formed question or a partially formed question;

for a fully formed question, replace a portion of the fully formed question with an argument value indicative of a role associated with the replaced portion of the fully formed question to generate a query template and store the query template in the template database; and

for a partially formed question, identify an entity, classify each identified entity, and store the entity and a classification for the entity in the entity database.

11. The system of claim 10, wherein the one or more computing devices are configured to execute a training dataset generation service, which is configured to generate fully formed questions using query templates from the template database and the entities and their associated classifications from the entity database.

12. The system of claim 10, wherein the one or more storage devices includes a uniform resource locator (URL)-to-topic mappings database containing mappings between URLs and associated web page topics, and wherein the query service is configured to:

receive a URL of a web page accessed from a list of search query results following submission of a particular search query character string;

access the URL-to-topic mappings database to determine a topic associated with the received URL; and

add a record to the query database including the particular search query character string and the associated topic.

13. The system of claim 10, wherein the natural language service is configured to identify fully formed questions from the query database through at least one technique selected from the group consisting of:

identification of those search query character strings that end with a question mark;

identification of those search query character strings for which an auxiliary verb syntactic role precedes a subject syntactic role; and

identification of those search query character strings which begin with an interrogative word.

14. The system of claim 10, wherein, for each determined fully formed question, the natural language service is configured to create a question template through execution of a slot language parser on the fully formed question and modification of the fully formed question to replace at least one portion of the fully formed question with an argument value indicative of a role associated with the replaced portion of the fully formed question.

15. A computer program product for computer generation of training questions for training a language processing system, said computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

from a plurality of search query character strings stored in a storage device, identify fully formed questions and identify partially formed questions;

create question templates from the identified fully formed questions and store the question templates in a template database;

identify entities within the partially formed questions, classify each entity, and store the classified entities in an entity database; and

generate fully formed questions through insertion of entities from the entity database into the question templates from the template database.

16. The computer program product of claim 15, wherein the program instructions further cause the processor to identify fully formed questions from a query database through at least one technique selected from the group consisting of:

identification of those search query character strings that end with a question mark;

identification of those search query character strings for which an auxiliary verb syntactic role precedes a subject syntactic role; and

identification of those search query character strings which begin with an interrogative word.

17. The computer program product of claim 15, wherein, for each identified fully formed question, the program instructions further cause the processor to create a corresponding question template through execution of a slot language parser on the fully formed question and replacement of at least one portion of the fully formed question with an argument value indicative of a role associated with the replaced portion of the fully formed question.

18. The computer program product of claim 15, wherein the program instructions further cause the processor to receive an uniform resource locator (URL) of web page accessed following submission of a particular search query character string, access a database containing mappings between URLs and associated web page topics to determine a topic associated with the received URL, and add a record to a query database including the particular search query character string and the associated topic and a timestamp.

19. The computer program product of claim 18, wherein the program instructions further cause the processor to create a question template from the record, retrieve the timestamp from the query database associated with the record from which the question template was created, and write the timestamp, the associated topic, and the question template to the template database.

20. The computer program product of claim 19, wherein the program instructions cause the processor to generate the fully formed questions by receiving a request containing a topic identifier, and by accessing the template database to retrieve those question templates associated with the same topic identifier.