Determining off-topic questions in a question answering system using probabilistic language models

• US 10,713,438 B2
• Filed: 06/15/2016
• Issued: 07/14/2020
• Est. Priority Date: 03/23/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bufe, John P., Gadde, Srinivasa Phani K., Goth, III, Julius
• Primary Examiner(s): Sitiriche, Luis A
• Application Number:
  US15/182,818
• Publication Number:
  US 20160300154A1
• Time in Patent Office: 1,490 Days
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 5/045 Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence

Abstract

A question answering system that determines whether a question is off-topic by performing the following steps: (i) receiving, by a question answering system, a set of documents; (ii) identifying topical subset(s) for each document of the set of documents using named entity recognition, where each topical subset relates to a corresponding topic; (iii) assigning a set of topic score(s) for each topical subset using natural language processing, where each topic score relates to a corresponding probability associated with the respective topical subset under a probabilistic language model; and (iv) determining, based, at least in part, on the topic score(s) corresponding to the topical subset(s), whether a question input into the question answering system is off-topic.

Claims

1. A computer-implemented method comprising:

receiving, by a question answering system adapted to answer questions for a particular domain, an input question;

determining;

(i) a first logarithm probability associated with the input question under an on-topic probabilistic language model for the particular domain, and (ii) a second logarithm probability associated with the input question under an off-topic probabilistic language model for the particular domain;

modelling, using a sigmoid function, a difference between the first logarithm probability and the second logarithm probability;

assigning, based, at least in part, on the modelled difference, a topic score to the input question; and

determining, based, at least in part, on the assigned topic score, whether the input question is off-topic with respect to the particular domain.

2. The method of claim 1, wherein the determining of whether the input question is off-topic with respect to the particular domain includes comparing the assigned topic score to a relevance score associated with an answer generated by the question answering system in response to the input question.

3. The method of claim 2, further comprising determining the input question is off-topic based, at least in part, on the assigned topic score being greater than the associated relevance score for the generated answer.

4. The method of claim 2, further comprising determining the input question is on-topic based, at least in part, on the assigned topic score being less than the associated relevance score for the generated answer.

5. The method of claim 1, further comprising:

responsive to determining that the input question is off-topic, informing a user who input the input question into the question answering system that the input question is off-topic.

6. The method of claim 1, wherein the on-topic probabilistic language model is trained using questions that have been identified as being on-topic with respect to the particular domain, and wherein the off-topic probabilistic language model is trained using questions that have been identified as being off-topic with respect to the particular domain.

7. A computer program product comprising a computer readable storage medium having stored thereon:

program instructions to receive, by a question answering system adapted to answer questions for a particular domain, an input question;

program instructions to determine;

(i) a first logarithm probability associated with the input question under an on-topic probabilistic language model for the particular domain, and (ii) a second logarithm probability associated with the input question under an off-topic probabilistic language model for the particular domain;

program instructions to model, using a sigmoid function, a difference between the first logarithm probability and the second logarithm probability;

program instructions to assign, based, at least in part, on the modelled difference, a topic score to the input question; and

program instructions to determine, based, at least in part, on the assigned topic score, whether the input question is off-topic with respect to the particular domain.

8. The computer program product of claim 7, wherein determining whether the input question is off-topic with respect to the particular domain includes comparing the assigned topic score to a relevance score associated with an answer generated by the question answering system in response to the input question.

9. The computer program product of claim 8, the computer readable storage medium having further stored thereon program instructions to determine the input question is off-topic based, at least in part, on the assigned topic score being greater than the associated relevance score for the generated answer.

10. The computer program product of claim 8, the computer readable storage medium having further stored thereon program instructions to determine the input question is on-topic based, at least in part, on the assigned topic score being less than the associated relevance score for the generated answer.

11. The computer program product of claim 7, the computer readable storage medium having further stored thereon program instructions to, responsive to determining that the input question is off-topic, inform a user who input the input question into the question answering system that the input question is off-topic.

12. The computer program product of claim 7, wherein the on-topic probabilistic language model is trained using questions that have been identified as being on-topic with respect to the particular domain, and wherein the off-topic probabilistic language model is trained using questions that have been identified as being off-topic with respect to the particular domain.

13. A computer system comprising:

a processor(s) set; and

a computer readable storage medium;

wherein;

the processor set is structured, located, connected and/or programmed to run program instructions stored on the computer readable storage medium; and

the stored program instructions include;

program instructions to receive, by a question answering system adapted to answer questions for a particular domain, an input question;

program instructions to determine;

(i) a first logarithm probability associated with the input question under an on-topic probabilistic language model for the particular domain, and (ii) a second logarithm probability associated with the input question under an off-topic probabilistic language model for the particular domain;

program instructions to model, using a sigmoid function, a difference between the first logarithm probability and the second logarithm probability;

program instructions to assign, based, at least in part, on the modelled difference, a topic score to the input question; and

program instructions to determine, based, at least in part, on the assigned topic score, whether the input question is off-topic with respect to the particular domain.

14. The computer system of claim 13, wherein determining whether the input question is off-topic with respect to the particular domain includes comparing the assigned topic score to a relevance score associated with an answer generated by the question answering system in response to the input question.

15. The computer system of claim 14, the stored program instructions further including program instructions to determine the input question is off-topic based, at least in part, on the assigned topic score being greater than the associated relevance score for the generated answer.

16. The computer system of claim 14, the stored program instructions further including program instructions to determine the input question is on-topic based, at least in part, on the assigned topic score being less than the associated relevance score for the generated answer.

17. The computer system of claim 13, the stored program instructions further including program instructions to, responsive to determining that the input question is off-topic, inform a user who input the input question into the question answering system that the input question is off-topic.

18. The computer system of claim 13, wherein the on-topic probabilistic language model is trained using questions that have been identified as being on-topic with respect to the particular domain, and wherein the off-topic probabilistic language model is trained using questions that have been identified as being off-topic with respect to the particular domain.