Interactive question-answering apparatus and method thereof

• US 10,650,814 B2
• Filed: 10/30/2017
• Issued: 05/12/2020
• Est. Priority Date: 11/25/2016
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Wang, Ji Hyun, Kim, Hyun Ki, Lee, Chung Hee, Lim, Soo Jong, Choi, Mi Ran, Park, Sang Kyu, Bae, Yong Jin, Lee, Hyung Jik, Lim, Joon Ho, Jang, Myung Gil, Heo, Jeong
• Primary Examiner(s): Trapanese, William C
• Application Number:
  US15/797,354
• Publication Number:
  US 20180151178A1
• Time in Patent Office: 925 Days
• Field of Search: None
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/00 Handling natural language data speech analysis or synthesis, speech recognition G10L
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

Provided are an interactive question answering apparatus and method, which improve an unnatural question sentence and prevent the impossibility of a question in a state which does not know additional information, thereby providing a multimodal-based question answering method enabling a person to provide a question like conversing. Also, a question answering service is exposed to potential product purchasers of content such as an image, video, audio, etc., thereby providing an environment available to advertisement market.

Claims

1. An interactive question answering method in a user terminal including a computer processor communicating with a server over a communication network, the interactive question answering method comprising:

receiving, by the computer processor, multimedia content and meta-information about a plurality of entities, which are predicted to attract interest of a user in the multimedia content, from the server;

receiving, by the computer processor, a question sentence about an entity, selected from among the plurality of entities by the user, from a multimodal interface, the question sentence including insufficient hint information for obtaining an answer sentence;

extracting, by the computer processor, identification information about the entity selected by the user from the meta-information; and

transmitting, by the computer processor, the identification information about the entity and the question sentence to the server and receiving the answer sentence, including a right answer candidate constrained by the identification information about the entity, as the answer sentence to the question sentence from the server,

wherein the identification information is constructed to have an attribute of multimodal information output from the multimodal interface included in the user terminal, the attribute of multimodal information used as a hint information.

2. The interactive question answering method of claim 1, wherein the extracting comprises:

receiving multimodal input information for identifying the entity selected by the user from the multimodal interface;

searching for the identification information about the entity corresponding to the multimodal input information in the meta-information; and

extracting the found identification information from the meta-information.

3. The interactive question answering method of claim 1, wherein the extracting comprises:

when the multimedia content is image content displayed on a display screen of the user terminal,

receiving touch coordinates of the entity selected by the user from the multimodal interface; and

extracting the identification information about the entity, corresponding to the received touch coordinates, from the meta-information.

4. The interactive question answering method of claim 1, wherein the extracting comprises:

when the multimedia content is video content,

calculating an input time when a user voice corresponding to the question sentence is input from the multimodal interface, based on a reproduction start time of the video content; and

extracting a uniform resource identifier of an entity, reproduced at the calculated input time, from the meta-information.

5. The interactive question answering method of claim 1, wherein the extracting comprises:

when the multimedia content is audio content,

calculating an input time when a user voice corresponding to the question sentence is input from the multimodal interface, based on a reproduction start time of the audio content; and

extracting a uniform resource identifier of an entity, included in the audio content and reproduced at the calculated input time, from the meta-information.

6. The interactive question answering method of claim 1, wherein the identification information comprises a uniform resource identifier of each of the plurality of entities, coordinates of an area where each of the plurality of entities is displayed on a display screen of the user terminal, a time section where video content including the plurality of entities is reproduced, a time section where audio content including the plurality of entities is reproduced, and an attribute name representing an attribute of each of the plurality of entities.

7. The interactive question answering method of claim 1, wherein each of the plurality of entities comprises one or more selected from the group consisting of a place name, an accident, a production year, a person, a home appliance, an article of clothing, a name of an actor, a producer, an article of clothing on an actor, shoes, a bag, a price, and a color.

8. An interactive question answering method in a server including a computer processor communicating with a user terminal over a communication network, the interactive question answering method comprising:

generating, by the computer processor, meta-information including identification information about a plurality of entities which are predicted to attract interest of a user in multimedia content and attribute information assigned to the identification information;

transmitting, by the computer processor, the multimedia content and the meta-information to the user terminal;

receiving, by the computer processor, a question sentence about an entity selected from among the plurality of entities by the user and identification information about the entity, selected by the user and extracted from the meta-information, from the user terminal; and

generating, by the computer processor, an answer sentence to the question sentence, based on attribute information assigned to the identification information about the entity selected by the user and transmitting the answer sentence to the user terminal,

wherein the identification information is constructed to have an attribute of multimodal information output from a multimodal interface included in the user terminal, the attribute of multimodal information being used as a hint information.

9. The interactive question answering method of claim 8, wherein the generating of the meta-information comprises:

classifying the plurality of entities which are predicted to attract interest of the user in the multimedia content, based on a pre-learned entity classification model;

generating identification information about each of the classified plurality of entities, based on the multimodal interface; and

generating meta-information including the generated identification information.

10. The interactive question answering method of claim 9, wherein the generating of the identification information based on the multimodal interface comprises:

generating the identification information including an attribute of multimodal input information, for comparing the identification information with the multimodal input information output from the multimodal interface; and

generating the meta-information including the generated identification information.

11. The interactive question answering method of claim 8, wherein the identification information comprises a uniform resource identifier of each of the plurality of entities, screen coordinates at which each of the plurality of entities is displayed on a display screen, a time section where an entity included in video content is reproduced, a time section where an entity included in audio content is reproduced, and an attribute name representing an attribute of each of the plurality of entities.

12. The interactive question answering method of claim 8, wherein the transmitting of the answer sentence to the user terminal comprises:

analyzing a question sentence transmitted from the user terminal to recognize a question centric word in the question sentence;

determining attribute information about the question centric word, based on the identification information transmitted from the user terminal; and

selecting the attribute information as a right answer candidate to generate the answer sentence including the selected right answer candidate.

13. The interactive question answering method of claim 12, wherein the determining of the attribute value comprises:

searching a database storing the attribute information to search for a uniform resource identifier which is the same as a uniform resource identifier included in the identification information transmitted from the user terminal; and

when the same uniform resource identifier is checked in the database, determining attribute information, allocated to the checked uniform resource identifier, as an attribute of the question centric word.

14. The interactive question answering method of claim 8, wherein each of the plurality of entities comprises one or more selected from the group consisting of a place name, an accident, a production year, a person, a home appliance, an article of clothing, a name of an actor, a producer, an article of clothing on an actor, shoes, a bag, a price, and a color.

15. An interactive question answering apparatus including a server including a computer processor communicating with a user terminal over a communication network, the server comprising:

a storage unit storing multimedia content, meta-information including identification information about a plurality of entities which are predicted to attract interest of a user in the multimedia content, and attribute information allocated to the identification information; and

a computer processor transmitting the multimedia content and the meta-information to the user terminal, receiving a question sentence about an entity selected from among the plurality of entities by the user and identification information about the entity, selected by the user and extracted from the meta-information, from the user terminal, and generating an answer sentence to the question sentence to transmit the answer sentence to the user terminal, based on attribute information allocated to the identification information about the entity selected by the user,

wherein the identification information is constructed to have an attribute of multimodal information output from a multimodal interface included in the user terminal, the attribute of multimodal information being used as a hint information.

16. The interactive question answering apparatus of claim 15, wherein the computer processor classifies the plurality of entities which are predicted to attract interest of the user in the multimedia content, based on a pre-learned entity classification model, generates identification information about each of the classified plurality of entities, based on the multimodal interface, and generates meta-information including the generated identification information to store the meta-information in the storage unit.

17. The interactive question answering apparatus of claim 16, wherein the computer processor generates the identification information including an attribute of multimodal input information, for comparing the identification information with the multimodal input information output from the multimodal interface.

18. The interactive question answering apparatus of claim 16, wherein the computer processor generates the meta-information including the identification information which includes a uniform resource identifier of each of the plurality of entities, screen coordinates at which each of the plurality of entities is displayed on a display screen, a time section where an entity included in video content is reproduced, a time section where an entity included in audio content is reproduced, and an attribute name representing an attribute of each of the plurality of entities, and stores the meta-information in the storage unit.

19. The interactive question answering apparatus of claim 16, wherein the computer processor analyzes a question sentence transmitted from the user terminal to recognize a question centric word in the question sentence, determines attribute information about the question centric word, based on the identification information transmitted from the user terminal, and selects the attribute information as a right answer candidate to generate the answer sentence including the selected right answer candidate.

20. The interactive question answering apparatus of claim 19, wherein the computer processor searches a database storing the attribute information to search for a uniform resource identifier which is the same as a uniform resource identifier included in the identification information transmitted from the user terminal, and when the same uniform resource identifier is checked in the database, determines attribute information, allocated to the checked uniform resource identifier, as an attribute of the question centric word.