System and method for cognitive intervention on human interactions

• US 10,235,990 B2
• Filed: 01/04/2017
• Issued: 03/19/2019
• Est. Priority Date: 01/04/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Corelli, Helena Morgado, Fernandes, Rodrigo Braga, Guimaraes, Rodrigo Laiola, Matos, Rafael Sorana de, Nogima, Julio, Silva, Rogerio Cesar Barbosa dos Santos, Vital, Thiago Luiz de Barcelos
• Primary Examiner(s): Rowland, Steve
• Application Number:
  US15/398,029
• Publication Number:
  US 20180190144A1
• Time in Patent Office: 804 Days
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/222 Barge in, i.e. overridable guidance for interrupting prompts
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

A determination regarding whether to intervene in a dialog to provide system-initiated assistive information involves monitoring a dialog between at least two participants and capturing data from a dialog environment containing at least one of the participants. The captured data represent the content of the dialog and physiological data for one or more participants. Assistive information relevant to the dialog and participants is identified, and the captured data are used to determine an intervention index of delivering the assistive information to one or more participants during the dialog. This intervention index is then used to determine whether or not to intervene in the dialog to deliver the assistive information to one or more participants.

Claims

1. A dialog intervention system comprising:

at least one dialog environment containing at least two participants engaged in a dialog; and

a cognitive system in communication with the at least one dialog environment to monitor the dialog between the at least two participants, the cognitive system comprising;

interface capture modules to capture data from the dialog environment, the captured data comprising at least one of content of the dialog and physiological data for one or more participants;

processing modules to identify assistive information relevant to the dialog and participants and to use the captured data to determine an intervention index for delivering the assistive information to one or more participants during the dialog, the processing modules comprising an assistive information value and intervention cost module to determine a value of the assistive information to the dialog and participants in the dialog and to determine an intervention cost on the dialog caused by intervening in the dialog to communicate the assistive information; and

interface presentation modules to intervene in the dialog to deliver the assistive information to one or more participants based on the intervention index.

2. The system of claim 1, further comprising data capture devices disposed in the dialog environment to monitor the dialog and to capture data.

3. The system of claim 2, wherein the data capture devices comprise still cameras, video cameras, microphones, telephone equipment, telephone conferencing equipment, video conference equipment, telephones, cellular telephones, smartphones, tablet computers, electronic mail applications, text message applications, thermostats, thermometers, motion sensors, physiological sensors, activity sensors, biomedical sensors or combinations thereof.

4. The system of claim 1, wherein the processing modules comprise an assistive information detection module to use the captured data to identify assistive information.

5. The system of claim 4, wherein the assistive information detection module uses the captured data in at least one of static searches and dynamic searches over fixed databases and continuous data streams.

6. The system of claim 1, wherein the processing modules use the captured data to identify a physical or emotional condition of at least one participant.

7. The system of claim 1, wherein the assistive information value and intervention cost module compares the value of the assistive information to the intervention cost on the dialog and the interface modules intervene in the dialog if the value of the assistive information is sufficient to justify the intervention cost on the dialog.

8. The system of claim 1, wherein:

the processing modules comprise an intervention delivery module determine intervention parameters for communicating the assistive information to the one or more participants; and

the interface modules intervene in the dialog in accordance with the intervention parameters.

9. The system of claim 8, wherein:

the intervention delivery module identifies target participants in the dialog to receive the assistive information;

the processing modules comprise an intervention opportunity analysis module to identify an appropriate time during the dialog for intervening in the dialog that minimizes interference with the participants; and

the dialog environment comprises intervention mechanisms associated with the target participants and in communication with the interface modules that can be used to deliver the assistive information to the target participants.

10. The system of claim 9, wherein the intervention mechanisms comprise speakers, video monitors, lights, haptic systems, kinesthetic systems, graphical user interfaces, smart phones, tablet computers, telephones, cellular telephones or combinations thereof.

11. The system of claim 1, wherein the processing modules comprise an intervention result analysis module to determine post-intervention impacts on the dialog resulting from intervening in the dialog and delivering the assistive information and to use the post-intervention impacts to determine an intervention index for delivering additional assistive information to one or more participants at a later time during the dialog.

12. The system of claim 11, wherein the intervention result analysis module analyzes data captured from the dialog and participants after intervening in the dialog to determine the post-intervention impacts of intervening in the dialog to deliver the assistive information.

13. The system of claim 11, wherein the intervention result analysis module associates the post-intervention impacts with at least one of the dialog, a portion of the dialog, a type of dialog, the assistive information, a category of assistive information, a delivery mechanism used to communicate the assistive information and one or more participants and stores the post-intervention impacts and associations for use in determining the intervention index for delivering additional assistive information to one or more participants at the later time during the dialog.

14. A dialog intervention system comprising:

at least one dialog environment containing at least two participants engaged in a dialog; and

a cognitive system in communication with the at least one dialog environment to monitor the dialog between the at least two participants, the cognitive system comprising;

interface capture modules to capture data from a dialog environment containing at least one of the participants, the captured data comprising at least one of content of the dialog and physiological data for one or more participants;

processing modules in communication with the interface capture modules, the processing modules comprising;

an assistive information detection module to identifying assistive information relevant to the dialog and participants; and

an assistive information value and intervention cost analysis module to use the captured data to determine a value of the assistive information to the dialog and participants in the dialog and to determine an intervention cost on the dialog caused by intervening in the dialog to communicate the assistive information in order to determine an intervention index for delivering the assistive information to one or more participants during the dialog.

15. The system of claim 14, wherein:

the assistive information value and intervention cost analysis module compares the value of the assistive information to the intervention cost on the dialog; and

the cognitive system comprises interface presentation modules to intervene in the dialog if the value of the assistive information is sufficient to justify the intervention cost on the dialog.

16. The system of claim 14, wherein the assistive information value and intervention cost analysis module:

assigns a low value to assistive information comprising a system-generated suggestion;

assigns a high value to assistive information comprising a resolution to a conflict occurring among the participants during the dialog; and

assigns a medium value between the low value and the high value to assistive information comprising an answer to a participant generated question.

17. The system of claim 14, wherein the assistive information value and intervention cost analysis module:

assigns a low intervention cost level when intervening in the dialog elicits a positive reaction from one or more participants;

assigns a high intervention cost level when intervening in the dialog elicits a negative reaction from one or more participants; and

assigns a medium intervention cost level between the low intervention cost level and the high intervention cost level when intervening in the dialog elicits a neutral reaction from one or more participants.

18. The system of claim 15, wherein the processing modules comprise an intervention result analysis module to determine post-intervention impacts on the dialog resulting from intervening in the dialog and delivering the assistive information and to use the post-intervention impacts to determine an intervention index for delivering additional assistive information to one or more participants at a later time during the dialog.

19. The system of claim 18, wherein the intervention result analysis module associates the post-intervention impacts with at least one of the dialog, a portion of the dialog, a type of dialog, the assistive information, a category of assistive information, a delivery mechanism used to communicate the assistive information and one or more participants and stores the post-intervention impacts and associations for use in determining the intervention index for delivering additional assistive information to one or more participants at the later time during the dialog.

20. A dialog intervention system comprising:

at least one dialog environment containing at least two participants engaged in a dialog; and

a cognitive system in communication with the at least one dialog environment to monitor the dialog between the at least two participants, the cognitive system comprising;

interface capture modules to capture data from the dialog environment, the captured data comprising at least one of content of the dialog and physiological data for one or more participants;

processing modules to identify assistive information relevant to the dialog and participants and to use the captured data to determine an intervention index for delivering the assistive information to one or more participants during the dialog, the processing modules comprising;

an intervention delivery module to determine intervention parameters for communicating the assistive information to the one or more participants and to identify target participants in the dialog to receive the assistive information; and

an intervention opportunity analysis module to identify an appropriate time during the dialog for intervening in the dialog that minimizes interference with the participants; and

interface presentation modules to intervene in the dialog to deliver the assistive information to one or more participants based on the intervention index, the interface presentation modules intervening in the dialog in accordance with the intervention parameters;

wherein the dialog environment comprises intervention mechanisms associated with the target participants and in communication with the interface modules that can be used to deliver the assistive information to the target participants.