Orator effectiveness through real-time feedback system with automatic detection of human behavioral and emotional states of orator and audience

• US 10,431,116 B2
• Filed: 12/10/2015
• Issued: 10/01/2019
• Est. Priority Date: 12/10/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Briggs, Benjamin D., Clevenger, Lawrence A., Clevenger, Leigh Anne H., Connell, II, Jonathan H., Ratha, Nalini K., Rizzolo, Michael
• Primary Examiner(s): Saint-Vil, Eddy
• Application Number:
  US14/965,667
• Publication Number:
  US 20170169727A1
• Time in Patent Office: 1,391 Days
• CPC Class Codes: G09B 19/04 Speaking with audible presentation of the material to be studied G09B5/04
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied
• CPC Class Codes: G09B 5/125 the stations being mobile
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

Techniques for leveraging the capabilities of wearable mobile technology to collect data and to provide real-time feedback to an orator about his/her performance and/or audience interaction are provided. In one aspect, a method for providing real-time feedback to a speaker making a presentation to an audience includes the steps of: collecting real-time data from the speaker during the presentation, wherein the data is collected via a mobile device worn by the speaker; analyzing the real-time data collected from the speaker to determine whether corrective action is needed to improve performance; and generating a real-time alert to the speaker suggesting the corrective action if the real-time data indicates that corrective action is needed to improve performance, otherwise continuing to collect data from the speaker in real-time. Real-time data may also be collected from members of the audience and/or from other speakers (if present) via wearable mobile devices.

Claims

1. A method for providing real-time feedback to a speaker making a presentation to an audience, the method comprising the steps of:

collecting real-time data from the speaker and from members of the audience during the presentation wherein the real-time data collected provides a combination of feedback from the speaker and from the members of the audience, wherein the real-time data is collected via mobile devices worn by the speaker and by the members of the audience, wherein the mobile devices comprise smartwatches worn on a user'"'"'s wrist comprising a trajectory and pose sensor configured to detect the user'"'"'s arm and hand movements and a speech detector sensor configured to recognize certain words and speech patterns, wherein the real-time data collected from the speaker further comprises at least i) trajectory and pose data for the speaker including the speaker'"'"'s arm and hand movements as well as ii) word recognition and speech pattern data including recognition of certain words spoken by the speaker, the speed at which the speaker is speaking, and pauses the speaker makes, which are all collected and analyzed by the smartwatch;

analyzing the real-time data collected from the speaker to make a determination of whether corrective action is needed to improve performance; and

generating a real-time alert to the speaker on the smartwatch worn on the speaker'"'"'s wrist suggesting the corrective action if the real-time data indicates that corrective action is needed to improve performance, otherwise continuing to collect data from the speaker in real-time, wherein the determination of whether corrective action is needed is based on the combination of feedback from the speaker and from the members of the audience wherein the real-time alert to the speaker is generated when the real-time data collected from the speaker indicates that no corrective action is needed but the real time data collected from the members of the audience indicates that the corrective action is needed to improve performance, and

wherein the speaker is one of multiple co-presenters presenting on different aspects of a particular topic to the audience, the method further comprising the steps of;

collecting the real-time data from each of the co-presenters via smartwatches worn by each of the co-presenters; and

providing the real-time feedback to the speaker on the smartwatch worn on the speaker'"'"'s wrist regarding the speaker'"'"'s performance and a performance of each of the co-presenters.

2. The method of claim 1, wherein the real-time data collected from the speaker comprises physiological data for the speaker selected from the group consisting of:

skin electrical characteristics, blood oxygen levels, heart rate, pulse, and combinations thereof.

3. The method of claim 1, wherein the trajectory and pose data collected from the speaker further comprises the real-time data selected from the group consisting of:

movement, speed, direction, orientation, location, and combinations thereof.

4. The method of claim 1, wherein the real-time data collected from the speaker comprises environmental data.

5. The method of claim 1, wherein the mobile devices comprise at least one sensor selected from the group consisting of:

an electrodermal activity (EDA) sensor, a pulse oximeter sensor, a heart rate sensor, and combinations thereof.

6. The method of claim 1, wherein the mobile devices comprise at least one sensor selected from the group consisting of:

a gyroscope sensor, a global positioning system (GPS) sensor, and combinations thereof.

7. The method of claim 1, wherein the mobile devices comprise an air temperature sensor.

8. The method of claim 1, wherein at least one of the members of the audience is at a remote location relative to the speaker.

9. The method of claim 1, further comprising the step of:

generating, in real-time, an alert to each of the co-presenters suggesting the corrective action.

10. The method of claim 1, wherein the real-time alert comprises a text message suggesting the corrective action needed to improve performance.

11. The method of claim 10, wherein the text message is accompanied by at least one of an audible or visual alert.

12. The method of claim 1, further comprising the step of:

determining a proper corrective action to improve performance based on the real-time data collected from the speaker.

13. The method of claim 12, wherein the proper corrective action to improve performance is determined based on a series of predetermined scenarios and respective corrective actions.

14. The method of claim 13, wherein the series of predetermined scenarios and respective corrective actions are accessed via a look-up table format.

15. A computer program for providing real-time feedback to a speaker making a presentation to an audience, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

collect real-time data from the speaker and from members of the audience during the presentation wherein the real-time data collected provides a combination of feedback from the speaker and from the members of the audience, wherein the real-time data is collected via mobile devices worn by the speaker and by the members of the audience, wherein the mobile devices comprise smartwatches worn on a user'"'"'s wrist comprising a trajectory and pose sensor configured to detect the user'"'"'s arm and hand movements and a speech detector sensor configured to recognize certain words and speech patterns, wherein the real-time data collected from the speaker further comprises at least i) trajectory and pose data for the speaker including the speaker'"'"'s arm and hand movements as well as ii) word recognition and speech pattern data including recognition of certain words spoken by the speaker, the speed at which the speaker is speaking, and pauses the speaker makes, which are all collected and analyzed by the smartwatch;

analyze the real-time data collected from the speaker to make a determination of whether corrective action is needed to improve performance; and

generate a real-time alert to the speaker on the smartwatch worn on the speaker'"'"'s wrist suggesting the corrective action if the real-time data indicates that corrective action is needed to improve performance, otherwise continuing to collect data from the speaker in real-time, wherein the determination of whether corrective action is needed is based on the combination of feedback from the speaker and from the members of the audience wherein the real-time alert to the speaker is generated when the real-time data collected from the speaker indicates that no corrective action is needed but the real-time data collected from the members of the audience indicates that the corrective action is needed to improve performance, and

wherein the speaker is one of multiple co-presenters presenting on different aspects of a particular topic to the audience, and wherein the program instructions further cause the computer to;

collect the real-time data from each of the co-presenters via smartwatches worn by each of the co-presenters; and

provide the real-time feedback to the speaker on the smartwatch worn on the speaker'"'"'s wrist regarding the speaker'"'"'s performance and a performance of each of the co-presenters.