Systems, methods, and computer-readable storage device for generating notes for a meeting based on participant actions and machine learning

• US 10,510,346 B2
• Filed: 11/09/2017
• Issued: 12/17/2019
• Est. Priority Date: 11/09/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Rahmel, Heiko, Qin, Li-Juan, Huang, Xuedong, Xiong, Wei
• Primary Examiner(s): Colucci, Michael C
• Application Number:
  US15/807,704
• Publication Number:
  US 20190139543A1
• Time in Patent Office: 768 Days
• Field of Search: 704500, 704233, 704235, 704260, 704 9, 709204, 715730, 715744, 715846
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals
• CPC Class Codes: H04N 7/08 Systems for the simultaneous or sequential transmission of more than one television signal, e.g. additional information signals, the signals occupying wholly or partially the same frequency band , e.g. by time division H04N7/007 takes precedence
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

Systems, methods, and computer-readable storage devices are disclosed for generating smart notes for a meeting based on participant actions and machine learning. One method including: receiving meeting data from a plurality of participant devices participating in an online meeting; continuously generating text data based on the received audio data from each participant device of the plurality of participant devices; iteratively performing the following steps until receiving meeting data for the meeting has ended, the steps including: receiving an indication that a predefined action has occurred on the first participating device; generating a participant segment of the meeting data for at least the first participant device from a first predetermined time before when the predefined action occurred to when the predefined action occurred; determining whether the receiving meeting data of the meeting has ended; and generating a summary of the meeting.

Claims

1. A computer-implemented method for generating notes for a meeting based on participant actions and machine learning, the method comprising:

receiving, over an electronic communications network, meeting data from a plurality of participant devices participating in an online meeting, the meeting data including audio data generated during the online meeting from the plurality of participant devices, and the meeting data including an audio time stamp of when audio of the audio data occurred;

continuously generating, by at least one server when receiving meeting data, text data based on the received audio data from each participant device of the plurality of participant devices, the text data being a transcription of audio captured during the online meeting and text data including the audio time stamp of when the audio corresponding to the text data was captured;

receiving, from a first participant device of the plurality of participant devices during receiving the meeting data, an indication that a first predefined action has occurred on the first participating device, the indication including a first action time stamp in which the first predefined action occurred and a first identifier that identifies the first participant device;

generating, by the at least one server, a first participant segment of the meeting data for the first participant device, the first participant segment including the first identifier and generated text data from a first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred;

receiving, from a second participant device of the plurality of participant devices during receiving the meeting data, an indication that a second predefined action has occurred on the second participating device, the indication including a second action time stamp in which the second predefined action occurred and a second identifier that identifies the second participant device;

generating, by the at least one server, a second participant segment of the meeting data for the second participant device, the second participant segment including the second identifier and generated text data from the first predetermined time before the second action time stamp in which the second predefined action occurred to the second action time stamp in which the second predefined action occurred;

determining whether text data of the first participant segment overlaps text data of the second participant segment;

extracting, when the text data of the first participant segment overlaps text data of the second participant segment, at least one of a keyword and a phrase from the overlapping text data; and

generating a summary of the meeting for the first participant device, the summary including the first participant segment and the extracted at least one of keyword and phrase from the overlapping text data.

2. The method according to claim 1, wherein a predefined action of one or more of the first and second predefined actions includes one or more of a participant clicking a physical button associated with a participant device of the participant, a participant clicking a button displayed on a display screen of the participant device of the participant, a participant performing a predetermined gesture, a participant speaking a predetermined word and/or phrase, and a participant requesting a virtual assistant generated a note for the meeting.

3. The method according to claim 1, further comprising:

extracting at least one of a keyword and a phrase from the generated text data from the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred, and

wherein the first participant segment further includes the extracted at least one of keyword and phrase from the generated text data from the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred.

4. The method according to claim 3, further comprising:

determining whether the extracted at least one of keyword and phrase is present in the continuously generated text data other than the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred; and

generating, for each occurrence of the extracted at least one of keyword and phrase in the continuously generated text data other than the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred, a machine learned segment of the meeting data, the machine learned segment including generated text data from a first predetermined time before the occurrence of the extracted at least one of keyword and phrase to a time of occurrence of the extracted at least one of keyword and phrase.

5. The method according to claim 3, further comprising:

determining, for each extracted at least one of keyword and phrase of a participant segment when receiving meeting data for the meeting has ended, whether the extracted at least one of keyword and phrase is present in the generated text data other than the text data of the first participant segment; and

generating, for each occurrence of the extracted at least one of keyword and phrase in the generated text data other than the text data of the first participant segment, a machine learned segment of the meeting data, the machine learned segment including generated text data from a first predetermined time before the occurrence of the extracted at least one of keyword and phrase to a time of occurrence of the extracted at least one of keyword and phrase.

6. The method according to claim 1, further comprising:

incrementing a point of interest counter for overlapping text data, the point of interest counter indicating that a plurality of participant segments have been generated for the overlapping text data.

7. The method according to claim 1, further comprising:

determining, for each extracted at least one of keyword and phrase from the overlapping text data when receiving meeting data for the meeting has ended, whether the extracted at least one of keyword and phrase is present in the generated text data other than the overlapping text data; and

generating, for each occurrence of the extracted at least one of keyword and phrase in the generated text data other than the overlapping text, a machine learned segment of the meeting data, the machine learned segment including generated text data from a first predetermined time before the occurrence of the extracted at least one of keyword and phrase to a time of occurrence of the extracted at least one of keyword and phrase.

8. The method according to claim 1, wherein generating the first participant segment includes:

generating the first participant segment of the meeting data for the first participant device, the generated first participant segment including the first identifier and the generated text data from the first predetermined time before the first action time stamp in which the first predefined action occurred to a second predetermined amount of time after the first predefined action occurred, wherein the first predetermined amount of time and second predetermined amount of time are one of a same amount of time and a different amount of time.

9. The method according to claim 1, further comprising:

determining, when receiving meeting data for the meeting has ended, whether the generated text data includes at least one of a predetermined time-based keyword and a predetermined time-based phrase;

generating, when the generated text data includes the at least one of the predetermined time-based keyword and the predetermined time-based phrase, a time-based segment for a predetermined time around the at least one of the predetermined time-based keyword and the predetermined time-based phrase, the time-based segment including the generated text data from the predetermined time around the at least one of the predetermined time-based keyword and the predetermined time-based phrase; and

generating, for each time-based segment, a follow-up action, the follow-up action including one or more of a calendar invite and a reminder based on the generated text data of the time-based segment.

10. The method according to claim 1, further comprising:

determining, when receiving meeting data for the meeting has ended, whether the generated text data includes at least one of a predetermined emphasis keyword and a predetermined emphasis phrase; and

generating, when the generated text data includes the at least one of the predetermined emphasis keyword and the predetermined emphasis phrase, an emphasis segment for a predetermined time around the at least one of the predetermined emphasis keyword and the predetermined emphasis phrase, the emphasis segment including the generated text data from the predetermined time around the at least one of the predetermined emphasis keyword and the predetermined emphasis phrase,

wherein the summary further includes a plurality of generated emphasis segments.

11. The method according to claim 1, further comprising:

receiving, from at least one participant device, an edit to the text data of the transcription of audio captured during the online meeting;

generating new text data based on the received audio data from each participant device of the plurality of participant devices and the edit to the text data of the transcription of audio captured during the online meeting.

12. A system for generating smart notes for a meeting based on participant actions and machine learning, the system including:

a data storage device that stores instructions for generating smart notes for a meeting based on participant actions and machine learning; and

a processor configured to execute the instructions to perform a method including;

receiving, over an electronic communications network, meeting data from a plurality of participant devices participating in an online meeting, the meeting data including audio data generated during the online meeting from the plurality of participant devices, and the meeting data including an audio time stamp of when audio of the audio data occurred;

continuously generating, when receiving meeting data, text data based on the received audio data from each participant device of the plurality of participant devices, the text data being a transcription of audio captured during the online meeting and text data including the audio time stamp of when the audio corresponding to the text data was captured;

receiving, from a first participant device of the plurality of participant devices during receiving the meeting data, an indication that a first predefined action has occurred on the first participating device, the indication including a first action time stamp in which the first predefined action occurred and a first identifier that identifies the first participant device;

generating a first participant segment of the meeting data for the first participant device, the first participant segment including the first identifier and generated text data from a first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred;

receiving, from a second participant device of the plurality of participant devices during receiving the meeting data, an indication that a second predefined action has occurred on the second participating device, the indication including a second action time stamp in which the second predefined action occurred and a second identifier that identifies the second participant device;

generating a second participant segment of the meeting data for the second participant device, the second participant segment including the second identifier and generated text data from the first predetermined time before the second action time stamp in which the second predefined action occurred to the second action time stamp in which the second predefined action occurred;

determining whether text data of the first participant segment overlaps text data of the second participant segment;

extracting, when the text data of the first participant segment overlaps text data of the second participant segment, at least one of a keyword and a phrase from the overlapping text data; and

generating a summary of the meeting for the first participant device, the summary including the first participant segment and the extracted at least one of keyword and phrase from the overlapping text data.

13. The system according to claim 12, wherein the processor is further configured to execute the instructions to perform the method including:

extracting at least one of a keyword and a phrase from the generated text data from the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred, and

wherein the first participant segment further includes the extracted at least one of keyword and phrase from the generated text data from the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred.

14. The system according to claim 13, wherein the processor is further configured to execute the instructions to perform the method including:

determining whether the extracted at least one of keyword and phrase is present in the continuously generated text data other than the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred; and

generating, for each occurrence of the extracted at least one of keyword and phrase in the continuously generated text data other than the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred, a machine learned segment of the meeting data, the machine learned segment including generated text data from a first predetermined time before the occurrence of the extracted at least one of keyword and phrase to a time of occurrence of the extracted at least one of keyword and phrase.

15. The system according to claim 13, wherein the processor is further configured to execute the instructions to perform the method including:

determining, for each extracted at least one of keyword and phrase of a participant segment when receiving meeting data for the meeting has ended, whether the extracted at least one of keyword and phrase is present in the generated text data other than the text data of the first participant segment; and

generating, for each occurrence of the extracted at least one of keyword and phrase in the generated text data other than the text data of the first participant segment, a machine learned segment of the meeting data, the machine learned segment including generated text data from a first predetermined time before the occurrence of the extracted at least one of keyword and phrase to a time of occurrence of the extracted at least one of keyword and phrase.

16. The system according to claim 12, wherein the processor is further configured to execute the instructions to perform the method including:

incrementing a point of interest counter for overlapping text data, the point of interest counter indicating that a plurality of participant segments have been generated for the overlapping text data.

17. The system according to claim 12, wherein the processor is further configured to execute the instructions to perform the method including:

determining, for each extracted at least one of keyword and phrase from the overlapping text data when receiving meeting data for the meeting has ended, whether the extracted at least one of keyword and phrase is present in the generated text data other than the overlapping text data; and

generating, for each occurrence of the extracted at least one of keyword and phrase in the generated text data other than the overlapping text, a machine learned segment of the meeting data, the machine learned segment including generated text data from a first predetermined time before the occurrence of the extracted at least one of keyword and phrase to a time of occurrence of the extracted at least one of keyword and phrase.

18. A computer-readable storage device storing instructions that, when executed by a computer, cause the computer to perform a method for generating smart notes for a meeting based on participant actions and machine learning, the method including:

receiving, over an electronic communications network, meeting data from a plurality of participant devices participating in an online meeting, the meeting data including audio data generated during the online meeting from the plurality of participant devices, and the meeting data including an audio time stamp of when audio of the audio data occurred;

continuously generating, when receiving meeting data, text data based on the received audio data from each participant device of the plurality of participant devices, the text data being a transcription of audio captured during the online meeting and text data including the audio time stamp of when the audio corresponding to the text data was captured;

receiving, from a first participant device of the plurality of participant devices during receiving the meeting data, an indication that a first predefined action has occurred on the first participating device, the indication including a first action time stamp in which the first predefined action occurred and a first identifier that identifies the first participant device;

generating a first participant segment of the meeting data for the first participant device, the first participant segment including the first identifier and generated text data from a first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred;

receiving, from a second participant device of the plurality of participant devices during receiving the meeting data, an indication that a second predefined action has occurred on the second participating device, the indication including a second action time stamp in which the second predefined action occurred and a second identifier that identifies the second participant device;

generating a second participant segment of the meeting data for the second participant device, the second participant segment including the second identifier and generated text data from the first predetermined time before the second action time stamp in which the second predefined action occurred to the second action time stamp in which the second predefined action occurred;

determining whether text data of the first participant segment overlaps text data of the second participant segment;

extracting, when the text data of the first participant segment overlaps text data of the second participant segment, at least one of a keyword and a phrase from the overlapping text data; and

generating a summary of the meeting for the first participant device, the summary including the first participant segment and the extracted at least one of keyword and phrase from the overlapping text data.

19. The computer-readable storage device according to claim 18, wherein the steps further include:

extracting at least one of a keyword and a phrase from the generated text data from the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred, and

wherein the first participant segment further includes the extracted at least one of keyword and phrase from the generated text data from the first predetermined time before the first action time stamp in which the first predefined action occurred to the first action time stamp in which the first predefined action occurred.

20. The computer-readable storage device according to claim 18, wherein the steps further include:

determining, for each extracted at least one of keyword and phrase from the overlapping text data when receiving meeting data for the meeting has ended, whether the extracted at least one of keyword and phrase is present in the generated text data other than the overlapping text data; and

generating, for each occurrence of the extracted at least one of keyword and phrase in the generated text data other than the overlapping text, a machine learned segment of the meeting data, the machine learned segment including generated text data from a first predetermined time before the occurrence of the extracted at least one of keyword and phrase to a time of occurrence of the extracted at least one of keyword and phrase.