Enhanced video conference management

• US 10,382,722 B1
• Filed: 09/11/2018
• Issued: 08/13/2019
• Est. Priority Date: 09/11/2017
• Status: Active Grant

• Current Assignee: Reelay Meetings, Inc.
• Original Assignee: Michael H. Peters
• Inventors: Peters, Michael H., Stufflebeam, Alexander M.
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US16/128,137
• Time in Patent Office: 336 Days
• Field of Search: 348 1401- 1416
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for enhanced video conference management. In some implementations, a media stream is received from each of multiple endpoint devices over a communication network. A video conference session among the endpoint devices is managed such that at least one or more of the media streams are transmitted over the communication network for display by the endpoint devices. A plurality of audio and/or video characteristics from the media stream from a particular endpoint device of the multiple endpoint devices are measured. Based on the audio and/or video characteristics, a collaboration factor score is determined for the particular endpoint device for each of a plurality of collaboration factors. The video conference of the endpoint devices by performing a video conference management action selected based on the collaboration factor scores.

Claims

1. A method of managing a video conference session involving multiple endpoint devices, the method comprising:

receiving, by one or more computers, a media stream from each of multiple endpoint devices over a communication network;

managing, by the one or more computers, a video conference session among the endpoint devices such that at least one or more of the media streams are transmitted over the communication network for display by the endpoint devices;

measuring, by the one or more computers, a plurality of audio and/or video characteristics from the media stream from a particular endpoint device of the multiple endpoint devices;

determining, by the one or more computers, based on the audio and/or video characteristics, a collaboration factor score for the particular endpoint device for each of a plurality of collaboration factors, wherein determining the scores for the plurality of collaboration factors comprises determining a score for each of a plurality of different neurological and/or emotional dimensions;

selecting, by the one or more computers, from among a plurality of video conference management actions based on the collaboration factor scores for the media stream for the particular endpoint; and

altering, by the one or more computers, the video conference session for the multiple endpoints by performing the selected video conference management action.

2. The method of claim 1, wherein the selected video conference management action comprises changing a size or resolution for the media stream for the particular endpoint;

and

wherein altering the management of the video conference session comprises changing a size or resolution at which the media stream for the particular endpoint is transmitted, or changing a size or resolution at which the media stream for the particular endpoint is presented by one or more of the multiple endpoints.

3. The method of claim 1, wherein the selected video conference management action comprises reducing bandwidth of transmission of the media stream for the particular endpoint;

and

wherein altering the management of the video conference session comprises reducing bandwidth of transmission of the media stream for the particular endpoint by increasing a compression level, changing a compression codec, reducing a frame rate, or stopping transmission of the media stream for the particular endpoint to one or more other endpoints.

4. The method of claim 1, wherein the selected video conference management action comprises altering an audio volume level for the media stream for the particular endpoint;

and

wherein altering the management of the video conference session comprises altering an audio volume level for the media stream for the particular endpoint by instructing a different volume level for the audio, muting the audio, or omitting the audio from the media stream for the particular endpoint transmitted to one or more other endpoints.

5. The method of claim 1, comprising classifying, by the one or more computers, the media stream for the particular endpoint device based on the collaboration factor scores;

wherein the video conference management action is selected based on the classification of the media stream for the particular endpoint device.

6. The method of claim 5, wherein classifying the media stream for the particular endpoint device comprises:

providing, to a trained machine learning classifier, the collaboration factor scores as inputs and receiving, as output of the trained machine learning classifier, one or more outputs indicative of likelihoods for one or more of a plurality of predetermined classifications, wherein the trained machine learning classifier comprises at least one of a neural network, a decision tree, a support vector machine, a logistic regression model, or a maximum entropy classifier.

7. The method of claim 5, wherein classifying the media stream comprises assigning a classification from among a plurality of predetermined classifications each having an associated combination of collaboration factor scores or ranges, wherein the collaboration factors are indicative of different neurological and/or emotional attributes.

8. The method of claim 5, wherein classifying the media stream comprises:

determining a distance between (i) a set of collaboration factor scores for the particular endpoint, and (ii) each of multiple different combinations of collaboration factor scores or ranges that are each associated with a different classification; and

selecting, as a classification for the media stream of the particular endpoint, the classification associated with the combination of collaboration factor scores or ranges having the lowest distance from the set of collaboration factor scores for the particular endpoint.

9. The method of claim 1, wherein measuring the plurality of audio and/or video characteristics from the media stream from the particular endpoint device comprises identifying characteristics using speech recognition, facial expression recognition, gesture recognition, intonation analysis, eye gaze position analysis, head position analysis, and/or pattern analysis.

10. The method of claim 1, comprising:

combining the scores for the collaboration factors into a composite score; and

outputting, during the video conference session to one or more of the endpoint devices, data providing a representation of a participant corresponding to the particular endpoint, a representation of the composite score, and an output media stream from at least one of the endpoint devices.

11. The method of claim 10, wherein the representation of the composite score comprises a symbol that changes in color according to a degree of quality and level of participation of a participant whose voice or image is indicated by the media stream for the particular endpoint, wherein the degree of quality and level of participation is indicated relative to degrees of quality and level of participation determined for participants corresponding to other endpoints involved in the video conference.

12. The method of claim 11, comprising accessing data indicating multiple predetermined thresholds for the degree of quality and level of participation;

setting the color of the symbol based on the degree of quality and level of participation relative to the multiple predetermined thresholds, the one or more computers being configured to indicate a shade of red if the degree of quality and level of participation is determined to be in excess of a first predetermined threshold, a shade of orange if the measurement is determined to be between the first predetermined threshold and a second predetermined threshold, and a shade of green if the measurement is determined be below the second predetermined threshold.

13. The method of claim 1, comprising:

determining a group composite score based on collaboration factor scores determined from each of the media streams of the endpoint devices, the group composite score being indicative of a current level of collaboration and participation among participants in the video conference; and

providing the group composite score to one or more of the endpoints over the communication network or using the group composite score to select the video conference management action.

14. The method of claim 1, wherein measuring the plurality of audio and/or video characteristics from the media stream from the particular endpoint device comprises using facial expression recognition to determine estimate levels of at least one of attention, enthusiasm, happiness, sadness, stress, boredom, dominance, fear, anger, or deception.

15. The method of claim 1, measuring the plurality of audio and/or video characteristics from the media stream from the particular endpoint device comprises at least one of:

determining a duration of speaking or participation time;

detecting utterance of one or more keywords from a set of predetermined keywords;

or

identifying an intonation pattern.

16. The method of claim 1, comprising:

determining a composite score for each of the multiple endpoints based on the respective media streams for the endpoints, the composite scores indicating respective levels of participation or collaboration in the video conference;

determining and outputting, in real-time during the video conference session, a representation of instructions to improve one or more of the composite scores indicating the lowest levels of participation or collaboration in the video conference; and

periodically recalculating the composite scores for the multiple endpoints and altering the instructions based on the recalculated composite scores.

17. The method of claim 1, wherein the one or more computers are configured to switch between enhancing the video conference in multiple different modes, comprising:

a passive public mode in which the one or more computers provide an output media stream display indicator for each of the multiple endpoints to each endpoint in the video conference;

a passive private mode in which the one or more computers (i) provide feedback based on composite scores for individual endpoints and/or for the overall collection of endpoints to a limited set of one or more devices having a predetermined permission level, and (ii) provide suggestions of moderator actions only to the limited set of one or more devices having a predetermined permission level; and

an active mode in which the one or more computers introduce direct integrated audio and visual indicators and messages through the output media stream of one or more conference participants.

18. The method of claim 1, wherein the scores for the neurological and/or emotional dimensions are based on facial characteristics detected from image data from the media stream from the particular endpoint device.

19. The method of claim 1, wherein the scores for the neurological and/or emotional dimensions are based on voice intonation characteristics detected from audio data from the media stream from the particular endpoint device.

20. The method of claim 1, wherein measuring the plurality of audio and/or video characteristics from the media stream from a particular endpoint device comprises analyzing facial expressions in image data from the media stream from the particular endpoint device;

and

wherein determining the scores for the plurality of different neurological and/or emotional dimensions comprises determining emotional attribute scores for different emotional attributes, the emotional attribute scores respectively indicating levels of the emotional attributes that a user of the particular endpoint device is determined to exhibit based on the analysis of the facial expressions in the image data from the media stream from the particular endpoint device.

21. A system comprising:

one or more computers; and

one or more computer-readable storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving, by the one or more computers, a media stream from each of multiple endpoint devices over a communication network;

managing, by the one or more computers, a video conference session among the endpoint devices such that at least one or more of the media streams are transmitted over the communication network for display by the endpoint devices;

measuring, by the one or more computers, a plurality of audio and/or video characteristics from the media stream from a particular endpoint device of the multiple endpoint devices;

determining, by the one or more computers, based on the audio and/or video characteristics, a collaboration factor score for the particular endpoint device for each of a plurality of collaboration factors, wherein determining the scores for the plurality of collaboration factors comprises determining a score for each of a plurality of different neurological and/or emotional dimensions;

selecting, by the one or more computers, from among a plurality of video conference management actions based on the collaboration factor scores for the media stream for the particular endpoint; and

altering, by the one or more computers, the video conference session for the multiple endpoints by performing the selected video conference management action.

22. One or more non-transitory computer-readable storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving, by the one or more computers, a media stream from each of multiple endpoint devices over a communication network;

managing, by the one or more computers, a video conference session among the endpoint devices such that at least one or more of the media streams are transmitted over the communication network for display by the endpoint devices;

measuring, by the one or more computers, a plurality of audio and/or video characteristics from the media stream from a particular endpoint device of the multiple endpoint devices;

determining, by the one or more computers, based on the audio and/or video characteristics, a collaboration factor score for the particular endpoint device for each of a plurality of collaboration factors, wherein determining the scores for the plurality of collaboration factors comprises determining a score for each of a plurality of different neurological and/or emotional dimensions;

selecting, by the one or more computers, from among a plurality of video conference management actions based on the collaboration factor scores for the media stream for the particular endpoint; and

altering, by the one or more computers, the video conference session for the endpoint devices by performing the selected video conference management action.