Speaker anticipation

• US 11,019,308 B2
• Filed: 11/08/2019
• Issued: 05/25/2021
• Est. Priority Date: 06/23/2017
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Bright-Thomas, Paul, Buckles, Nathan, Griffin, Keith, Chen, Eric, Kesavan, Manikandan, Nedeltchev, Plamen, Latapie, Hugo Mike, Fenoglio, Enzo
• Primary Examiner(s): Nguyen, Phung-Hoang J
• Application Number:
  US16/678,729
• Publication Number:
  US 20200077049A1
• Time in Patent Office: 564 Days
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

Systems and methods are disclosed for anticipating a video switch to accommodate a new speaker in a video conference comprising a real time video stream captured by a camera local to a first videoconference endpoint is analyzed according to at least one speaker anticipation model. The speaker anticipation model predicts that a new speaker is about to speak. Video of the anticipated new speaker is sent to the conferencing server in response to a request for the video on the anticipated new speaker from the conferencing server. Video of the anticipated new speaker is distributed to at least a second videoconference endpoint.

Claims

1. A video conference method comprising:

receiving sensory input from at least one sensor local to a participant of a videoconference;

analyzing the sensory input based on historical data to yield a prediction score that the participant is about to participate in the videoconference, the prediction score defined based on a range, the range including a point indicating the participant is equally likely to participate and not participate;

distributing video of the participant, via the videoconference, when the prediction score exceeds a threshold defined within the range, the threshold being a predetermined percentage or a predetermined value, the threshold displaced from the point of the range.

2. The method of claim 1, wherein the historical data is a historical video feed including the participant.

3. The method of claim 1,

wherein,

the sensory input is a real time video stream captured by a camera local to a first videoconference endpoint, and

the analyzing of the sensory input is performed by at least one speaker anticipation model to yield the prediction score.

4. The method of claim 3,

wherein,

the at least one speaker anticipation model is derived by:

training a first machine learning algorithm on a guided learning dataset, wherein the first machine learning algorithm analyzes static frames to identify visual speaker anticipation cues, and

providing an output of the first machine learning algorithm as an input to a second machine learning algorithm,

the second machine learning algorithm analyzes sequences of the static frames and sequences of the visual speaker anticipation cues, and

the output of the second machine learning algorithm is the at least one speaker anticipation model.

5. The method of claim 3, further comprising:

analyzing video data consisting of the real time video stream captured by the camera local to the first videoconference endpoint to generate labeled real time video frames, the labeled real time video frames labeled as real time video speaking frames or real time video pre-speaking frames; and

applying a machine learning algorithm to analyze the labeled real time video frames and update the at least one speaker anticipation model.

6. The method of claim 3,

wherein,

the videoconference includes a plurality of conference participants, and

the at least one speaker anticipation model is a semantic representation model created by a machine learning algorithm that has analyzed a plurality of signals including at least one of video data, audio data, and conference participant in-room location data, from historical videoconferences taking place in a specific meeting room.

7. The method of claim 6, further comprising:

ranking the plurality of signals captured during a real time video conference according to the semantic representation model; and

ignoring signals having a ranking below the threshold.

8. A videoconference system comprising:

a processor;

a storage device in communication with the processor; and

a videoconference endpoint configured, via the storage device and the processor, to;

receive sensory input from at least one sensor local to a participant of a videoconference;

analyze the sensory input based on historical data to yield a prediction score that the participant is about to participate in the videoconference, the prediction score defined based on a range, the range including a point indicating the participant is equally likely to participate and not participate; and

distribute video of the participant, via the videoconference, when the prediction score exceeds a threshold defined within the range, the threshold being a predetermined percentage or a predetermined value, the threshold displaced from the point of the range.

9. The videoconference system of claim 8, wherein the historical data is a historical video feed including the participant.

10. The videoconference system of claim 8,

wherein,

the sensory input is a real time video stream captured by a camera local to a first videoconference endpoint, and

the sensory input is analyzed using at least one speaker anticipation model to yield the prediction score.

11. The videoconference system of claim 10, wherein the at least one speaker anticipation model is derived by:

training a first machine learning algorithm on a guided learning dataset; and

providing an output of the first machine learning algorithm as an input to a second machine learning algorithm, the output of the second machine learning algorithm being the at least one speaker anticipation model.

12. The videoconference system of claim 11, wherein the first machine learning algorithm analyzes static frames to identify visual speaker anticipation cues.

13. The videoconference system of claim 12, wherein the second machine learning algorithm analyzes sequences of the static frames and sequences of visual speaker anticipation cues.

14. The videoconference system of claim 8, further comprising:

an attention mechanism configured to;

collect video, audio, and gesture recognition to perform a predictive analysis;

allow camera feed focus in real time; and

learn which actions are significant to focus camera attention on.

15. At least one non-transitory computer readable medium comprising instructions that, when executed, cause at least one computing device to perform operations comprising:

receiving sensory input from at least one sensor local to a participant of a videoconference;

analyzing the sensory input based on historical data to yield a prediction score that the participant is about to participate in the videoconference, the prediction score defined based on a range, the range including a point indicating the participant is equally likely to participate and not participate; and

distributing video of the participant, via the videoconference, when the prediction score exceeds a threshold defined within the range, the threshold being a predetermined percentage or a predetermined value, the threshold displaced from the point of the range.

16. The at least one non-transitory computer readable medium of claim 15, wherein the historical data is a historical video feed including the participant.

17. The at least one non-transitory computer readable medium of claim 16, wherein the operations include analyzing static frames of the historical video feed to identify visual speaker anticipation cues.

18. The at least one non-transitory computer readable medium of claim 17, wherein the operations include analyzing sequences of the static frames and sequences of the visual speaker anticipation cues.

19. The at least one non-transitory computer readable medium of claim 15, wherein,

the sensory input is a real time video stream captured by a camera local to a first videoconference endpoint, and

the sensory input is analyzed using at least one speaker anticipation model to yield the prediction score.

20. The at least one non-transitory computer readable medium of claim 19, wherein the at least one speaker anticipation model is a semantic representation model created by a machine learning algorithm after analysis of a plurality of signals including video data, audio data, and conference participant in-room location data from historical videoconferences taking place in a meeting room.