Adapting a continuous presence layout to a discussion situation

• US 9,516,272 B2
• Filed: 08/19/2014
• Issued: 12/06/2016
• Est. Priority Date: 03/31/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (Plantronics, Inc.)
• Inventors: Avni, Uri, Leviav, Eyal
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US14/463,506
• Publication Number:
  US 20140354764A1
• Time in Patent Office: 840 Days
• Field of Search: 345/178, 348/14.01, 348/14.07, 348/14.08, 348/14.09, 348/14.02, 348/14.03, 370/265
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A system and method is disclosed for adapting a continuous presence videoconferencing layout according to interactions between conferees. Using regions of interest found in video images, the arrangement of images of conferees may be dynamically arranged as displayed by endpoints. Arrangements may be responsive to various metrics, including the position of conferees in a room and dominant conferees in the videoconference. Video images may be manipulated as part of the arrangement, including cropping and mirroring the video image. As interactions between conferees change, the layout may be automatically rearranged responsive to the changed interactions.

Claims

1. A non-transitory computer readable medium containing executable instructions comprising instruction when executed cause a programmable device to:

determine automatically an interaction between a first conferee and a second conferee of a plurality of conferees of a continuous presence videoconference;

employ a layout for a continuous presence video image for a first endpoint responsive to the interaction between the first conferee and the second conferee;

assign a first video image corresponding to the first conferee to a first segment of the layout; and

assign a second video image corresponding to the second conferee to a second segment of the layout relative to the first video image in the continuous presence video image, responsive to the interaction between the first conferee and the second conferee; and

compose the continuous presence video image responsive to information related to the layout and the assignment of the first video image and the assignment of the second video image by an editor for presentation by the first endpoint.

2. The computer readable medium of claim 1, wherein the editor is located at a multipoint control unit.

3. The computer readable medium of claim 1, wherein the instructions further comprise instructions that when executed cause the programmable device to design the layout responsive to the determined interaction between the first conferee and the second conferee.

4. The computer readable medium of claim 3, wherein the instructions that when executed cause the programmable device to design the layout comprise instructions that when executed cause the programmable device to:

receive content from an endpoint corresponding to the first conferee; and

present the content in the continuous presence video image,

wherein the first segment and the second segment of the layout are adjacent to the content in the layout.

5. The computer readable medium of claim 4, wherein the instructions further comprise instructions that when executed cause the programmable device to:

process the first video image and the second video image to present the first conferee and the second conferee looking toward the content.

6. The computer readable medium of claim 1, wherein the instructions that when executed cause the programmable device to compose the continuous presence video image comprise instructions than when executed cause the programmable device to:

generate instructions for the first endpoint for creating the continuous presence video image at the first endpoint responsive to the interaction between the first conferee and the second conferee;

send the instructions to the first endpoint; and

create the continuous presence video image at the first endpoint responsive to the instructions.

7. The computer readable medium of claim 1, wherein the instructions that when executed cause the programmable device to compose the continuous presence video image comprise instructions than when executed cause the programmable device to:

create, by the editor, a continuous presence video image for the first endpoint responsive to the interaction between the first conferee and the second conferee; and

send the continuous presence video image to the first endpoint.

8. The computer readable medium of claim 1, wherein the instructions further comprise instructions that when executed cause the programmable device to:

employ a second layout for a continuous presence video image for a second endpoint of the continuous presence videoconference automatically responsive to the interaction between the first conferee and the second conferee, the continuous presence video image for the second endpoint different from the continuous presence video image for the first endpoint; and

display the continuous presence video image for the second endpoint at the second endpoint according to the second layout.

9. The computer readable medium of claim 1, wherein the instructions that when executed cause the programmable device to determine automatically an interaction between a first conferee and a second conferee of the plurality of conferees comprise instructions that when executed cause the programmable device to:

obtain, during a period of time, information related to audio energy obtained from each of a plurality of endpoints participating in the continuous presence videoconference; and

calculate, at the end of the period, a contribution of each endpoint of the plurality of endpoints to a total audio energy indication for the period.

10. The computer readable medium of claim 9, wherein the instructions that when executed cause the programmable device to determine automatically an interaction between a first conferee and a second conferee of the plurality of conferees further comprise instructions that when executed cause the programmable device to:

define the interaction between the first conferee and the second conferee during the period as a first type of interaction responsive to the first and the second conferee each contributing a portion of the total audio energy indication within a predefined range; and

define the interaction between the first conferee and the second conferee during the period as a second type of interaction responsive to the first and the second conferee each contributing a portion of the total audio energy indication less than a predefined threshold.

11. A non-transitory computer readable medium containing executable instructions comprising instructions that when executed cause a programmable device to:

obtain periodically an activity indication that is related to the activity of each conferee of a plurality of conferees during a first period of time;

store the activity indication of each conferee;

determine a current mode of interaction by periodically analyzing the stored activity indications that were stored during a second period of time;

store the current mode of interaction;

determine a recommendation whether to adapt a layout for a continuous presence video image, to be used for a next third period of time to reflect the current mode of interaction by periodically analyzing the current mode of interaction stored during a current third period of time; and

transfer the recommendation to an editor processor that is configured to compose the continuous presence video image according to the layout.

12. The computer readable medium of claim 11, wherein the first period is related to an audio frame rate used by an endpoint associated with one of the conferees.

13. The computer readable medium of claim 11, wherein the second period is an integer multiple of the first period.

14. The computer readable medium of claim 11, wherein the third period is an integer multiple of the second period.

15. The computer readable medium of claim 11, wherein the duration of the third period is equal to the duration of the second period.

16. The computer readable medium of claim 11, wherein the activity indication of each conferee reflects audio energy obtained from an endpoint that is associated with that conferee.

17. The computer readable medium of claim 11, wherein the instructions that when executed cause the programmable device to determine a current mode of interaction comprise instructions that when executed cause the programmable device to:

calculate a periodic activity indication of each conferee that reflects activity of that conferee during the second period of time; and

compare the periodic activity indication of each conferee.

18. The computer readable medium of claim 17, wherein the instructions that when executed cause the programmable device to determine a current mode of interaction further comprise instructions that when executed cause the programmable device to:

determine that the current mode of interaction is a first mode of interaction between a first conferee and a second conferee when the periodic activity indications of the first and the second conferees are substantially higher than the periodic activity indications of every other conferee.

19. The computer readable medium of claim 18, wherein the layout that reflects the first mode of interaction comprises a plurality of segments wherein two of the plurality of segments are larger than the others.

20. The computer readable medium of claim 17, wherein the instructions that when executed cause the programmable device to determine a current mode of interaction further comprise instructions that when executed cause the programmable device to:

determine that the current mode of interaction is a second type of interaction when the periodic activity indications of the plurality of conferees are substantially similar to each other.

21. The computer readable medium of claim 20, wherein the layout that reflects the second type of interaction comprises a plurality of substantially similarly sized segments.

22. The computer readable medium of claim 17,

wherein the activity indication of each conferee reflects audio energy obtained from an endpoint that is associated with that conferee, and

wherein the periodic activity indication of each conferee is calculated as a ration between a total audio energy of that conferee and a total audio energy of all of the plurality of conferees during the second period of time.

23. The computer readable medium of claim 17,

wherein the activity indication of each conferee reflects audio energy obtained from an endpoint that is associated with that conferee;

and

wherein the periodic activity indication of each conferee is calculated responsive to how often the audio energy of that conferee was the highest audio energy of all conferees during the second period of time.

24. A method for defining a layout for a continuous presence video image automatically for a first endpoint of a continuous presence videoconference responsive to an interaction between a plurality of conferees, comprising:

determining an interaction between two or more conferees of a plurality of conferees of a continuous presence videoconference;

creating a layout for a continuous presence video image for a first endpoint responsive to the interaction between the two or more conferees, wherein the layout comprises a plurality of segments;

generating a plurality of video images, wherein a video image of the plurality of video images corresponds to each of the two or more conferees;

assigning an video image of the plurality of video images to a segment of the plurality of segments of the layout, wherein the assignment of the video image is responsive to the interaction between the two or more conferees; and

generating the continuous presence video image responsive to information related to the layout and the assignment of the video image by an editor for presentation by the first endpoint.

25. A method for automatically determining a mode of interaction between two or more conferees of a plurality of conferees in a continuous presence videoconference, comprising:

determining the activity of each conferee of a plurality of conferees during a first period of time;

generating an activity indication that is related to the activity of each conferee;

selecting a current mode of interaction by periodically analyzing the generated activity indications that were generated during a second period of time;

determining a recommendation whether to adapt a layout for a continuous presence video image, to be used for a next third period of time to reflect the current mode of interaction by periodically analyzing the current mode of interaction selected during a current third period of time; and

conveying the recommendation to an editor processor that is configured to compose the continuous presence video image according to the layout.

26. A system for adapting a continuous presence layout for a discussion situation, comprising:

a multipoint control unit (MCU), configured to;

monitor activity of each conferees of a plurality of conferees of a continuous presence videoconference;

determine a plurality of video images to display in a continuous presence video image;

obtain information regarding a region of interest for each video image of the plurality of video images;

modify a continuous presence layout according to the information regarding the region of interest of the video image and according to the activity of the plurality of conferees; and

generate instructions for creating the continuous presence video image based on the continuous presence layout; and

a plurality of endpoints, wherein each endpoint of the plurality of endpoints is configured to;

capture the activity of the plurality of conferees; and

construct the continuous presence video image responsive to information related to the layout.

27. The system of claim 26, wherein the region of interest is based on the activity of the plurality of conferees.

28. The system of claim 26, wherein information regarding the region of interest of the video image comprises a relative location of the region of interest in the video image and a size of the region of interest in the video image.

29. A multipoint control unit (MCU) for adapting a continuous presence layout for a discussion situation comprising:

a control module (CM), configured to;

monitor the activity of each conferees of a plurality of conferees;

determine a plurality of video images to display in a continuous presence video image;

obtain information regarding a region of interest for each video image of the plurality of video images, wherein the region of interest is based on the activity of the plurality of conferees;

modify a continuous presence layout according to the information regarding the region of interest of the video image and according to the activity of the plurality of conferees; and

generate instructions for creating a continuous presence video image based on the continuous presence layout.