Dynamic adaption of a continuous presence videoconferencing layout based on video content

• US 8,446,454 B2
• Filed: 01/07/2010
• Issued: 05/21/2013
• Est. Priority Date: 05/21/2007
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (Plantronics, Inc.)
• Inventors: Decker, Eran, Wagner, Niv, Barazany, Avihay, Vistuch, Roni
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US12/683,806
• Publication Number:
  US 20100103245A1
• Time in Patent Office: 1,230 Days
• Field of Search: 324/642, 342/22, 348/14.08, 348/14.09, 348/14.16, 348/556, 358/1.18, 370/263, 375/240.01, 375/240.08, 375/240.21, 375/240.2, 375/240.24, 382/103, 382/238, 382/239, 382/282, 382/173, 382/236, 382/254, 386/278, 600/458, 715/806, 345/55, 702/104
• US Class Current: 348/14.09
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

Dynamically adapting a continuous presence (CP) layout in a videoconference enhances a videoconferencing experience by providing optimum visibility to regions of interest within the CP layout and ignoring regions of no interest. Based on the CP layout, a CP video image can be built, in which a conferee at a receiving endpoint can observe, simultaneously, several other participants'"'"' sites in the conference. For example, more screen space within the CP layout is devoted to presenting the participants in the conference and little or no screen space is used to present an empty seat, an empty room, or an unused portion of a room. Aspect ratios of segments of the CP layout (e.g., landscape vs. portrait) can be adjusted to optimally present the regions of interest. The CP layout can be adjusted as regions of interest change depending on the dynamics of the video conference.

Claims

1. An apparatus for building, in a video conference, a continuous presence (CP) video image, comprising:

a video module and a control module, wherein;

the video module determines a region of interest (ROI) in at least one video image received from an endpoint and provides the control module video parameters for the ROI;

the control module designs a CP layout based on the parameters for the ROI and instructs the video module how to manipulate the at least one video image, and how to place the manipulated at least one video image into the CP video image according to the designed CP layout; and

the video module builds the CP video image and sends the CP video image toward a display unit of the receiving endpoint.

2. The apparatus of claim 1, wherein manipulating the at least one video image comprises at least one manipulation selected from a group consisting of cropping and scaling.

3. The apparatus of claim 1, wherein control module designs the CP layout at least in part according to management instructions.

4. The apparatus of claim 1, wherein CP layout includes only video images that have an identified ROI.

5. The apparatus of claim 1, wherein the video module comprises a face detector processor.

6. The apparatus of claim 1, wherein the video module is associated with the endpoint that sends video images.

7. The apparatus of claim 6, wherein the video image is a video frame received from the associated endpoint.

8. The apparatus of claim 1, wherein the video module is associated with an endpoint that receives the CP video image.

9. The apparatus of claim 1, wherein the video module is embedded within a multipoint control unit.

10. The apparatus of claim 1, wherein the CP layout is periodically updated.

11. The apparatus of claim 1, wherein the CP layout is adapted to the size of the at least one ROI.

12. The apparatus of claim 1, wherein the video module periodically captures a frame of decoded video data received from the endpoint and determines a new ROI in the frame of decoded video data.

13. The apparatus of claim 12, wherein the control module adjusts the design of the CP layout based on the new ROI.

14. A video conferencing system, for adapting a continuous presence (CP) video image, comprising:

a plurality of endpoints;

one or more region-of-interest (ROI) detector modules that detect an ROI in video images generated by at least one of the endpoints;

a control unit comprising a control module that designs a CP layout based on parameters of the ROIs and instructs one or more editor modules how to manipulate the at least one video image, and how to place the at least one manipulated video image in the CP video image according to the designed CP layout.

15. The video conferencing system of claim 14, wherein the control unit is a multipoint control unit (MCU).

16. The video conferencing system of claim 14, wherein to manipulating the at least one video image comprises at least one manipulation selected from a group consisting of cropping and scaling.

17. The video conferencing system of claim 14, further comprising a video module that encodes the CP video image and sends the encoded CP video image toward a display unit of at least one of the endpoints.

18. The video conferencing system claim 14, wherein at least one of the ROI detectors comprises a face detector processor.

19. The video conferencing system claim 14, wherein at least one of the one or more ROI detectors is associated with one of the endpoints.

20. The video conferencing system claim 14, wherein the CP layout is designed based in part on the size of the at least one ROI.

21. The video conferencing system of claim 14, wherein the ROI detector periodically captures a video frame generated by one of the endpoints to detect changes in the ROI in the video image.

22. The video conferencing system of claim 21, wherein the CP layout is periodically adjusted based on changes in the ROI of at least one video image.

25. The method of claim 22, wherein the management instructions are based at least in part on audio energy received from the one or more transmitting endpoints.

23. A method for adapting a video conferencing continuous presence (CP) layout, comprising:

determining one or more regions of interest (ROIs) in one or more video images received from one or more transmitting endpoints;

designing a CP video image to be displayed at receiving endpoint based at least in part on the ROIs;

building the CP video image from the video images received from the one or more transmitting endpoints; and

delivering the CP video image toward a display unit of the receiving endpoint.

24. The method of claim 23, wherein designing the CP video image is based at least in part on management instructions.

26. The method of claim 23, wherein determining one or more ROIs is executed based at least in part on data generated by a face detector.

27. The method of claim 23, wherein the CP video image includes video images that have one or more ROIs.

28. The method of claim 23, wherein the CP video image is periodically updated.

29. The method of claim 23, wherein designing a CP video image is based at least in part on the size of at least one ROI.

30. The method of claim 23, wherein determining one or more ROIs comprises capturing a video frame transmitted by one of the transmitting endpoints and determining an ROI in the capture video frame.

31. The method of claim 23, wherein delivering the CP video image comprises encoding the CP video image and transmitting the encoded CP video image toward the display unit of the receiving endpoint.

32. The method of claim 23, wherein determining one or more ROIs comprises periodically capturing a video frame transmitted by one of the transmitting endpoints and determining an ROI in the capture video frame.

33. The method of claim 32, wherein designing a CP video image comprises periodically adjusting the CP video image based on the ROI in the captured video frame.

34. The method of claim 33, wherein adjusting comprises changing the size of a received video image within the CP video image.

35. The method of claim 31, wherein adjusting comprises eliminating a received video image from the CP video image.