Multi-way video conferencing user interface

• US 8,464,165 B2
• Filed: 12/06/2010
• Issued: 06/11/2013
• Est. Priority Date: 06/25/2004
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: van Os, Marcel, Westen, Peter T., Chaudhri, Imran A.
• Primary Examiner(s): Nunez, Jordany
• Application Number:
  US12/961,150
• Publication Number:
  US 20110078591A1
• Time in Patent Office: 918 Days
• Field of Search: 715/756, 715/753, 345/419, 348/14.08, 709/204
• US Class Current: 715/756
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 21/4438 Window management, e.g. event handling following interaction with the user interface
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A videoconferencing application includes a user interface that provides multiple participant panels, each of which is displayed with perspective with the panels appearing to be angled with respect to the user interface window. The participant panels display live video streams from remote participants. A two-way layout provides two participant panels for two remote participants, each of which is angled inwardly towards a center position. A three-way layout provides three participant panels for three remote participants, with a left, center and right panel, with the left and right panels angled inwardly towards a center position.

Claims

1. A computer-implemented method for displaying participant panels in a user interface window of a videoconferencing application, the method comprising:

receiving in a single computer at least two live video streams, each live video stream associated with a remote participant;

generating in the computer a single user interface window;

determining a three dimensional model view for displaying participant panels;

generating in the three dimensional model view a participant panel for each of the live video streams, each participant panel associated with a remote participant, the participant panels including a first participant panel and a second participant panel;

determining a viewport for rendering the three dimensional model view, including the first participant panel and the second participant panel, in the single user interface window;

determining a perspective projection of the participant panels in the viewport such that a projected size of each participant panel in the viewport is the same number of units as a size of the participant panel in the model view;

causing for display the first participant panel in the user interface window; and

causing for display the second participant panel in the user interface window,

wherein the first participant panel and the second participant panel are displayed in a perspective orientation,

wherein displaying the first participant panel and the second participant panel in the perspective orientation comprises;

defining a plurality of model parameters that place the first participant panel and the second participant panel in the three dimensional model view having an X-axis, a Y-axis, and a Z-axis, the model parameters including an angle of the first participant panel and the second participant panel with respect to the X-axis;

determining the viewport for rendering the three dimensional model view as a function of a size of the user interface window;

determining a projection frustum of the three dimensional model view including the participant panels as a function essentially of a size of the viewport and the model parameters scaled by the size of the viewport;

rendering the projection frustum in the viewport; and

displaying the rendered frustum in the user interface window,

wherein there is a third participant panel and the projection frustum is defined by the variables FrustumTop, FrustumBottom, FrustumRight, FrustumLeft, FrustumNear, and FrustumFar, according to the following equations;

2. The computer-implemented method of claim 1, further comprising:

displaying a spotlight effect on one of the participant panels;

responsive to displaying the spotlight effect, displaying the live video stream associated with the participant panel; and

displaying a foreground reflection of the spotlight effect in front of the participant panel.

3. The computer-implemented method of claim 1, further comprising:

displaying in the user interface window a user panel containing a live video stream of the user of the videoconferencing application in a foreground between the first participant panel and the second participant panel.

4. The computer-implemented method of claim 1, further comprising:

animating a transition between the user interface window displaying the first participant panel and the second participant panel in a coplanar orientation and displaying the first participant panel and the second participant panel in a perspective orientation; and

animating a transition between the user interface window displaying the first participant panel and the second participant panel in the perspective orientation and displaying the first participant panel and the second participant panel in the coplanar orientation.

5. The computer-implemented method of claim 4, further comprising:

animating the transition in response to a user selection of a mode of operation of the videoconferencing application.

6. The computer-implemented method of claim 1, wherein the projection frustum is defined by the variables FrustumTop, FrustumBottom, FrustumRight, FrustumLeft, FrustumNear, and FrustumFar, according to the following equations:

7. The computer-implemented method of claim 1, wherein the projection frustum has a top boundary, and a virtual camera is placed such that a view axis of the virtual camera is above the top boundary, and parallel to the z-axis.

8. The computer-implemented method of claim 1, further comprising:

displaying a participant panel at a first size, with a text banner, the text banner having a text height and a first width;

receiving a reduction in size of the user interface window;

displaying the participant panel at a reduced size in response to the reduction in size of the user interface window; and

displaying the text banner over the participant panel with a retained size.

9. The computer-implemented method of claim 1, wherein displaying the first participant panel and the second participant panel in the perspective orientation comprises:

defining a plurality of model parameters that place the first participant panel and the second participant panel in a three dimensional model view having an X-axis, a Y-axis, and a Z-axis, the model parameters including an angle of the first participant panel and the second participant panel with respect to the X-axis;

determining the viewport for rendering the three dimensional model view as a function of a size of the user interface window;

determining a projection frustum of the three dimensional model view, wherein determining the projection frustum comprises;

determining for each participant panel, a panel height and a panel width, wherein the panel height and panel width are based at least on a width of the view port; and

determining a plurality of frustum boundaries for placement of a camera, wherein the plurality of frustum boundaries are based at least on a scaling factor, the scaling factor a function of the panel width;

rendering the projection frustum in the viewport; and

displaying the rendered frustum in the user interface window.

10. A computer program product comprising a non-transitory computer-readable storage medium including computer executable instructions for controlling a computer system to execute a videoconferencing application for displaying participant panels in a user interface window of a videoconferencing application, by performing the operations of:

receiving in a single computer at least two live video streams, each live video stream associated with a remote participant;

generating in the computer a single user interface window;

determining a three dimensional model view for displaying participant panels;

generating in the three dimensional model view a participant panel for each of the live video streams, each participant panel associated with a remote participant, the participant panels including a first participant panel and a second participant panel;

determining a viewport for rendering the three dimensional model view, including the first participant panel and the second participant panel, in the single user interface window;

determining a perspective projection of the participant panels in the viewport such that a projected size of each participant panel in the viewport is the same number of units as a size of the participant panel in the model view;

causing for display the first participant panel in the user interface window; and

causing for display the second participant panel in the user interface window,

wherein the first participant panel and the second participant panel are displayed in a perspective orientation,

wherein displaying the first participant panel and the second participant panel in the perspective orientation comprises;

defining a plurality of model parameters that place the first participant panel and the second participant panel in the three dimensional model view having an X-axis, a Y-axis, and a Z-axis, the model parameters including an angle of the first participant panel and the second participant panel with respect to the X-axis;

determining the viewport for rendering the three dimensional model view as a function of a size of the user interface window;

determining a projection frustum of the three dimensional model view including the participant panels as a function essentially of a size of the viewport and the model parameters scaled by the size of the viewport;

rendering the projection frustum in the viewport; and

displaying the rendered frustum in the user interface window,

wherein there is a third participant panel and the projection frustum is defined by the variables FrustumTop, FrustumBottom, FrustumRight, FrustumLeft, FrustumNear, and FrustumFar, according to the following equations;

11. The computer program product of claim 10, wherein the first participant panel and the second participant panel are displayed in a perspective orientation.

12. The computer program product of claim 11, wherein displaying the first participant and the second participant panel in the perspective orientation comprises:

defining a plurality of model parameters that place the first participant panel and the second participant panel in the three dimensional model view having an X-axis, a Y-axis axis, and a Z-axis, the model parameters including an angle of the first participant panel and the second participant panel with respect to the X-axis;

determining the viewport for rendering the three dimensional model view as a function of a size of the user interface window;

determining a projection frustum of the three dimensional model view including the participant panels as a function essentially of a size of the viewport and the model parameters scaled by the size of the viewport;

rendering the projection frustum in the viewport; and

displaying the rendered frustum in the user interface window.

13. The computer program product of claim 10, wherein the computer readable storage medium further includes computer executable instructions for:

displaying in the user interface window a user panel containing a live video stream of the user of the videoconferencing application in a foreground between the first participant panel and the second participant panel.

14. A computer system for displaying participant panels in a user interface window of a videoconferencing application, the system comprising:

a computer processor; and

a computer-readable storage medium storing instructions when executed by the processor perform a method, the method comprising;

receiving in a single computer at least two live video streams, each live video stream associated with a remote participant;

generating in the computer a single user interface window;

determining a three dimensional model view for displaying participant panels;

generating in the three dimensional model view a participant panel for each of the live video streams, each participant panel associated with a remote participant, the participant panels including a first participant panel and a second participant panel;

determining a viewport for rendering the three dimensional model view, including the first participant panel and the second participant panel, in the single user interface window;

determining a perspective projection of the participant panels in the viewport such that a projected size of each participant panel in the viewport is the same number of units as a size of the participant panel in the model view;

causing for display the first participant panel in the user interface window; and

causing for display the second participant panel in the user interface window,

wherein the first participant panel and the second participant panel are displayed in a perspective orientation,

wherein displaying the first participant panel and the second participant panel in the perspective orientation comprises;

defining a plurality of model parameters that place the first participant panel and the second participant panel in the three dimensional model view having an X-axis, a Y-axis, and a Z-axis, the model parameters including an angle of the first participant panel and the second participant panel with respect to the X-axis;

determining the viewport for rendering the three dimensional model view as a function of a size of the user interface window;

determining a projection frustum of the three dimensional model view including the participant panels as a function essentially of a size of the viewport and the model parameters scaled by the size of the viewport;

rendering the projection frustum in the viewport; and

displaying the rendered frustum in the user interface window,

wherein there is a third participant panel and the projection frustum is defined by the variables FrustumTop, FrustumBottom, FrustumRight, FrustumLeft, FrustumNear, and FrustumFar, according to the following equations;

15. The computer system of claim 14, wherein the first participant panel and the second participant panel are displayed in a perspective orientation.

16. The computer system of claim 15, wherein displaying the first participant and the second participant panel in the perspective orientation comprises:

defining a plurality of model parameters that place the first participant panel and the second participant panel in the three dimensional model view having an X-axis, a Y-axis, and a Z-axis, the model parameters including an angle of the first participant panel and the second participant panel with respect to the X-axis;

determining the viewport for rendering the three dimensional model view as a function of a size of the user interface window;

determining a projection frustum of the three dimensional model view including the participant panels as a function essentially of a size of the viewport and the model parameters scaled by the size of the viewport;

rendering the projection frustum in the viewport; and

displaying the rendered frustum in the user interface window.

17. The computer system of claim 14, wherein the computer readable storage medium further includes instructions for:

displaying in the user interface window a user panel containing a live video stream of the user of the videoconferencing application in a foreground between the first participant panel and the second participant panel.

18. A computer-implemented method for displaying participant panels in a user interface window of a videoconferencing application, the method comprising:

receiving in a single computer at least two live video streams, each live video stream associated with a remote participant;

generating in the computer a single user interface window;

determining a three dimensional model view for displaying participant panels;

generating in the three dimensional model view a participant panel for each of the live video streams, each participant panel associated with a remote participant, the participant panels including a first participant panel and a second participant panel;

determining a viewport for rendering the three dimensional model view, including the first participant panel and the second participant panel, in the single user interface window;

determining a projection of the participant panels in the viewport such that a projected size of each participant panel in the viewport is the same number of units as a size of the participant panel in the model view;

causing for display the first participant panel in the user interface window; and

causing for display the second participant panel in the user interface window,

wherein the first participant panel and the second participant panel are displayed in a perspective orientation, comprising;

defining a plurality of model parameters that place the first participant panel and the second participant panel in the three dimensional model view having an X-axis, a Y-axis, and a Z-axis, the model parameters including an angle of the first participant panel and the second participant panel with respect to the X-axis;

determining the viewport for rendering the three dimensional model view as a function of a size of the user interface window;

determining a projection frustum of the three dimensional model view including the participant panels as a function essentially of a size of the viewport and the model parameters scaled by the size of the viewport;

rendering the projection frustum in the viewport; and

displaying the rendered frustum in the user interface window, and

wherein there is a third participant panel and the projection frustum is defined by the variables FrustumTop, FrustumBottom, FrustumRight, FrustumLeft, FrustumNear, and FrustumFar, according to the following equations;

19. The computer-implemented method of claim 18, wherein the projection frustum is defined by the variables FrustumTop, FrustumBottom, FrustumRight, FrustumLeft, FrustumNear, and FrustumFar, according to the following equations: