Heterogeneous video conferencing

• US 7,949,117 B2
• Filed: 04/28/2005
• Issued: 05/24/2011
• Est. Priority Date: 04/28/2005
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Normile, Jim, Tung, Berkat Shing, Jeong, Hyeonkuk, Salsbury, Ryan, Abuan, Joe
• Primary Examiner(s): Gauthier; Gerald
• Assistant Examiner(s): ISLAM, MOHAMMAD K
• Application Number:
  US11/118,297
• Publication Number:
  US 20060245377A1
• Time in Patent Office: 2,217 Days
• Field of Search: 705/8, 705/9, 707/10, 709/204, 379/202.01, 348/14.01, 348/14.02, 370260-267
• US Class Current: 379/202.01
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04Q 2213/13093 Personal computer, PC
• CPC Class Codes: H04Q 2213/1324 Conference call
• CPC Class Codes: H04Q 3/0045 involving hybrid, i.e. a mixture of public and private, or multi-vendor systems

Abstract

Some embodiments provide an architecture for establishing a multi-participant conference. This architecture has one participant'"'"'s computer in the conference act as a central content distributor for the conference. The central distributor receives data (e.g., video and/or audio streams) from the computer of each other participant, and distributes the received data to the computers of all participants. In some embodiments, the central distributor receives A/V data from the computers of the other participants. From such received data, the central distributor of some embodiments generates composite data (e.g., composite image data and/or composite audio data) that the central distributor distributes back to the participants. The central distributor in some embodiments can implement a heterogeneous audio/video conference. In such a conference, different participants can participate in the conference differently. For instance, different participants might use different audio or video codecs. Moreover, in some embodiments, one participant might participate in only the audio aspect of the conference, while another participant might participate in both audio and video aspects of the conference.

Claims

1. A heterogeneous conference system comprising:

first and second non-focus point computers for participating in a heterogeneous conference; and

a focus point computer for participating in said heterogeneous conference, wherein said participating comprises (i) receiving conference content data from first and second remote conference participants through the first and second remote non-focus point computers, respectively, (ii) receiving conference content data at the focus point computer from a local conference participant that is locally using the focus point computer to participate in the conference, (iii) generating composite conference content data streams that include the conference content data received from the remote conference participants and the conference content data received from the local conference participant that uses the focus point computer, and (iv) distributing the composite conference content data streams to the non-focus point computers,

wherein the first non-focus point computer is for providing conference content data from the first remote conference participant to the focus point computer, said conference content data based on a first set of conference parameters, wherein the second non-focus point computer is for providing conference content data from the second remote conference participant that is based on a different second set of conference parameters to the focus point computer.

2. The system of claim 1, wherein the first and second sets of conference parameters comprise different types of conference content data being provided, wherein one of said different types of conference content data is audio data and another of said different types of conference content data is video data.

3. The system of claim 1, wherein the first and second sets of conference parameters comprise sizes for frames of the conference content data, wherein the first and second non-focus point computers are for providing different sized frames to the focus point computer.

4. The system of claim 3, wherein the focus point computer comprises more than one resizer for resizing content data from the first and second non-focus point computers.

5. The system of claim 1, wherein the conference content data comprises video data, wherein the conference parameters comprise types of video encoders used by the first and second non-focus point computers, wherein the first and second non-focus point computers are for using different types of video encoders.

6. The system of claim 1, wherein the conference content data comprises audio data, wherein the conference parameters comprise types of audio encoders used by the first and second non-focus point computers, wherein the first and second non-focus point computers are for using different types of audio encoders.

7. The system of claim 1, wherein the conference content data from the first non-focus point computer comprises video data and the conference content data from the second non-focus point computer does not comprise video data.

8. The system of claim 7, wherein the focus point computer comprises a video decoder, wherein the video decoder defines a video-decoding pipeline for the first non-focus point computer and foregoes defining a video-decoding pipeline for the second non-focus point computer.

9. The system of claim 8, wherein a composite conference content data stream from the focus point computer to the first non-focus point computer comprises a composite video frame that does not contain video data from the second non-focus point computer.

10. The system of claim 8, wherein a conference content data stream from the focus point computer to the second non-focus point computer only comprises audio data.

11. A computer readable medium storing a computer program for performing a heterogeneous conference between a focus point computer and at least first and second non-focus point computers, the computer program comprising sets of instructions for:

receiving, at said focus point computer, conference content data from at least first and second remote conference participants through the first and second non-focus point computers, respectively;

capturing, locally at said focus point computer, conference content data from a local conference participant that is using the focus point computer to participate in the conference;

generating, at said focus point computer, composite conference content data streams that include the conference content data received from the remote first and second conference participants and the locally captured conference content data from the local conference participant; and

distributing, from said focus point computer, the composite conference content data streams to the non-focus point computers,

wherein the first non-focus point computer provides conference content data that is based on a first set of conference parameters while the second non-focus point computer provides conference content data that is based on a different second set of conference parameters, wherein said focus point computer and non-focus point computers are participant computers of said heterogeneous conference.

12. The computer readable medium of claim 11, wherein the first and second conference parameters comprise different types of conference content data being provided, wherein one of said different types of conference content data is audio data and another of said different types of conference content data is video data.

13. The computer readable medium of claim 11, wherein the first and second conference parameters comprise sizes for frames of the conference content data, wherein the first and second non-focus point computers provide different sized frames to the focus point computer.

14. The computer readable medium of claim 13, wherein the focus point computer comprises more than one resizer for resizing conference content data from the first and second non-focus point computers.

15. The computer readable medium of claim 11, wherein the conference content data comprises video data, wherein the conference parameters comprise types of video encoders used by the first and second non-focus point computers, wherein the first and second non-focus point computers use different types of video encoders.

16. The computer readable medium of claim 11, wherein the conference content data comprises audio data, wherein the conference parameters comprise types of audio encoders used by the first and second non-focus point computers, wherein the first and second non-focus point computers use different types of audio encoders.

17. The computer readable medium of claim 11, wherein the conference content data from the first non-focus point computer comprises video data and the conference content data from the second non-focus point computer does not comprise video data.

18. The computer readable medium of claim 17, wherein the focus point computer comprises a video decoder, wherein the video decoder defines a video-decoding pipeline for the first non-focus point computer and foregoes defining a video-decoding pipeline for the second non-focus point computer.

19. The computer readable medium of claim 18, wherein the conference content data stream distributed from the focus point computer to the first non-focus point computer comprises a composite frame that does not contain video data from the second non-focus point computer.

20. The computer readable medium of claim 18, wherein a conference content data stream from the focus point computer to the second non-focus point computer only comprises audio data.

21. A method for providing a conferencing application of a particular device participating in a conference comprising a plurality of devices, the method comprising:

providing a local conference content capture module for receiving conference content data at the particular device from a conference participant that is using the particular device to participate in the conference;

providing a first input module for receiving, at the particular device, conference content data based on a first set of conference parameters from a first remote conference participant through a first device of the plurality of devices;

providing a second input module for receiving, at the particular device, conference content data based on a second set of conference parameters from a second remote conference participant through a second device of the plurality of devices;

providing a compositing module for producing a composite signal comprising the conference content data from the local conference content capture module and the received conference content data from at least one of the first and second devices; and

providing an output module for distributing the composite signal to at least one of the first and second remote conference participants through the first and second devices, respectively.

22. The method of claim 21, wherein at least one of said conference parameters specifies one of an encoding type and a bit rate for the conference content data.

30. The method of claim 21, wherein the output module is further for providing a presentation of the composite signal to an output device at the particular device.

23. A method of conferencing between at least first, second, and third conference participants through at least first, second, and third computers, respectively, the method comprising:

receiving, at the first computer, (i) local conference content data from the first conference participant that is using the first computer to participate in the conference, (ii) remote conference content data from the second conference participant through the second computer based on a first set of conference parameters, and (iii) remote conference content data from the third conference participant through the third computer based on a different second set of conference parameters;

generating, at the first computer, composite conference content data streams comprising the local conference content data and the remote conference content data;

distributing, from the first computer, the composite conference content data streams to the second and third conference participants through the second and third computers, respectively; and

displaying, at the first computer, a composite conference content data stream for the first conference participant.

24. The method of claim 23, wherein the conference content data received from the second computer comprises video data and the conference content data received from the third computer does not comprise video data.

25. The method of claim 23, wherein the first and second sets of conference parameters comprise different types of conference content data being provided, wherein one of said different types of conference content data is audio data and another of said different types of conference content data is video data.

26. The method of claim 23, wherein a composite conference content data stream distributed to the second computer does not comprise conference content data received from the second computer.

27. A computer readable medium storing a computer program which when executed at a focus point computer performs a heterogeneous conference between the focus point computer and at least first, second, and third non-focus point computers, the computer program comprising sets of instructions for:

receiving (i) conference audio and video data from a first remote participant that is using the first non-focus point computer to participate in the conference, (ii) conference audio and video data from a second remote participant that is using the second non-focus point computer to participate in the conference, and (iii) only conference audio data from a third remote participant that is using the third non-focus point computer to participate in the conference;

capturing conference audio and video data from a local conference participant that is using the focus point computer to participate in the conference;

generating (i) a first composite conference video data stream that includes the locally captured conference video data and the conference video data from the first non-focus point computer and (ii) a second composite conference video data stream that includes the locally captured conference video data and the conference video data from the second non-focus point computer;

generating composite audio data streams for each of the remote conference participants, wherein the composite audio data stream for a particular remote conference participant includes (i) the locally captured conference audio data and (ii) the conference audio data received from the remote participants other than the particular participant; and

distributing (i) the second composite conference video data stream and the composite audio stream for the first participant to the first non-focus point computer, (ii) the first composite conference video data stream and the composite audio stream for the second participant to the second non-focus point computer, and (iii) only the composite audio stream for the third participant to the third non-focus point computer.

28. The computer readable medium of claim 27, wherein the computer program further comprises a set of instructions for displaying the received video and locally captured video on a display device at the focus point computer.

29. The computer readable medium of claim 27, wherein the computer program further comprises sets of instructions for:

generating a composite audio data stream for the local conference participant that includes the conference audio data received from each of the remote participants; and

playing the composite audio data stream for the local conference participant on a set of speakers at the focus point computer.