Teleconferencing system in which location video mosaic generator sends combined local participants images to second location video mosaic generator for displaying combined images

• US 5,802,294 A
• Filed: 06/07/1996
• Issued: 09/01/1998
• Est. Priority Date: 10/01/1993
• Status: Expired due to Term

• Current Assignee: Pragmatus AV LLC (Pragmatus, LLC)
• Original Assignee: Vicor Corporation
• Inventors: Lantz, Keith A., Lauwers, J. Chris, Burns, Emmett R., Burnett, Gerald J., Ludwig, Lester F.
• Primary Examiner(s): Lee, Thomas C.
• Assistant Examiner(s): Kim, Ki S.
• Application Number:
  US08/660,461
• Time in Patent Office: 816 Days
• Field of Search: 395/200.2, 395/2.69, 395/155, 395/331, 395/200.34, 395/200.68, 370/259, 370/260, 370/267, 348/15, 379/202
• US Class Current: 709/204
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04L 12/1813 for computer conferences, e.g. chat rooms instant messaging H04L51/04; protocols for multimedia communication H04L65/1101; arrangements for multi-party communication H04L65/403; telephonic conference arrangements H04M3/56; television conference systems H04N7/15
• CPC Class Codes: H04L 12/1818 Conference organisation arrangements, e.g. handling schedules, setting up parameters needed by nodes to attend a conference, booking network resources, notifying involved parties
• CPC Class Codes: H04L 12/1822 Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 12/1831 Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status
• CPC Class Codes: H04L 12/1836 with heterogeneous network architecture
• CPC Class Codes: H04L 12/185 with management of multicast group membership
• CPC Class Codes: H04L 51/00 User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04M 2201/38 Displays
• CPC Class Codes: H04M 2203/5018 Initiating a conference during a two-party conversation, i.e. three-party-service or three-way-call computer conference organisation arrangements H04L12/1818
• CPC Class Codes: H04M 2203/5027 Dropping a party from a conference conducting a computer conference, e.g. admission detection H04L12/1822
• CPC Class Codes: H04M 3/42221 : Conversation recording systems at the subscriber's set H04M1/656
• CPC Class Codes: H04M 3/42323 : PBX's with CTI arrangements
• CPC Class Codes: H04M 3/4285 : Notifying, informing or entertaining a held party while on hold, e.g. Music On Hold
• CPC Class Codes: H04M 3/5307 : for recording messages comprising any combination of audio and non-audio components
• CPC Class Codes: H04M 3/562 : where the conference facilities are distributed
• CPC Class Codes: H04M 3/563 : User guidance or feature selection
• CPC Class Codes: H04M 3/564 : whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/567 : Multimedia conference systems
• CPC Class Codes: H04M 7/006 : Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42
• CPC Class Codes: H04M 7/12 : for working between exchanges having different types of switching equipment, e.g. power-driven and step by step or decimal and non-decimal
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/148 : Interfacing a video terminal to a particular transmission medium, e.g. ISDN
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04N 7/152 : Multipoint control units therefor
• CPC Class Codes: H04Q 2213/1305 : Software aspects
• CPC Class Codes: H04Q 2213/13093 : Personal computer, PC
• CPC Class Codes: H04Q 2213/13175 : Graphical user interface [GUI], WWW interface, visual indication
• CPC Class Codes: H04Q 2213/1319 : Amplifier, attenuation circuit, echo suppressor
• CPC Class Codes: H04Q 2213/1322 : PBX
• CPC Class Codes: H04Q 2213/1324 : Conference call
• CPC Class Codes: H04Q 2213/1326 : Consultation call, broker's call, call hold, toggling
• CPC Class Codes: H04Q 2213/1332 : Logic circuits
• CPC Class Codes: H04Q 2213/13337 : Picturephone, videotelephony
• CPC Class Codes: H04Q 2213/13389 : LAN, internet

Abstract

A multimedia collaboration system that integrates separate real-time and asynchronous networks--the former for real-time audio and video, and the latter for control signals and textual, graphical and other data--in a manner that is interoperable across different computer and network operating system platforms and which closely approximates the experience of face-to-face collaboration, while liberating the participants from the limitations of time and distance. These capabilities are achieved by exploiting a variety of hardware, software and networking technologies in a manner that preserves the quality and integrity of audio/video/data and other multimedia information, even after wide area transmission, and at a significantly reduced networking cost as compared to what would be required by presently known approaches. The system architecture is readily scalable to the largest enterprise network environments. It accommodates differing levels of collaborative capabilities available to individual users and permits high-quality audio and video capabilities to be readily superimposed onto existing personal computers and workstations and their interconnecting LANs and WANs. In a particular preferred embodiment, a plurality of geographically dispersed multimedia LANs are interconnected by a WAN. The demands made on the WAN are significantly reduced by employing multi-hopping techniques, including dynamically avoiding the unnecessary decompression of data at intermediate hops, and exploiting video mosaicing, cut-and-paste and audio mixing technologies so that significantly fewer wide area transmission paths are required while maintaining the high quality of the transmitted audio/video.

Claims

1. A teleconferencing system for conducting a teleconference between first and second locations, the system comprising;

(a) a plurality of workstations at the first location and at least one workstation at the second location, each workstation being associated with at least a participant and having a monitor and participant audio and video capture and reproduction capabilities;

(b) an AV path for carrying AV signals representing participant video images and spoken audio among the workstations;

(c) a first video mosaic generator associated with the first location and configured to

(1) combine captured video images, of at least two first-location-participants, into a first mosaic image,

(2) receive a second-location-participant video image from the second location, and

(3) combine at least a portion of the first mosaic image with the received second-location-participant video image to form a first combined mosaic image for reproduction at a first location workstation, and

(d) a second video mosaic generator associated with the second location and configured to

(1) receive the first mosaic image from the first mosaic generator, and

(2) combine at least a portion of the first mosaic image with the second-location-participant video image to form a second combined mosaic image for reproduction at a workstation at the second location.

2. The teleconferencing system of claim 1, further comprising:

(a) a close-up selector responsive to a participant input to select one paricipant'"'"'s image in a reproduced combined mosaic image and to replace the reproduced combined mosaic image with the selected image.

3. The teleconferencing system of claim 1, further comprising:

(a) a plurality of video mosaic generators at one location each configured to combine the captured images of a plurality of participants into respective first and second mosaic images both for reproduction at the workstations of at least one participant.

4. The teleconferencing system of claim 1, further comprising:

(a) a participant display selector responsive to an input to select which participants are to have their corresponding captured video images displayed in a combined mosaic image.

5. The teleconferencing system of claim 1, further comprising:

(a) at least one codec configured to compress the first mosaic image and the second-location-participant image and wherein at least one mosaic generator can combine the compressed mosaic image and the compressed participant image into a combined mosaic image.

6. The teleconferencing system of claim 1, further comprising:

(a) an audio summer in communication with the AV path to receive captured audio of first, second and third participant and configured to combine the received audio of only the second and third participants into an audio sum for reproduction at the workstation of the first participant.

7. The teleconferencing system of claim 3, further comprising:

(a) an image synchronizer responsive to information relating to the first and second mosaic images to synchronize the mosaic images for joint reproduction at the workstation of at least one participant.

8. The teleconferencing system of claim 7, further comprising:

(a) a participant display selector configured to select which of the participants are to have their corresponding captured video image displayed in a mosaic image.

9. The teleconferencing system of claim 8, wherein the participant display selector selects the participants automatically.

10. The teleconferencing system of claim 4, wherein the participant display selector can select the participants automatically.

11. The teleconferencing system claim 4, wherein the system is operable to generate a combined video mosaic of fewer participants than the number of actual participants and the participant display selector is operable to select which of the actual participants will have a corresponding video image displayed.

12. The teleconferencing system of claim 6, further comprising:

(a) an echo canceler to reduce echo during the reproduction of the audio sum.

13. The teleconferencing system of claim 6, wherein the audio reproduction capabilities associated with at least the workstation of the first participant includes a plurality of speakers and further comprising:

(a) an audio control configured to control the audio sum at the first participant'"'"'s workstation such that the composition of the audio originating from the second and third participants, is reproduced at each speaker based on a position of the images of the second and third participants in a combined mosaic image.

14. A teleconferencing system for conducting a teleconference between first and second locations the system comprising:

(a) a plurality of workstations at the first location and at least one workstation at the second location, each workstation being associated with at least a participant and having a monitor and participant audio and video capture and reproduction capabilities;

(b) an AV path for carrying AV signals representing participant video images and spoken audio among the workstations;

(c) a first video mosaicing generator associated with the first location and configured to combine captured video images, of at least two first-location-participants, into a first mosaic image;

(d) a second video mosaic generator configured to receive the first mosaic image and combine at least a portion thereof with a participant video image captured at the second location to form a second mosaic image for reproduction at least one workstation;

(e) a data path providing a data path along which data can be shared among the workstations; and

(f) a data conference manager for managing a data conference, during which the shared data is displayed interactively on the workstation monitors of at least two participants.

15. A teleconferencing system for conducting a teleconference between first and second locations, the system comprising:

(a) a plurality of workstations at thc first location and at least one workstation at the second location, each workstation being associated with at least a participant and having a monitor and participant audio and video capture and reproduction capabilities;

(b) a first video mosaic generator associated with the first location and configured to combine captured video images, of at least two first-location-participants, into a first mosaic image;

(c) a second video mosaic generator configured to receive the first mosaic image and combine at least a portion thereof with a participant video image captured at the second location to form a second mosaic image for reproduction at a workstation;

(d) an AV path for carrying AV signals representing participant video images and spoken audio among the workstations; and

(e) an audio/video switch in association with the AV path and configured to control the carrying of AV signals representing individual participant and mosaic video images between the mosaic generator, the locations and the workstations.

16. A method for conducting a teleconference between first and second locations in which the first location includes a plurality of workstations and the second location includes at least one workstation, and in which each workstation is associated with at least one participant and has a monitor and participant audio and video capture an reproduction capabilities, the method comprising the steps of:

(a) moving AV signals representing participant video images and spoken audio among the workstations;

(b) creating a first combined mosaic image at the first location by

(1) combining captured video images, of at least two first-location-participants, into a first mosaic image,

(2) receiving a second-location-participant video image from the second location, and

(3) combining at least a portion of the first mosaic image with the received second-location-participant video image to form the first combined mosaic image;

(d) reproducing the first combined mosaic image at a first location workstation;

(e) creating a second combined mosaic image at the second location by

(1) receiving the first mosaic image from the first mosaicing circuitry, and

(2) combining at least a portion of the first mosaic image with the second-location-participant video image to form the second combined mosaic image; and

(f) reproducing the second combined mosaic image at a second location workstation.

17. The method of claim 16, further comprising the steps of:

(a) selecting one participant'"'"'s image reproduced in a combined mosaic image; and

(b) replacing the combined mosaic image with the selected image.

18. The method of claim 16, further comprising the steps of:

(a) combining the captured images of a plurality of participants into first and second mosaic images both for reproduction at the workstations of at least one participant.

19. The method of claim 16, further comprising the steps of:

(a) receiving the captured audio of a first, second and third participant;

(b) combining the received audio of only the second and third participants into an audio sum; and

(c) reproducing the audio sum at the workstation of the first participant.

20. The method of claim 17, further comprising the step of:

(a) selecting which of the participants are to have their corresponding captured video image displayed in the mosaic image.

21. The method of claim 18, further comprising the step of:

(a) combining the capture images of a plurality of participants into first and second mosaic images; and

(b) synchronizing the mosaic images such that a plurality of mosaic images can be reproduced at the workstations of at least one of the participants.

22. The method of claim 19, wherein the audio reproduction capabilities associated with at least the workstation of the first participant includes a plurality of speakers, the method further comprising the steps of:

(a) controlling the reproduction of the audio sum at the first participant'"'"'s workstation such that the composition of the audio, originating from the second and third participants, is reproduced at each speaker based on a position of the images of the second and third participants in the combined mosaic image.