Multimedia mail, conference recording and documents in video conferencing

• US 5,978,835 A
• Filed: 06/07/1996
• Issued: 11/02/1999
• Est. Priority Date: 10/01/1993
• Status: Expired due to Term

• Current Assignee: Pragmatus AV LLC (Pragmatus, LLC)
• Original Assignee: Collaboration Properties Incorporated
• Inventors: Lantz, Keith A., Lauwers, J. Chris, Burns, Emmett R., Burnett, Gerald J., Ludwig, Lester F.
• Primary Examiner(s): Dinh, Dung C.
• Application Number:
  US08/659,949
• Time in Patent Office: 1,243 Days
• Field of Search: 395/324, 395/330, 395/806, 395/807, 395/773, 395/331, 395/200.04, 395/200.34, 395/200.35, 395/200.36, 395/200.37, 379/201, 379/202, 379/204, 379/205, 379/206, 370/260, 370/261, 707/515, 707/530, 707/901, 707/902, 709/204, 709/205, 709/206, 709/207, 709/231, 709/248
• US Class Current: 709/204
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04L 12/1813 for computer conferences, e.g. chat rooms instant messaging H04L51/04; protocols for multimedia communication H04L65/1101; arrangements for multi-party communication H04L65/403; telephonic conference arrangements H04M3/56; television conference systems H04N7/15
• CPC Class Codes: H04L 12/1818 Conference organisation arrangements, e.g. handling schedules, setting up parameters needed by nodes to attend a conference, booking network resources, notifying involved parties
• CPC Class Codes: H04L 12/1822 Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 12/1831 Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status
• CPC Class Codes: H04L 12/1836 with heterogeneous network architecture
• CPC Class Codes: H04L 12/185 with management of multicast group membership
• CPC Class Codes: H04L 51/00 User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04M 2201/38 Displays
• CPC Class Codes: H04M 2203/5018 Initiating a conference during a two-party conversation, i.e. three-party-service or three-way-call computer conference organisation arrangements H04L12/1818
• CPC Class Codes: H04M 2203/5027 Dropping a party from a conference conducting a computer conference, e.g. admission detection H04L12/1822
• CPC Class Codes: H04M 3/42221 : Conversation recording systems at the subscriber's set H04M1/656
• CPC Class Codes: H04M 3/42323 : PBX's with CTI arrangements
• CPC Class Codes: H04M 3/4285 : Notifying, informing or entertaining a held party while on hold, e.g. Music On Hold
• CPC Class Codes: H04M 3/5307 : for recording messages comprising any combination of audio and non-audio components
• CPC Class Codes: H04M 3/562 : where the conference facilities are distributed
• CPC Class Codes: H04M 3/563 : User guidance or feature selection
• CPC Class Codes: H04M 3/564 : whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/567 : Multimedia conference systems
• CPC Class Codes: H04M 7/006 : Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42
• CPC Class Codes: H04M 7/12 : for working between exchanges having different types of switching equipment, e.g. power-driven and step by step or decimal and non-decimal
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/148 : Interfacing a video terminal to a particular transmission medium, e.g. ISDN
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04N 7/152 : Multipoint control units therefor
• CPC Class Codes: H04Q 2213/1305 : Software aspects
• CPC Class Codes: H04Q 2213/13093 : Personal computer, PC
• CPC Class Codes: H04Q 2213/13175 : Graphical user interface [GUI], WWW interface, visual indication
• CPC Class Codes: H04Q 2213/1319 : Amplifier, attenuation circuit, echo suppressor
• CPC Class Codes: H04Q 2213/1322 : PBX
• CPC Class Codes: H04Q 2213/1324 : Conference call
• CPC Class Codes: H04Q 2213/1326 : Consultation call, broker's call, call hold, toggling
• CPC Class Codes: H04Q 2213/1332 : Logic circuits
• CPC Class Codes: H04Q 2213/13337 : Picturephone, videotelephony
• CPC Class Codes: H04Q 2213/13389 : LAN, internet

Abstract

A multimedia collaboration system that integrates separate real-time and asynchronous networks--the former for real-time audio and video, and the latter for control signals and textual, graphical and other data--in a manner that is interoperable across different computer and network operating system platforms and which closely approximates the experience of face-to-face collaboration, while liberating the participants from the limitations of time and distance. These capabilities are achieved by exploiting a variety of hardware, software and networking technologies in a manner that preserves the quality and integrity of audio/video/data and other multimedia information, even after wide area transmission, and at a significantly reduced networking cost as compared to what would be required by presently known approaches. The system architecture is readily scalable to the largest enterprise network environments. It accommodates differing levels of collaborative capabilities available to individual users and permits high-quality audio and video capabilities to be readily superimposed onto existing personal computers and workstations and their interconnecting LANs and WANs. In a particular preferred embodiment, a plurality of geographically dispersed multimedia LANs are interconnected by a WAN. The demands made on the WAN are significantly reduced by employing multi-hopping techniques, including dynamically avoiding the unnecessary decompression of data at intermediate hops, and exploiting video mosaicing, cut-and-paste and audio mixing technologies so that significantly fewer wide area transmission paths are required while maintaining the high quality of the transmitted audio/video.

Claims

1. A method for conducting a teleconference among a plurality of participants, comprising the steps of:

(a) capturing audio and video generated, during a videoconference, at the workstation of a preparing participant;

(b) preparing and storing, as a multimedia mail message, the captured video and audio; and

(c) managing a data conference during which data is shared among a plurality of the participants having workstations including monitors configured to reproduce the shared data;

(d) synchronizing both the video image and spoken audio of the participants during the videoconference and the data shared during the data conference;

(e) recording the synchronized video, audio and data as a multimedia message; and

(f) forwarding the multimedia mail message for receipt, by a receiving participant, at any one of the venues selected from the group consisting of

(i) in real time at a location removed from the preparing participant;

(ii) at a different time at the same location as the message was prepared; and

(iii) at a different time at a location removed from the preparing participant, in such a manner that subsequent playback of the recorded message will represent the relationship in which the audio, video and data were originally represented.

2. The method of claim 1, further comprising the steps of

(a) defining a marked portion of a multimedia mail message;

and

(b) selectively displaying the marked portion when the multimedia mail message is reproduced.

3. The method of claim 2, further comprising the steps of:

(a) searching for a defined tag; and

(b) selectively displaying the portion of the message.

4. The method of claim 1, further comprising the steps of:

(a) capturing the data to be shared; and

(b) annotating the shared data during the data conference.

5. The method of claim 1, further comprising the step of storing a multimedia document such that the multimedia document can be retrieved by a participant.

6. A teleconferencing system comprising:

(a) at least one video signal source;

(b) at least one audio source;

(c) at least one video display device;

(d) at least one audio reproduction facility;

(e) at least one video communication path

(i) arranged for transport of audio and video signals,

(1) originating at one of the signal sources,

(2) to at least one of the display devices;

wherein the system is configured to

(ii) reproduce video images and audio,

(1) based on the transported video and audio signals,

(2) at one of the display devices,

(3) to define a video conference;

(f) at least one processor

(1) capable of providing data conferencing signals;

(g) at least one data communication path

(i) arranged for transmission of

(1) data conferencing visual information signals,

(ii) to one of the display devices;

wherein the system is further configured to

(iii) display visual information,

(1) based on the carried data conferencing signals,

(2) at one of the display devices

(3) to define a data conference;

(h) data annotation tools,

(i) that permit at least one associated participant

(1) to annotate the visual information displayed during the data conference; and

(f) at least one storage medium configured to

(i) store at least some of the data signals, annotations and related audio and/or video signals

(1) from the data and video conferences

(2) for subsequent retrieval, and

(3) synchronized play back

(4) in a manner representing the relationship in which they were created.

7. The teleconferencing system of claim 6, further comprising:

(a) a multimedia mail subsystem operable to receive and store multimedia mail messages under direction of at least one preparing participant.

8. The teleconferencing system of claim 7, wherein the multimedia mail is operable to process mail messages having at least one component selected from the group consisting of audio, video and data components.

9. The teleconferencing system of claim 6, wherein the AV capture tools, data capture tools and data annotation tools can be used to generate a multimedia document capable of being stored, in real time, on the storage medium.

10. The teleconferencing system of claim 9, further comprising:

(a) a multimedia mail system, in communication with the storage medium and configured to allow a sending participant to send the multimedia document to a receiving participant at a venue selected from the group consisting of;

(i) in real time at a location removed from the preparing participant;

(ii) at a different time at the same location as the document was prepared; and

(iii) at a different time at a location removed from the preparing participant.

11. The teleconferencing system of claim 10, wherein the multimedia mail system is in communication with a graphical animations originator device which can generate, save and replay animated graphical images that can be included in the multimedia mail document.

12. The teleconferencing system of claim 11, further comprising:

(a) a message marker that can be used to mark a portion of the multimedia document, wherein the marked portion can be selectively displayed by the receiving participant when the multimedia document is reproduced.

13. A teleconferencing system comprising:

(a) at least one video signal source;

(b) at least one audio source;

(c) at least one video display device;

(d) at least one audio reproduction facility;

(e) at least one video communication path

(i) arranged for transport of audio and video signals,

(1) originating at one of the signal sources,

(2) to at least one of the display devices;

wherein the system is configured to

(i) reproduce video images and audio,

(1) based on the transported video and audio signals,

(2) at one of the display devices,

(3) to define a video conference;

(f) at least one processor

(1) capable of providing data conferencing signals;

(g) at least one data communication path

(i) arranged for transmission of

(1) data conferencing visual information signals,

(ii) to one of the display devices;

wherein the system is further configured to

(i) display visual information,

(1) based on the carried data conferencing signals,

(2) at one of the display devices

(3) to define a data conference; and

(f) at least one storage medium configured to

(i) store at least some of the data signals plus related audio and/or video signals

(1) from the data and video conferences

(2) for subsequent retrieval, and

(3) synchronized play back

(4) in a manner representing the relationship in which they were created,

the system still further configured to

(i) prepare and store,

(1) as a multimedia mail message,

(2) the captured and annotated data, and

(ii) forward the multimedia mail message

(1) to a receiving participant,

(iii) such that the multimedia mail message can be received at a venue selected from the group consisting of;

(1) in real time at a location removed from the preparing participant;

(2) at a different time at the same location as the message was prepared; and

(3) at a different time at a location removed from the preparing participant.

14. The teleconferencing system of claim 13, further comprising

(a) a graphical animations originator device which can

(1) generate, save and replay animated graphical images

(2) to be included in the multimedia mail message.

15. The teleconferencing system of claim 13, further comprising:

(a) a message marker capable of defining a marked portion of the multimedia mail message, wherein the marked portion can be selectively displayed a the receiving participant when the multimedia mail message is reproduced.

16. The teleconferencing system of claim 15, further comprising a tag searcher configured to search for a defined tag, such that the portion of the message can be selectively displayed.

17. The teleconferencing system of claim 16, wherein the tag searcher can search a plurality of multimedia mail messages to locate the defined tag.

18. The teleconferencing system of claim 13, wherein the system is further configured to

(a) store audio and/or video signals

(i) which are either analog or digital signals.

19. A teleconferencing system comprising:

(a) at least one video signal source;

(b) at least one audio source;

(c) at least one video display device;

(d) at least one audio reproduction facility;

(e) at least one video communication path

(i) arranged for transport of audio and video signals,

(1) originating at one of the signal sources,

(2) to at least one of the display devices;

wherein the system is configured to

(i) reproduce video images and audio,

(1) based on the transported video and audio signals,

(2) at one of the display devices,

(3) to define a video conference;

(f) at least one processor

(1) capable of providing data conferencing signals;

(g) at least one data communication path

(i) arranged for transmission of

(1) data conferencing visual information signals,

(ii) to one of the display devices;

wherein the system is further configured to

(i) display visual information,

(1) based on the carried data conferencing signals,

(2) at one of the display devices

(3) to define a data conference;

(h) a multimedia conference recorder capable of

(i) synchronizing and recording

(1) both the video image and spoken audio of participants in the videoconference and

(2) the visual information displayed during the data conference; and

(f) at least one storage medium configured to

(i) store at least some of the data signals plus related audio and/or video signals

(1) from the data and video conferences

(2) for subsequent retrieval, and

(3) synchronized play back

(4) in a manner representing the relationship in which they were created.

20. The teleconferencing system of claim 19, wherein the recorded video images and audio includes video images and spoken audio directly associated with the participants and at least one other video image and spoken audio not directly associated with the preparing participants.

21. The teleconferencing system of claim 19, further comprising:

(a) annotation tools capable of annotating the shared data during a data conference, wherein the the system is further configured

(i) to synchronize the playback of the annotated data conference recordings with the videoconference recordings.

22. The teleconferencing system of claim 19, wherein the system is further configured to store and retrieve both the video image and spoken audio of the preparing participant and the recorded video image and spoken audio.

23. A method of making and storing a multimedia message, comprising the steps of:

(a) capturing

(i) images and/or audio

(ii) of a user

(iii) at an associated workstation;

(b) creating

(i) data signals,

(1) representing visual information

(2) other than the captured image

(ii) during the step of capturing;

(c) reproducing

(i) one or more of

(1) the visual information and

(2) captured image

(3) at the workstation; and

(d) storing

(i) the audio and/or video signals and

(ii) the associated visual information

(iii) in a synchronized manner

(iv) such that subsequent retrieval and playback

(1) of the associated visual information and the captured image and/or audio

(2) is in a manner representing the relationship in which they were created.