Audio/video storage and retrieval for multimedia workstations by interleaving audio and video data in data file

• US 5,623,690 A
• Filed: 07/16/1992
• Issued: 04/22/1997
• Est. Priority Date: 06/03/1992
• Status: Expired due to Term

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Digital Equipment Corporation (HP Inc.)
• Inventors: Palmer, Larry G., Palmer, Ricky S.
• Primary Examiner(s): Herndon, Heather R.
• Assistant Examiner(s): Fetting, Anton W.
• Application Number:
  US07/915,088
• Time in Patent Office: 1,741 Days
• Field of Search: 395/153, 395/154, 358/402, 358/343, 348/515, 348/423, 434/308, 434/309
• US Class Current: 715/202
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4363 Adapting the video stream to a specific local network, e.g. a Bluetooth® network
• CPC Class Codes: H04N 21/4381 Recovering the multiplex stream from a specific network, e.g. recovering MPEG packets from ATM cells transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

An apparatus and method for storing and retrieving synchronized audio/video "filmclips" to and from a data file of a multimedia computer workstation includes a storage means for a workstation to store audio and video data as digital data packets to the data file, and retrieval means for the workstation to retrieve audio and video data from the data file. The video data is presented as an image on the display of the workstation, while the audio data is sent to either amplified speakers or headphones. An audio data stream is stored to the data file such that the audio data can be retrieved from the data file and reconstructed into a continuous audio signal. The video data is stored to the data file such that each frame of video data is inserted into the stored audio data stream without affecting the continuity of the audio signal reconstructed by the workstation. Timing information is attached to each frame of video data stored to the file, and indicates a point in the continuous audio data stream which corresponds in time to the frame of video data. A synchronizer displays a frame of video data when the point in the audio data stream, corresponding to the timing information of the retrieved video frame is audibly reproduced by the workstation.

Claims

1. A method for storing and retrieving an audio data stream and corresponding frames of video data on a multimedia workstation, comprising the steps of

storing the audio data stream in blocks from the workstation to a data file accessible by the workstation such that the audio data stream can be retrieved from the data file and reconstructed by the workstation into a continuous audio signal;

storing the frames of video data from the workstation to the data file by inserting each of the frames of video data between the blocks of the audio data stream to position the frames of the video data with the blocks of the audio data stream;

attaching timing information to each of the frames of video data stored to the data file indicating a point in the audio data stream, also stored to the data file, corresponding in time to the frames of video data;

retrieving and audibly reproducing the audio data stream from the data file;

retrieving the frames of video data from the data file; and

displaying the frames of video data in response to audibly reproducing a portion of the audio data stream, retrieved from the data file, that corresponds to the timing information of the frames of video data.

2. The method of claim 1 further comprising the steps of

a) the workstation internally generating timing information by determining an amount of audio data in the audio stream retrieved from the data file;

and

b) the workstation comparing the internally generated timing information with the timing information stored in the data file corresponding to a most recently retrieved one of the frames of data from the data file to determine when to display the most recently retrieved frame of video data on the display of the workstation.

3. The method of claim 1 wherein the blocks of audio data stream are audio packets including an audio header indicating a length of the audio packets.

4. The method of claim 1 wherein the frames of the video data include a video header indicating a size of the frames of video data.

5. The method of claim 4 wherein the video header includes a video height field and a video width field indicating a height and width of the frames of video data.

6. A multimedia workstation, comprising:

a storage application for enabling the workstation to store audio and video data, representing an audio data stream and frames of video data, as digital data packets, the packets including header information indicating lengths of the packets, wherein the storage application attaches timing information to each one of the frames of the video data stored to the data file indicating a point in the audio data stream also stored to the data file which corresponds in time to the frames of video data, the storage application including;

an audio storage routine for storing the audio data stream from the workstation to the data file such that the audio data stream can be retrieved from the data file and reconstructed by the workstation into a continuous audio signal, and

a video storage routine for storing the video data from the workstation to the data file such that each one of the frames of the video data stored to the data file is inserted into the audio data stream also stored in the data file by the workstation without affecting the continuity of the audio signal stream retrieved from the data file and reconstructed by the workstation; and

a retrieval application for enabling the workstation to retrieve audio and video data from the data file which was stored as digital data packets by the storage application, the retrieval application retrieving and audibly reproducing the audio data stream from the data file, retrieving the frames of video data from the data file, and video data in response to audibly reproducing a portion of the audio data stream, retrieved from the data file, that corresponds to the timing information of the frames of video data.

7. The multimedia workstation of claim 6, wherein

a) the storage application executes on a processor of the workstation;

and

b) the retrieval application executes on a processor of the workstation;

wherein the storage application formats and stores audio and video data to the data file as the data packets, and the retrieval application retrieves the data packets from the data file and reconstructs the audio and video data stored by the storage application for audible and visual reproduction, respectively.

8. The multimedia workstation of claim 6 wherein

a) the retrieval application internally generates timing information by counting an amount of audio data in the audio data stream retrieved from the data file and compares the internally generated timing information with the timing information stored in the data file corresponding a most recently retrieved one of the frames of video data from the data file to determine when to display the most recently retrieved frame of video data on the display of the workstation.

9. The multimedia workstation of claim 6 wherein the data file is accessed using a TCP/IP protocol data socket.

10. The workstation of claim 6 wherein the header information includes an audio header indicating a length of audio packets.

11. The workstation of claim 6 wherein the header information includes a video header indicating a size of frames of the video data.

12. The workstation of claim 11 wherein the header information includes a video height field and a video width field indicating a height and width of the frames of the video data.

13. The workstation of claim 6 wherein the audio data is stored in blocks and the video data is stored as frames in the data file by inserting each of the frames of the video data between the blocks of the audio data to interleave the frames of the video data with the blocks of the audio data in the data file.

14. A multimedia computer workstation, comprising:

a video source for providing sequential frames of digitized video data;

an audio source for providing a digitized audio data stream that represents a continuous audio signal synchronized to the sequential frames of digitized video data;

a data storage device for storing blocks of the audio data stream and the frames of video data to a data file such that each one of the frames of video data stored to the data file is sequentially inserted into the audio data stream to interleave the frames of the video data with the blocks of the audio data stream, each one of the frames of video data comprising timing information indicating a point in the audio data stream also stored to the data file which corresponds in time to the frames of video data;

a retrieval application for retrieving the audio data stream and the sequential frames of video data from the data file, and for synchronizing the frames of video data to the audio data stream in response to the timing information by displaying the frames of video data in response to audibly reproducing a portion of the audio data stream that corresponds to the timing information of the frames of video data;

a device for audibly reproducing the retrieved audio data stream to generate a continuous audio signal; and

a display for displaying the retrieved sequential frames of video data synchronized to the reproduced continuous audio signal.

15. The multimedia workstation of claim 14, wherein the video source comprises a video camera providing sequential frames of live video images, and a frame grabber for capturing, digitizing, and storing each sequential frame of the live video images.

16. The multimedia workstation of claim 14, wherein the video source provides pre-recorded sequential frames of analog video, and comprises a frame grabber for capturing, digitizing, and storing each sequential frame of pre-recorded analog video.

17. The multimedia workstation of claim 16, wherein the video source comprises a video tape recorder.

18. The multimedia workstation of claim 16, wherein the video source comprises a video laser disk player.

19. The multimedia workstation of claim 14 wherein the audio source comprises a microphone for transducing live audio into an analog audio signal, and an audio controller for digitizing and storing the analog audio signal.

20. The multimedia workstation of claim 19 wherein the audio controller applies mu-law compression in digitizing the analog audio signal.

21. The multimedia workstation of claim 14 wherein the audio source comprises means for providing pre-recorded audio signals corresponding to sequential frames of pre-recorded video.

22. The multimedia workstation of claim 14 wherein the data file is accessed using a TCP/IP protocol data socket.

23. The multimedia workstation of claim 14 wherein the data storage device comprises video compression means for reducing the quantity of data required to be stored to the data file to represent each frame of video data.

24. The multimedia workstation of claim 23 wherein the video compression means applies JPEG compression to the video data.

25. The multimedia workstation of claim 23 wherein the video compression means applies MPEG compression to the video data.

26. The multimedia workstation of claim 14 wherein the workstation comprises an IBM compatible personal computer.

27. The multimedia workstation of claim 14 wherein the workstation comprises a RISC processor.

28. A multimedia message storage apparatus for a video teleconferencing workstation, comprising

a receiver application program, executing on a processor of the workstation, for enabling the workstation to receive an audio and video data message from a remote source across a digital data network as digital data packets and store the received audio and video data packets to a data file accessible by the workstation, the audio and video data packets being formatted by the remote source so that an audio data stream and frames of video data from the remote source are stored to the data file such that the audio data stream can be retrieved from the data file and reconstructed by the workstation into a continuous audio signal and video data from the remote source is stored to the data file such that each frame of the video data is inserted into the audio data stream also stored in the data file without affecting the continuity of the audio signal stream retrieved from the data file and reconstructed by the workstation, the audio and video data including timing information attached to each frame of video data stored to the data file to indicate a point in the audio data stream also stored to the data file which corresponds in time to the frame of video data;

and

a retrieval application program, executing on the processor, for enabling the workstation to retrieve the audio and video data message from the data file which was stored as digital data packets by the receiver application program, the retrieval application retrieving and audibly reproducing the audio data stream from the data file, retrieving the frames of video data from the data file, displaying the frames of video data in response to audibly reproducing a portion of the audio data stream, retrieved from the data file, that corresponds to the timing information of the frames of video data.

29. The multimedia message apparatus of claim 28, further comprising

means for informing a user of the workstation that an audio and video data message has been received by the workstation.

30. The multimedia message apparatus of claim 29, wherein

the means for informing comprises a message displayed on the workstation display.

31. The multimedia message apparatus of claim 28, further comprising

screening means for monitoring the received audio and video data message as it is being received from the remote source.

32. The multimedia message apparatus of claim 28,

wherein the receiver application program receives the audio and video message data packets formatted by the remote source and stores the data packets to the data file as received from the remote source, and the retrieval application program retrieves the data packets from the data file and reconstructs the audio and video data stored by the receiver application program for audible and visual reproduction, respectively.

33. The multimedia message apparatus of claim 28 wherein

a) the retrieval application program generates timing information by counting the amount of audio data in the audio data stream retrieved from the data file;

and

b) the retrieval application program compares the internally generated timing information with the timing information stored in the data file corresponding to the frame of video data most recently retrieved from the data file to determine when to display the most recently retrieved frame of video data on the display of the workstation.

34. The multimedia message apparatus of claim 28 wherein the data file is accessed using a TCP/IP protocol data socket.

35. The apparatus of claim 28 wherein the data packets include header information indicating lengths of the packets.

36. The apparatus of claim 35 wherein the header information includes a video header indicating a size of frames of the video data.

37. The apparatus of claim 36 wherein the header information includes a video height field and a video width field indicating a height and width of the frames of the video data.

38. The apparatus of claim 28 wherein the audio data is stored in blocks and the video data is stored as frames in the data file by inserting each of the frames of the video data between the blocks of the audio data to interleave the frames of the video data with the blocks of the audio data in the data file.

39. A method for storing and retrieving an audio data stream and corresponding frames of video data on a multimedia workstation, comprising the steps of

storing the audio data stream in blocks from the workstation to a data file accessible by the workstation such that the audio data stream can be retrieved from the data file and reconstructed by the workstation into a continuous audio signal;

storing the frames of video data from the workstation to the data file by inserting each of the frames of video data between the blocks of the audio data stream to position the frames of the video data with the blocks of the audio data stream also stored in the data file;

attaching timing information to each of the frames of video data stored to the data file indicating a point in the audio data stream, also stored to the data file, corresponding in time to the frames of video data;

retrieving and audibly reproducing the audio data stream from the data file;

retrieving the frames of video data from the data file; and

displaying the frames of video data in response to audibly reproducing a portion of the audio data stream, retrieved from the data file, that corresponds to the timing information of the frames of video data.