Systems and methods for playing, browsing and interacting with MPEG-4 coded audio-visual objects

• US 6,654,931 B1
• Filed: 01/25/1999
• Issued: 11/25/2003
• Est. Priority Date: 01/27/1998
• Status: Expired due to Term

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Puri, Atul, Schmidt, Robert Lewis, Haskell, Barin Geoffry
• Primary Examiner(s): Hong, Stephen S.
• Application Number:
  US09/236,462
• Time in Patent Office: 1,765 Days
• Field of Search: 715/500.1, 709/217, 348/553, 345/473
• US Class Current: 715/202
• CPC Class Codes: H04N 19/25 with scene description coding, e.g. binary format for scenes [BIFS] compression
• CPC Class Codes: H04N 19/90 using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

A number of novel configurations for MPEG-4 playback, browsing and user interaction are disclosed. MPEG-4 playback systems are not simple extensions of MPEG-2 playback systems, but, due to object based nature of MPEG-4, present new opportunities and challenges in synchronized management of independent coded objects as well as scene composition and presentation. Therefore, these configurations allow significantly new and enhanced multimedia services and systems. In addition, MPEG-4 aims for an advanced functionality, called Adaptive Audio Visual Session (AAVS) or MPEG-J. Adaptive Audio Visual Session (AAVS) (i.e., MPEG-AAVS, MPEG-Java or MPEG-J) requires, in addition to the definition of configurations, a definition of an application programming interface (API) and its organization into Java packages. Also disclosed are concepts leading to definition of such a framework.

Claims

1. An audio-visual object playback system, comprising:

a delivery media integration framework and flexible demultiplexing layer;

an access layer connected to the delivery media integration framework and flexible demultiplexing layer;

an MPEG-4 media decoder structure connected to the access layer and having at least one media decoder;

a binary format of scene decoder connected to the access layer and the MPEG-4 media decoder structure;

a compositor connected to the MPEG-4 media decoder structure; and

a display connected to the compositor.

2. The audio-visual object playback system of claim 1, wherein the MPEG-4 media decoder structure comprises at least one of:

a video object decoder;

a facial/object animation object decoder;

an image texture object decoder;

an audio object decoder; and

a structured audio object decoder.

3. The audio-visual object playback system of claim 1, wherein the binary format of scene decoder is further connected to the compositor.

4. The audio-visual object playback system of claim 1, wherein at least one control signal from at least one user input device is provided to the compositor.

5. The audio-visual object playback system of claim 1, wherein:

the delivery media integration framework and flexible demultiplexing layer receives MPEG-4 coded content, the MPEG-4 coded content including at least one of audio media, visual media and synthetic media; and

the delivery media integration framework and flexible demultiplexing layer outputs to the access layer at least one flexmux protocol data unit extracted from the MPEG-4 coded content.

6. The audio-visual object playback system of claim 5, wherein the access layer outputs at least one unformatted access layer protocol data unit extracted from the at least one flexmux protocol data unit.

7. The audio-visual object playback system of claim 6, wherein unformatted access layer protocol data units extracted from the at least one flexmux protocol data unit that correspond to coded video streams are output to a video object decoder of the MPEG-4 media decoder structure.

8. The audio-visual object playback system user of claim 6, wherein unformatted access layer protocol data units extracted from the at least one flexmux protocol data unit that correspond to coded facial and/or animation streams are output to a facial/object animation object decoder of the MPEG-4 media decoder structure.

9. The audio-visual object playback system of claim 6, wherein unformatted access layer protocol data units extracted from the at least one flexmux protocol data unit that correspond to coded audio streams are output to an audio object decoder of the MPEG-4 media decoder structure.

10. The audio-visual object playback system of claim 6, wherein unformatted access layer protocol data units extracted from the at least one flexmux protocol data unit that correspond to coded speech streams are output to a structured audio object decoder of the MPEG-4 media decoder structure.

11. The audio-visual object playback system of claim 6, wherein unformatted access layer protocol data units extracted from the at least one flexmux protocol data unit that correspond to scene description representations are output to the binary format of scene decoder.

12. The audio-visual object playback system of claim 1, wherein the audio-visual object playback system is provided as a plug-in to a document browser.

13. The audio-visual object playback system of claim 1, wherein the audio-visual object playback system is at least partially integrated with a document browser.

14. The audio-visual object playback system of claim 13, wherein the audio-visual object playback system is fully integrated with the document browser.

15. The audio-visual object playback system of claim 1, wherein the audio-visual object playback system is implemented as at least one native method.

16. The audio-visual object playback system of claim 15, wherein the at least one native method can be called by a Java method.

17. The audio-visual object playback system of claim 15, wherein the at least one native method can call a Java method.

18. A method of generating audio-visual content, the method comprising:

extracting at least one flexmux protocol data unit from received MPEG-4 coded data at a delivery media integration framework and flexible demultiplexing layer, the MPEG-4 coded data comprising at least one of audio media, visual media and synthetic media;

extracting at least one unformatted access layer protocol data united from a received extracted flexmux protocol data unit at an access layer;

transmitting the extracted unformatted access layer protocol data units to a decoder;

transmitting unformatted access layer protocol data units corresponding to scene description representation, if any, to a binary format of scene (BIFS) decoder; and

transmitting output from the decoder and output from the BIFS decoder to a compositor and renderer, wherein the compositor and renderer output a scene for display to a user.

19. The method of claim 18, wherein the decoder comprises an MPEG-4 media decoder.

20. The method of claim 19, wherein the MPEG-4 media decoder comprises one or more of a video decoder, a facial/object animation decoder, an image texture decoder, an audio decoder, and a structure audio decoder.