Systems and methods for playing, browsing and interacting with MPEG-4 coded audio-visual objects

• US 7,281,200 B2
• Filed: 09/15/2003
• Issued: 10/09/2007
• Est. Priority Date: 01/27/1998
• Status: Expired due to Fees

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Haskell, Barin Geoffry, Puri, Atul, Schmidt, Robert Lewis
• Primary Examiner(s): HUYNH, CONG LAC T
• Application Number:
  US10/662,831
• Publication Number:
  US 20040054965A1
• Time in Patent Office: 1,485 Days
• Field of Search: 715/500.1, 715/716, 715/727
• US Class Current: 715/201
• CPC Class Codes: H04N 19/25 with scene description coding, e.g. binary format for scenes [BIFS] compression
• CPC Class Codes: H04N 19/90 using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

A number of novel configurations for MPEG-4 playback, browsing and user interaction are disclosed. MPEG-4 playback systems are not simple extensions of MPEG-2 playback systems, but, due to object based nature of MPEG-4, present new opportunities and challenges in synchronized management of independent coded objects as well as scene composition and presentation. Therefore, these configurations allow significantly new and enhanced multimedia services and systems. In addition, MPEG-4 aims for an advanced functionality, called Adaptive Audio Visual Session (AAVS) or MPEG-J. Adaptive Audio Visual Session (AAVS) (i.e., MPEG-AAVS, MPEG-Java or MPEG-J) requires, in addition to the definition of configurations, a definition of an application programming interface (API) and its organization into Java packages. Also disclosed are concepts leading to definition of such a framework.

Claims

1. An audio-visual object browser system that inputs MPEG-4 data having a scene description graph and data related to at least one object, comprising:

an audiovisual object demultiplexer and binary format of scene browser for interacting with a user;

a binary format of scene (BIFS) description graph interpreter connected to the audiovisual object demultiplexer and binary format of scene browser for interpreting the MPEG-4 data, and

a media decoder, compositor and renderer that receives the interpreted MPEG-4 data and presents at least one object on the audiovisual object demultiplexer and binary format of scene browser.

2. The audio-visual browser system of claim 1, wherein at least one control signal from at least one user input device is provided to the audiovisual object demultiplexer and binary format of scene browser.

3. The audio-visual browser system of claim 1, wherein the MPEG-4 data having a scene description graph and data related to at least one object comprises MPEG-4 coded content, the MPEG-4 coded content including at least one of audio media visual media.

4. The audio-visual browser system of claim 1, wherein the binary format of scene (BIFS) scene description graph interpreter invokes at least one media decoder based on the scene description graph.

5. The audio-visual browser system of claim 1, wherein the binary format of scene (BIFS) scene description graph interpreter comprises:

at least one object node; and

at least one corresponding object programmer interface, each object programmer interface connected to a corresponding one of the at least one object mode.

6. The audio-visual browser system of claim 5, wherein the media decoders, compositor and renderer comprises:

at least one decoder, each decoder connected to a corresponding one the at least one object programmer interface; and

a scene compositor connected to the at least one decoder.

7. The audio-visual browser system of claim 6, wherein the scene compositor is connected to the audiovisual object demultiplexer and binary format of scene browser and the audiovisual object demultiplexer and binary format of scene browser is connected to the scene compositor.

8. The audio-visual browser system of claim 1, wherein the binary format of scene (BIFS) scene description graph interpreter comprises:

a VideoObject2D node or a MovieTexture node connected to the audiovisual object demultiplexer and binary format of scene browser;

a video object programmer interface connected to the VidioObject2D or MovieTexture node;

an AudioSource node connected to the audiovisual object demultiplexer and binary format of scene browser;

an audio object programmer interface connected to the Audio Source node;

an ImageTexture node connected to the audiovisual object demultiplexer and binary format of scene browser; and

an image object programmer interface connected to the ImageTexture node.

9. The audio-visual browser system of claim 8, wherein the media decoders, compositor and renderer comprises:

a video object decoder connected to the video object programmer interface;

an audio object decoder connected to the audio object programmer interface;

an image object decoder connected to the image object programmer interface; and

a scene compositor connected to each of the video object decoder, the audio object decoder, and the image object decoder.

10. The audio-visual browser system of claim 9, wherein the scene compositor is connected to the audiovisual object demultiplexer and binary format of scene browser and the audio visual object demultiplexer and binary format of scene browser is connected to the scene compositor.

11. The audio-visual browser system of claim 8, wherein the binary format of scene (BIFS) scene description graph interpreter further comprises:

a proto note; and

a native proto programmer interface connected to the proto node.

12. The audio-visual browser system of claim 11, wherein the media decoders, compositor and renderer comprises:

a video object decoder connected to the video object programmer interface;

an audio object decoder connected to the audio object programmer interface;

an image object decoder connected to the image object programmer interface;

a native proto implementation connected to the native proto programmer interface; and

a scene compositor connected to each of the native proto implementation, the video object decoder, the audio object decoder, and the image object decoder.

13. The audio-visual browser system of claim 12, wherein the scene compositor is connected to the audiovisual object demultiplexer and binary format of scene browser and the audiovisual object demultiplexer and binary format of scene browser is connected to the scene compositor.

14. The audio-visual browser system of claim 8, wherein the binary format of scene (BIFS) scene description graph interpreter further comprises:

a script node;

an interpreter programmer interface connected to the script node; and

a scripting interface.

15. The audio-visual browser system of claim 14, wherein the media decoders, compositor and renderer comprises:

a video object decoder connected to the video object programmer interface;

an audio object decoder connected to the audio object programmer interface;

an image object decoder connected to the image object programmer interface;

a native proto implementation connected to the native proto programmer interface;

a JavaScript interpreter connected to the interpreter programmer interface;

a scene compositor connected to each of the native proto implementation, the video object decoder, the audio object decoder, and the image object decoder.

16. The audio-visual browser system of claim 15, wherein the scene compositor is connected to the audiovisual object demultiplexer and binary format of scene browser and the audiovisual object demultiplexer and binary format of scene browser is connected to the scene compositor.

17. The audio-visual browser system of claim 15, wherein the JavaScript interpreter and the Java interpreter are connected to the scripting interface.

18. The audio-visual browser system of claim 17, wherein the multimedia browser supports programmatic behavior and interaction via the JavaScript interpreter and the Java interpreter to modify the scene internally.

19. The audio-visual browser system of claim 14, wherein the scripting interface is connected to the audiovisual object demultiplexer and binary format of scene browser.

20. The audio-visual browser system of claim 1, wherein at least one control signal from at least one user input device is provided to the audiovisual object demultiplexer and binary format of scene browser and to the binary format of scene (BIFS) scene description graph interpreter.

21. The audio-visual browser system of claim 1, further comprising an adaptive audio visual session connected to the audiovisual object demultiplexer and binary format of scene browser.

22. The audio-visual browser system of claim 21, wherein at least one control signal from at least one user input device is provided to the adaptive audio visual session.

23. The audio-visual browser system of claim 21, wherein the adaptive audio visual session comprises:

and adaptive audio visual session external interface;

a browser specific binding connected to the adaptive audio visual session external interface; and

a browser programmer interface connected to the browser specific binding.

24. The audio-visual browser system of claim 23, wherein the audiovisual object demultiplexer and binary format of scene browser is connected to the browser programmer interface.

25. The audio-visual browser system of claim 23, wherein at least one control signal is provided to the adaptive audio visual session external interface.

26. The audio-visual browser system of claim 25, wherein at least one control signal is an adaptive audio visual session external script or applet.

27. The audio-visual browser system of claim 1, wherein the multimedia browser supports programmatic behavior and interaction via at least one of Java and JavaScript to modify the scene internally.

28. The audio-visual browser system of claim 1, wherein the multimedia browser supports external interface for BIFS player control in response to changing resources and support of user interaction.

29. A computer-readable medium having instructions that control a computing device to provide an audio-visual object browser that operates on MPEG-4 data having a scene description graph and data related to at least one object, the instructions comprising:

providing an audiovisual object demultiplexer and binary format of scene browser for interacting with a user;

interpreting a binary format of scene (BIFS) description graph associated with the MPEG-4 data from the audiovisual object demultiplexer and binary format of scene browser; and

decoding, composing and rendering the interpreted BIFS scene description graph on the audiovisual object demultiplexer and binary format of scene browser.