Systems and methods for playing, browsing and interacting with MPEG-4 coded audio-visual objects

• US 8,276,056 B1
• Filed: 09/04/2007
• Issued: 09/25/2012
• Est. Priority Date: 01/27/1998
• Status: Expired due to Term

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Haskell, Barin Geoffry, Puri, Atul, Schmidt, Robert Lewis
• Primary Examiner(s): Patel, Manglesh M
• Application Number:
  US11/849,399
• Time in Patent Office: 1,848 Days
• Field of Search: 715200-203
• US Class Current: 715/202
• CPC Class Codes: H04N 19/25 with scene description coding, e.g. binary format for scenes [BIFS] compression
• CPC Class Codes: H04N 19/90 using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

A number of novel configurations for MPEG-4 playback, browsing and user interaction are disclosed. MPEG-4 playback systems are not simple extensions of MPEG-2 playback systems, but, due to object based nature of MPEG-4, present new opportunities and challenges in synchronized management of independent coded objects as well as scene composition and presentation. Therefore, these configurations allow significantly new and enhanced multimedia services and systems. In addition, MPEG-4 aims for an advanced functionality, called Adaptive Audio Visual Session (AAVS) or MPEG-J. Adaptive Audio Visual Session (AAVS) (i.e., MPEG-AAVS, MPEG-Java or MPEG-J) requires, in addition to the definition of configurations, a definition of an application programming interface (API) and its organization into Java packages. Also disclosed are concepts leading to definition of such a framework.

Claims

1. A device that processes MPEG-4 data having a scene description graph and data related to at least one object, the device comprising:

means for interacting with a user configured for basic user interaction via control signals from at least one user input device;

a binary format of scene description graph interpreter connected to the means for interacting with the user for interpreting the MPEG-4 data to yield interpreted MPEG-4 data; and

a media decoder, compositor and renderer that receives the interpreted MPEG-4 data and presents at least one object on the means for interacting with the user,

wherein at least one of the binary format of scene description graph interpreter and the media decoder, compositor and renderer comprises at least one programmer interface accessible via the means for interacting with the user, and

wherein at least one of the means for interacting with the user and the binary format of scene description graph interpreter is further configured for local user interaction via at least one script obtained from a JavaScript interpreter.

2. The device of claim 1, wherein the means for interacting with the user comprises an audio visual object demultiplexer format of a scene browser.

3. The device of claim 1, wherein the MPEG-4 data having a scene description graph and data related to at least one object comprises MPEG-4 coded content, the MPEG-4 coded content including at least one of audio media, visual media and synthetic media.

4. The device of claim 1, wherein the binary format of scene description graph interpreter invokes at least one media decoder based on the scene description graph.

5. The device of claim 1, wherein the binary format of scene description graph interpreter further comprises:

at least one object node; and

at least one corresponding object programmer interface, each object programmer interface connected to a corresponding one of the at least one object node.

6. The device of claim 5, wherein the media decoders, compositor and renderer further comprises:

at least one decoder, each decoder connected to a corresponding one the at least one object programmer interface; and

a scene compositor connected to the at least one decoder.

7. The device of claim 6, wherein the scene compositor is connected to the audiovisual object demultiplexer and binary format of scene browser and the audiovisual object demultiplexer and binary format of scene browser is connected to the scene compositor.

8. The device of claim 1, wherein the binary format of scene description graph interpreter further comprises:

a VideoObject2D node or a MovieTexture node connected to the audiovisual object demultiplexer and binary format of scene browser;

a video object programmer interface connected to the VideoObject2D or MovieTexture node;

an AudioSource node connected to the means for interacting with the user;

an audio object programmer interface connected to the AudioSource node;

an ImageTexture node connected to the means for interacting with the user; and

an image object programmer interface connected to the ImageTexture node.

9. The device of claim 8, wherein the media decoders, compositor and renderer further comprise:

a video object decoder connected to the video object programmer interface;

an audio object decoder connected to the audio object programmer interface;

an image object decoder connected to the image object programmer interface; and

a scene compositor connected to each of the video object decoder, the audio object decoder, and the image object decoder.

10. The device of claim 9, wherein the scene compositor is connected to the means for interacting with the user and the means for interacting with the user is connected to the scene compositor.

11. The device of claim 8, wherein the binary format of scene description graph interpreter further comprises:

a proto node; and

a native proto programmer interface connected to the proto node.

12. The device of claim 11, wherein the media decoders, compositor and renderer further comprise:

a video object decoder connected to the video object programmer interface;

an audio object decoder connected to the audio object programmer interface;

an image object decoder connected to the image object programmer interface;

a native proto implementation connected to the native proto programmer interface; and

a scene compositor connected to each of the native proto implementation, the video object decoder, the audio object decoder, and the image object decoder.

13. The device of claim 12, wherein the scene compositor is connected to the means for interacting with the user and the means for interacting with the user is connected to the scene compositor.

14. The device of claim 8, wherein the binary format of scene description graph interpreter further comprises:

a script node;

an interpreter programmer interface connected to the script node; and

a scripting interface.

15. The device of claim 14, wherein the media decoders, compositor and renderer further comprise:

a video object decoder connected to the video object programmer interface;

an audio object decoder connected to the audio object programmer interface;

an image object decoder connected to the image object programmer interface;

a native proto implementation connected to the native proto programmer interface;

a JavaScript interpreter connected to the interpreter programmer interface;

a Java interpreter connected to the interpreter programmer interface; and

a scene compositor connected to each of the scripting interface, the native proto implementation, the video object decoder, the audio object decoder, and the image object decoder.

16. The device of claim 15, wherein the scene compositor is connected to the means for interacting with the user and the means for interacting with the user is connected to the scene compositor.

17. The device of claim 15, wherein the JavaScript interpreter and the Java interpreter are connected to the scripting interface.

18. The device of claim 17, wherein the multimedia browser supports programmatic behavior and interaction via the JavaScript interpreter and the Java interpreter to modify the scene internally.

19. The device of claim 14, wherein the scripting interface is connected to the means for interacting with the user.

20. The device of claim 1, wherein at least one control signal from at least one user input device is provided to the audiovisual object demultiplexer and binary format of scene browser and to the binary format of scene description graph interpreter.

21. The device of claim 1, further comprising an adaptive audio visual session connected to the means for interacting with the user.

22. The device of claim 21, wherein at least one control signal from at least one user input device is provided to the adaptive audio visual session.

23. The device of claim 21, wherein the adaptive audio visual session comprises:

an adaptive audio visual session external interface;

a browser specific binding connected to the adaptive audio visual session external interface; and

a browser programmer interface connected to the browser specific binding.

24. The device of claim 23, wherein the means for interacting with the user is connected to the browser programmer interface.

25. The device of claim 23, wherein at least one control signal is provided to the adaptive audio visual session external interface.

26. The device of claim 25, wherein the at least one control signal is an adaptive audio visual session external script or applet.

27. The device of claim 1, wherein the multimedia browser supports programmatic behavior and interaction via at least one of Java and JavaScript to modify the scene internally.

28. The device of claim 1, wherein the multimedia browser supports external interface for BIFS player control in response to changing resources and support of user interaction.

29. A method of providing an audio-visual object browser that operates on MPEG-4 data having a scene description graph and data related to at least one object, the method comprising:

providing an audiovisual object demultiplexer and binary format of scene browser for basic user interaction with a user via control signals from at least one user input device;

interpreting a binary format of scene description graph associated with the MPEG-4 data from the audiovisual object demultiplexer and binary format of scene browser;

decoding, composing and rendering the interpreted binary format of scene description graph on the audiovisual object demultiplexer and binary format of scene browser;

providing at least one programmer interface accessible via the audiovisual object demultiplexer and binary format of scene browser for adjusting at least one of the interpreting and the decoding, composing and rendering; and

configuring at least one of the audiovisual object demultiplexer and binary format of scene browser and the binary format of scene description graph interpreter for local user interaction via at least one script obtained from a JavaScript interpreter.

30. A method of rendering MPEG-4 data having a scene description graph and data related to at least one object, the method comprising:

interacting with a user through an audiovisual object demultiplexer and binary format of scene browser via a basic user interaction using control signals from at least one input device;

interpreting a binary format of scene description graph associated with the MPEG-4 data from the audiovisual object demultiplexer and binary format of scene browser;

decoding, composing and rendering the interpreted binary format of scene description graph on the audiovisual object demultiplexer and binary format of scene browser;

providing at least one programmer interface accessible via the audiovisual object demultiplexer and binary format of scene browser for adjusting at least one of the interpreting and the decoding, composing and rendering; and

interacting with at least one of the audiovisual object demultiplexer and binary format of scene browser and the binary format of scene description graph interpreter via a local user interaction using at least one script obtained from a JavaScript interpreter.