Coding/decoding apparatus, coding/decoding system and multiplexed bit stream

• US 6,584,125 B1
• Filed: 12/22/1998
• Issued: 06/24/2003
• Est. Priority Date: 12/22/1997
• Status: Expired due to Term

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Katto, Jiro
• Primary Examiner(s): Kizou, Hassan
• Assistant Examiner(s): LOGSDON, JOSEPH B
• Application Number:
  US09/218,141
• Time in Patent Office: 1,645 Days
• Field of Search: 370/465, 370/498, 370/503, 370/529, 370/535, 370/536, 370/537, 370/542, 348/500, 348/512, 382/232
• US Class Current: 370/537
• CPC Class Codes: H04N 19/27 involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 7/52 Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal assembling of a multiplex stream by combining a video stream with other content or additional data, remultiplexing of multiplex streams, insertion of stuffing bits into the multiplex stream, assembling of a packetised elementary stream at server side H04N21/236; disassembling of a multiplex stream, remultiplexing of multiplex streams, extraction or processing of Service Information, disassembling of packetised elementary stream at client side H04N21/434

Abstract

A coding apparatus of the present invention comprises coding circuit 1 for audio signals, coding circuit 2 for video signals, interface circuit 3 on input of scene data, coding circuit 4 for scene data, composition circuit 5, multiplexing circuit 6, display circuit 7 and clock generating circuit 8. Each of coding circuits 1, 2 and 4 outputs time information representing a decoding timing, and composition circuit 5 outputs time information representing a composition timing. Multiplexing circuit 6 multiplexes time information together with the compressed data given from each of coding circuits 1, 2 and 4, thereby generating a bit stream.

Claims

1. A coding apparatus comprising:

audio signal coding means for coding an audio signal to output compressed audio data, local-decoding the compressed audio data to output local-decoded audio data and outputting time information which represent the decoding timing of the compressed audio data;

video signal coding means for coding a video signal to output compressed video data, local-decoding the compressed video data to output local-decoded video data and outputting time information which represent the decoding timing of the compressed video data;

interface means for accepting information on a composite scene;

scene data coding means for coding scene data supplied from said interface means to output compressed scene data, local-decoding the compressed scene data to output local-decoded scene data and outputting time information which represent the decoding timing of the compressed scene data;

composition means for composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting the time information which represent the composite timing of the composed scene;

means for reproducing/displaying the composed scene supplied from said composition means;

clock supply means for supplying at least one clock to said audio signal coding means, said video signal coding means, said scene data coding means and said composition means; and

multiplexing means for creating a bit stream on the basis of the time information and compressed audio data supplied from said audio signal coding means, the time information and compressed video data supplied from said video signal coding means, the time information and compressed scene data supplied from said scene data coding means, the time information supplied from said composition means, and at least one clock value of at least one clock supplied from said clock supply means.

2. The coding apparatus as claimed in claim 1, further comprising means for detecting the delay status of a scene composite processing on said composition means and controlling the operation of said video signal coding means to pause a coding processing of a frame which has been late for the scene composite processing.

3. The coding apparatus as claimed in claim 1, further comprising means for detecting the delay status of each coding processing on said audio signal coding means, said video signal coding means and said scene data coding means, and controlling the operation of said video signal coding means to pause a scene composite processing of audio signal, video signal or scene data on a frame which has been late for a scene composite processing.

4. A coding apparatus comprising:

audio signal coding means for coding an audio signal to output compressed audio data, local-decoding the compressed audio data to output local-decoded audio data and outputting time information which represent the decoding timing of the compressed audio data;

video signal coding means for coding a video signal to output compressed video data, local-decoding the compressed video data to output local-decoded video data and outputting time information which represent the decoding timing of the compressed video data;

interface means for accepting information on a composite scene;

scene data coding means for coding scene data supplied from said interface means to output compressed scene data, local-decoding the compressed scene data to output local-decoded scene data and outputting time information which represent the decoding timing of the compressed scene data;

composition means for composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting the time information which represent the composite timing of the composed scene;

means for reproducing/displaying the composed scene supplied from said composition means;

clock supply means for supplying at least one clock to said audio signal coding means, said video signal coding means, said scene data coding means and said composition means; and

multiplexing means for creating a bit stream on the basis of the time information and compressed audio data supplied from said audio signal coding means, the time information and compressed video data supplied from said video signal coding means, the time information and compressed scene data supplied from said scene data coding means, the time information supplied from said composition means, and at least one clock value of at least one clock supplied from said clock supply means;

wherein said clock supply means includes first clock supply means for supplying first clock to said audio signal coding means, second clock supply means for supplying second clock to said video signal coding means and third clock supply means for supplying third clock to said scene data coding means and composition means, and said multiplexing means multiplexes clock values of the first to third clocks supplied from said first to third clock supply means, respectively.

5. A coding apparatus comprising:

audio signal coding means for coding an audio signal to output compressed audio data, local-decoding the compressed audio data to output local-decoded audio data and outputting time information which represent the decoding timing of the compressed audio data;

video signal coding means for coding a video signal to output compressed video data, local-decoding the compressed video data to output local-decoded video data and outputting time information which represent the decoding timing of the compressed video data;

interface means for accepting information on a composite scene;

scene data coding means for coding scene data supplied from said interface means to output compressed scene data, local-decoding the compressed scene data to output local-decoded scene data and outputting time information which represent the decoding timing of the compressed scene data;

composition means for composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting the time information which represent the composite timing of the composed scene;

means for reproducing/displaying the composed scene supplied from said composition means;

clock supply means for supplying at least one clock to said audio signal coding means, said video signal coding means, said scene data coding means and said composition means; and

multiplexing means for creating a bit stream on the basis of the time information and compressed audio data supplied from said audio signal coding means, the time information and compressed video data supplied from said video signal coding means, the time information and compressed scene data supplied from said scene data coding means, the time information supplied from said composition means, and at least one clock value of at least one clock supplied from said clock supply means;

wherein said clock supply means includes first clock supply means for supplying first clock to said audio signal coding means, second clock supply means for supplying second clock to said video signal coding means and composition means, and third clock supply means for supplying third clock to said scene data coding means, and said multiplexing means multiplexes clock values of the first to third clocks supplied from said first to third clock supply means, respectively.

6. A decoding apparatus comprising:

separating means for separating first compressed data of an audio signal and first time information which represent the decoding timing of the first compressed data, second compressed data of a video signal and second time information which represent the decoding timing of the second compressed data, third compressed data of scene data and third time information which represent the decoding timing of the third compressed data, fourth time information of scene composition and at least one clock value, from a bit stream;

audio signal decoding means for decoding the audio signal on the basis of the first compressed data and the first time information;

video signal decoding means for decoding the video signal on the basis of the second compressed data and the second time information;

scene data decoding means for decoding the scene data on the basis of the third compressed data and the third time information;

composition means for composing a scene from the audio signal supplied from said audio signal decoding means, the video signal supplied from said video signal decoding means and the scene data supplied from said scene data decoding means, on the basis of the fourth time information supplied from said separation means;

means for generating at least one clock according to at least one clock value supplied from said separating means and supplying the clock to said audio signal decoding means, said video signal decoding means, said scene data decoding means and said composition means;

means for reproducing/displaying the composed scene supplied from said composition means.

7. The decoding apparatus as claimed in claim 6, further comprising interface means for accepting an interaction from a viewer to a composite picture, wherein said interface means accepts information on a composite scene according to the interaction and controls the scene composite processing of said composition means and reproducing/displaying.

8. The decoding apparatus as claimed in claim 6, wherein said separation means separates independent clock values from said bit stream, and the independent clock values are input to means for supplying the clock to said decoding means for the audio signal, means for supplying the clock to said decoding means for the video signal, and means for supplying the clock to said decoding means for the scene data and said composition means.

9. The decoding apparatus as claimed in claim 6, wherein said decoding apparatus decodes the bit stream generated by a generating method comprising the steps of:

coding an audio signal to output compressed audio data, local-decoding the compressed audio data to output local-decoded audio data and outputting the first time information which represent the decoding timing of the compressed audio data;

coding a video signal to output compressed video data, local-decoding the compressed video data to output local-decoded video data and outputting the second time information which represent the decoding timing of the compressed video data;

accepting information on a composite scene to generate scene data, coding the scene data to output compressed scene data, local-decoding the compressed scene data to output local-decoded scene data and outputting the third time information which represent the decoding timing of the compressed scene data;

composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting the fourth time information which represent the composite timing of the composed scene;

reproducing/displaying the composed scene; and

multiplexing the first time information and compressed audio data, the second time information and compressed video data, the third time information and compressed scene data, and the fourth time information to create a bit stream,

wherein a flag representing whether at least one time information of the first to third timing information doubles as time information about reproducing/displaying of the composed scene is added to said one time information.

10. A decoding apparatus comprising:

separating means for separating first compressed data of an audio signal and first time information which represent the decoding timing of the first compressed data, second compressed data of a video signal and second time information which represent the decoding timing of the second compressed data, third compressed data of scene data and third time information which represent the decoding timing of the third compressed data, fourth time information of scene composition and at least one clock value, from a bit stream;

audio signal decoding means for decoding the audio signal on the basis of the first compressed data and the first time information;

video signal decoding means for decoding the video signal on the basis of the second compressed data and the second time information;

scene data decoding means for decoding the scene data on the basis of the third compressed data and the third time information;

composition means for composing a scene from the audio signal supplied from said audio signal decoding means, the video signal supplied from said video signal decoding means and the scene data supplied from said scene data decoding means, on the basis of the fourth time information supplied from said separation means;

means for generating at least one clock according to at least one clock value supplied from said separating means and supplying the clock to said audio signal decoding means, said video signal decoding means, said scene data decoding means and said composition means;

means for reproducing/displaying the composed scene supplied from said composition means;

wherein said separation means separates independent clock values from said bit stream, and the independent clock values are input to means for supplying the clock to said decoding means for the audio signal, means for supplying the clock to said decoding means for the video signal and said composition means, and means for supplying the clock to said decoding means for the scene data.

11. A coding/decoding system comprising a coding apparatus and a decoding apparatus wherein said coding apparatus comprises:

audio signal coding means for coding an audio signal to output compressed audio data, local-decoding the compressed audio data to output local-decoded audio data and outputting time information which represent the decoding timing of the compressed audio data;

video signal coding means for coding a video signal to output compressed video data, local-decoding the compressed video data to output local-decoded video data and outputting time information which represent the decoding timing of the compressed video data;

interface means for accepting information on a composite scene;

scene data coding means for coding scene data supplied from said interface means to output compressed scene data, local-decoding the compressed scene data to output local-decoded scene data and outputting time information which represent the decoding timing of the compressed scene data;

composition means for composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting the time information which represent the composite timing of the composed scene;

means for reproducing/displaying the composed scene supplied from said composition means;

clock supply means for supplying at least one clock to said audio signal coding means, said video signal coding means, said scene data coding means and said composition means; and

multiplexing means for creating a bit stream on the basis of the time information and compressed audio data supplied from said audio signal coding means, the time information and compressed video data supplied from said video signal coding means, the time information and compressed scene data supplied from said scene data coding means, the time information supplied from said composition means, and at least one clock value of at least one clock supplied from said clock supply means;

and wherein said decoding apparatus comprises;

separating means for separating first compressed data of an audio signal and first time information which represent the decoding timing of the first compressed data, second compressed data of a video signal and second time information which represent the decoding timing of the second compressed data, third compressed data of scene data and third time information which represent the decoding timing of the third compressed data, fourth time information of scene composition and at least one clock value, from a bit stream;

audio signal decoding means for decoding the audio signal on the basis of the first compressed data and the first time information;

video signal decoding means for decoding the video signal on the basis of the second compressed data and the second time information;

scene data decoding means for decoding the scene data on the basis of the third compressed data and the third time information;

composition means for composing a scene from the audio signal supplied from said audio signal decoding means, the video signal supplied from said video signal decoding means and the scene data supplied from said scene data decoding means, on the basis of the fourth time information supplied from said separation means;

means for generating at least one clock according to at least one clock value supplied from said separating means and supplying the clock to said audio signal decoding means, said video signal decoding means, said scene data decoding means and said composition means; and

means for reproducing/displaying the composed scene supplied from said composition means.

12. A coding/decoding system comprising a coding apparatus and a decoding apparatus wherein said coding apparatus comprises:

audio signal coding means for coding an audio signal to output compressed audio data and local-decoding the compressed audio data to output local-decoded audio data;

video signal coding means for coding a video signal to output compressed video data and local-decoding the compressed video data to output local-decoded video data;

interface means for accepting information on a composite scene;

scene data coding means for coding scene data supplied from said interface means to output compressed scene data and local-decoding the compressed scene data to output local-decoded scene data;

composition means for composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting time information which represent the composite timing of the composed scene;

means for reproducing/displaying the composed scene supplied from said composition means;

clock supply means for supplying at least one clock to said audio signal coding means, said video signal coding means, said scene data coding means and said composition means; and

multiplexing means for creating a bit stream on the basis of the compressed audio data, the compressed video data, the compressed scene data, the time information and at least one clock value of at least one clock supplied from said clock supply means;

wherein said clock supply means includes first clock supply means for supplying first clock to said audio signal coding means, second clock supply means for supplying second clock to said video signal coding means and third clock supply means for supplying third clock to said scene data coding means and composition means, and said multiplexing means multiplexes clock values of the first to third clocks supplied from said first to third clock supply means, respectively;

and wherein said decoding apparatus comprises;

separating means for separating compressed data of an audio signal, compressed data of a video signal, compressed data of scene data and time information of scene composition from a bit stream;

audio signal decoding means for decoding the audio signal on the basis of the compressed data of the audio signal;

video signal decoding means for decoding the video signal on the basis of the compressed data of the video signal;

scene data decoding means for decoding the scene data on the basis of the compressed data of the scene data;

composition means for composing a scene from the audio signal supplied from said audio signal decoding means, the video signal supplied from said video signal decoding means and the scene data supplied from said scene data decoding means, on the basis of the time information for the scene composition supplied from said separation means; and

means for reproducing/displaying the composed scene supplied from said composition means;

wherein said separation means separates independent clock values from said bit stream, and the independent clock values are input to means for supplying the clock to said decoding means for the audio signal, means for supplying the clock to said decoding means for the video signal, and means for supplying the clock to said decoding means for the scene data and said composition means.

13. A coding/decoding system comprising a coding apparatus and a decoding apparatus wherein said coding apparatus comprises:

audio signal coding means for coding an audio signal to output compressed audio data and local-decoding the compressed audio data to output local-decoded audio data;

video signal coding means for coding a video signal to output compressed video data and local-decoding the compressed video data to output local-decoded video data;

interface means for accepting information on a composite scene;

scene data coding means for coding scene data supplied from said interface means to output compressed scene data and local-decoding the compressed scene data to output local-decoded scene data;

composition means for composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting time information which represent the composite timing of the composed scene;

means for reproducing/displaying the composed scene supplied from said composition means;

clock supply means for supplying at least one clock to said audio signal coding means, said video signal coding means, said scene data coding means and said composition means; and

multiplexing means for creating a bit stream on the basis of the compressed audio data, the compressed video data, the compressed scene data, the time information and at least one clock value of at least one clock supplied from said clock supply means;

wherein said clock supply means includes first clock supply means for supplying first clock to said audio signal coding means, second clock supply means for supplying second clock to said video signal coding means and composition means, and third clock supply means for supplying third clock to said scene data coding means, and said multiplexing means multiplexes clock values of the first to third clocks supplied from said first to third clock supply means, respectively;

and wherein said decoding apparatus comprises;

separating means for separating compressed data of an audio signal, compressed data of a video signal, compressed data of scene data and time information of scene composition from a bit stream;

audio signal decoding means for decoding the audio signal on the basis of the compressed data of the audio signal;

video signal decoding means for decoding the video signal on the basis of the compressed data of the video signal;

scene data decoding means for decoding the scene data on the basis of the compressed data of the scene data;

composition means for composing a scene from the audio signal supplied from said audio signal decoding means, the video signal supplied from said video signal decoding means and the scene data supplied from said scene data decoding means, on the basis of the time information for the scene composition supplied from said separation means; and

means for reproducing/displaying the composed scene supplied from said composition means;

wherein said separation means separates independent clock values from said bit stream, and the independent clock values are input to means for supplying the clock to said decoding means for the audio signal, means for supplying the clock to said decoding means for the video signal and said composition means, and means for supplying the clock to said decoding means for the scene data.

14. A coding method of a composite scene having a picture and an audio, which comprises the steps of:

coding an audio signal to output compressed audio data, local-decoding the compressed audio data to output local-decoded audio data and outputting the first time information which represent the decoding timing of the compressed audio data;

coding a video signal to output compressed video data, local-decoding the compressed video data to output local-decoded video data and outputting the second time information which represent the decoding timing of the compressed video data;

accepting information on a composite scene to generate scene data, coding the scene data to output compressed scene data, local-decoding the compressed scene data to output local-decoded scene data and outputting the third time information which represent the decoding timing of the compressed scene data;

composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting the fourth time information which represent the composite timing of the composed scene;

reproducing/displaying the composed scene; and

creating a bit stream on the basis of the first time information and compressed audio data, the second time information and compressed video data, the third time information and compressed scene data, the fourth time information and at least one clock value referred for coding the audio signal, the video signal and the scene data.

15. A decoding method of a bit stream, which comprises the steps of:

separating compressed data of an audio signal and first time information which represent the decoding timing of the compressed audio data, compressed data of a video signal and second time information which represent the decoding timing of the compressed video data, compressed data of scene data and third time information which represent the decoding timing of the compressed scene data, fourth time information of scene composition and at least one clock value, from a bit stream which a composite scene of a picture and an audio is coded and multiplexed;

generating at least one clock according to the separated clock value;

decoding the audio signal on the basis of the compressed data of the audio signal and the first time information with referring to the generated clock;

decoding the video signal on the basis of the compressed data of the video signal and the second time information with referring to the generated clock;

decoding the scene data on the basis of the compressed data of scene data and the third time information with referring to the generated clock;

composing a scene from the decoded audio signal, the decoded video signal and the decoded scene data with referring to the generated clock, on the basis of the fourth time information; and

reproducing/displaying the composed scene.

16. A generating method of a bit stream, which comprises the steps of:

coding an audio signal to output compressed audio data, local-decoding the compressed audio data to output local-decoded audio data and outputting the first time information which represent the decoding timing of the compressed audio data;

coding a video signal to output compressed video data, local-decoding the compressed video data to output local-decoded video data and outputting the second time information which represent the decoding timing of the compressed video data;

accepting information on a composite scene to generate scene data, coding the scene data to output compressed scene data, local-decoding the compressed scene data to output local-decoded scene data and outputting the third time information which represent the decoding timing of the compressed scene data;

composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting the fourth time information which represent the composite timing of the composed scene;

reproducing/displaying the composed scene; and

multiplexing the first time information and compressed audio data, the second time information and compressed video data, the third time information and compressed scene data, and the fourth time information to create a bit stream,

wherein a flag representing whether at least one time information of the first to third timing information doubles as time information about reproducing/displaying of the composed scene is added to said one time information.

17. A coding apparatus comprising:

audio signal coding means for coding an audio signal to output compressed audio data, local-decoding the compressed audio data to output local-decoded audio data and outputting time information which represent the decoding timing of the compressed audio data;

video signal coding means for coding a video signal to output compressed video data, local-decoding the compressed video data to output local-decoded video data and outputting time information which represent the decoding timing of the compressed video data;

interface means for accepting information on a composite scene;

scene data coding means for coding scene data supplied from said interface means to output compressed scene data, local-decoding the compressed scene data to output local-decoded scene data and outputting time information which represent the decoding timing of the compressed scene data;

composition means for composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting the time information which represent the composite timing of the composed scene;

means for reproducing/displaying the composed scene supplied from said composition means;

clock supply means for supplying at least one clock to said audio signal coding means, said video signal coding means, said scene data coding means and said composition means; and

multiplexing means for creating a bit stream on the basis of the time information and compressed audio data supplied from said audio signal coding means, the time information and compressed video data supplied from said video signal coding means, the time information and compressed scene data supplied from said scene data coding means, the time information supplied from said composition means, and at least one clock value of at least one clock supplied from said clock supply means;

wherein said multiplexing means generate the bit stream that a flag representing whether at least one time information of the first to third timing information doubles as time information about reproducing/displaying of the composite scene is added to said one time information.

18. A coding/decoding system comprising a coding apparatus and a decoding apparatus wherein said coding apparatus comprises:

audio signal coding means for coding an audio signal to output compressed audio data and local-decoding the compressed audio data to output local-decoded audio data;

video signal coding means for coding a video signal to output compressed video data and local-decoding the compressed video data to output local-decoded video data;

interface means for accepting information on a composite scene;

scene data coding means for coding scene data supplied from said interface means to output compressed scene data and local-decoding the compressed scene data to output local-decoded scene data;

composition means for composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting time information which represent the composite timing of the composed scene;

display means for reproducing/displaying the composed scene supplied from said composition means; and

multiplexing means for creating a bit stream on the basis of the compressed audio data, the compressed video data, the compressed scene data and the time information;

wherein said multiplexing means generate the bit stream that a flag representing whether at least one time information of the first to third timing information doubles as time information about reproducing/displaying of the composite scene is added to said one time information;

and wherein said decoding apparatus comprises;

separating means for separating first compressed data of an audio signal and first time information which represent the decoding timing of the first compressed data, second compressed data of a video signal and second time information which represent the decoding timing of the second compressed data, third compressed data of scene data and third time information which represent the decoding timing of the third compressed data, and fourth time information of scene composition, from a bit stream;

audio signal decoding means for decoding the audio signal on the basis of the first compressed data and the first time information;

video signal decoding means for decoding the video signal on the basis of the second compressed data and the second time information;

scene data decoding means for decoding the scene data on the basis of the third compressed data and the third time information;

composition means for composing a scene from the audio signal supplied from said audio signal decoding means, the video signal supplied from said video signal decoding means and the scene data supplied from said scene data decoding means, on the basis of the fourth time information supplied from said separation means; and

means for reproducing/displaying the composed scene supplied from said composition means;

wherein said decoding apparatus decodes the bit stream generated by a generating method comprising the steps of;

coding an audio signal to output compressed audio data, local-decoding the compressed audio data to output local-decoded audio data and outputting the first time information which represent the decoding timing of the compressed audio data;

coding a video signal to output compressed video data, local-decoding the compressed video data to output local-decoded video data and outputting the second time information which represent the decoding timing of the compressed video data;

accepting information on a composite scene to generate scene data, coding the scene data to output compressed scene data, local-decoding the compressed scene data to output local-decoded scene data and outputting the third time information which represent the decoding timing of the compressed scene data;

composing a scene from the local-decoded audio data, the local-decoded video data and the local-decoded scene data to output a composed scene and outputting the fourth time information which represent the composite timing of the composed scene;

reproducing/displaying the composed scene; and

multiplexing the first time information and compressed audio data, the second time information and compressed video data, the third time information and compressed scene data, and the fourth time information to create a bit stream,

wherein a flag representing whether at least one time information of the first to third timing information doubles as time information about reproducing/displaying of the composed scene is added to said one time information.