Packet stream arrangement in multimedia transmission

• US 7,751,324 B2
• Filed: 11/19/2004
• Issued: 07/06/2010
• Est. Priority Date: 11/19/2004
• Status: Expired due to Fees

• Current Assignee: Nokia Corporation
• Original Assignee: Nokia Corporation
• Inventors: Hannuksela, Miska, Vadakital, Vinod Kumar Malamal, Wenger, Stephan
• Primary Examiner(s): Patel; Jayanti K
• Assistant Examiner(s): ZHU, BO HUI ALVIN
• Application Number:
  US10/993,918
• Publication Number:
  US 20060109805A1
• Time in Patent Office: 2,055 Days
• Field of Search: 370/229, 370/230, 370/230.1, 370/230.2, 714/752
• US Class Current: 370/231
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 21/236 Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/2383 Channel coding or modulation of digital bit-stream, e.g. QPSK modulation arrangements for detecting or preventing errors in the information received by adapting the channel coding H04L1/0009; analogue front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/434 Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream demultiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/4381 Recovering the multiplex stream from a specific network, e.g. recovering MPEG packets from ATM cells transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4382 Demodulation or channel decoding, e.g. QPSK demodulation analog front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/6131 involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/631 Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/6437 Real-time Transport Protocol [RTP]

Abstract

Transmitting media information from a transmitting device to a receiving device. To perform the transmission a media stream is encoded and packets are formed from the encoded media stream, the packets containing application data units, the application data units having a decoding order. At least two blocks are generated from the packets. The packets of the at least two blocks are organized into a first group and a second group, the first group being decodable without reception of said second group. The transmission order of the packets of said first group is arranged succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order.

Claims

1. Apparatus for transmitting media information comprising:

an encoder for encoding at least one media stream;

a packetizer for forming packets from the encoded media stream, the packets containing application data units, the application data units having a decoding order;

a packet block generator for generating at least two blocks from the packets;

an organizer for organizing the packets within a block of the at least two blocks into a first group and a second group, the first group being decodable without reception of said second group;

an orderer for arranging the transmission order of the packets of said first group succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order; and

a transmitter for transmitting the packets according to the transmission order, wherein said media stream comprises at least a video stream and an audio stream, said encoder comprising a video encoder and an audio encoder;

wherein said video encoder is adapted to encode the video stream to form coded pictures, said audio encoder is adapted to encode the audio stream to form encoded audio, and said organizer is adapted to organize the encoded audio into said first group and the coded pictures into said second group, wherein said encoder is adapted to encode the video stream to form coded pictures, and said organizer is adapted to organize the coded pictures into a reference picture group and a non-reference picture group, the reference picture group being decodable without reception of the non-reference picture group.

2. The apparatus according to claim 1 wherein said media stream comprises at least a video stream and an audio stream, said encoder comprising a video encoder and an audio encoder;

wherein said video encoder is adapted to encode the video stream to form coded pictures, said audio encoder is adapted to encode the audio stream to form encoded audio, and said organizer is adapted to organize the encoded audio into said first group and the coded pictures into said second group.

3. The apparatus according to claim 1 wherein said reorderer and depacketizer are combined into one operational block.

4. A device comprising:

an encoder for encoding at least one media stream;

a packetizer for forming packets from the encoded media stream, the packets containing application data units, the application data units having a decoding order;

a packet block generator for generating at least two blocks from the packets;

an organizer for organizing the packets within a block into a first group and a second group, the first group being decodable without reception of said second group; and

an orderer for arranging the transmission order of the packets of said first group succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order, said device further comprising a

controller configured for defining first error correction data for said first group on the basis of the packets of said first group, and a second error correction data for said second group on the basis of the packets of said second group, said packetizer is configured for selecting a first integral group of packets on the basis of said first group and said first error correction data and for selecting a second integral group on the basis of said second group and said second error correction data wherein said first group of packets can be restored from said first integral group and said second group of packet can be restored from said second integral group, and

wherein the orderer is adapted to order the transmission of said first integral group after the packets of said second integral group are ordered to be transmitted.

5. A device according to claim 4 wherein the packet selector is adapted to select said first integral group to comprise said first group and said second integral group to comprise said second group.

6. A device according to claim 5 wherein the orderer is configured to order the packets in at least one of the groups among said first integral group and said second integral group in reverse decoding order.

7. A device according to claim 4 wherein the encoder is adapted to form reference data units and non-reference data units wherein the organizer is adapted to organize the reference data units into said first group and the non-reference data units into said second group.

8. A device according to claim 7 wherein the reference data units and non-reference data units form more than one IDR period, wherein the orderer is adapted to order the transmission of packets formed from the IDR periods in decoding order of the IDR periods.

9. A device according to claim 4 wherein the packetizer is adapted to select a forward error correction code rate for said first group different from a forward error correction code rate for said second group.

10. A device comprising:

an encoder for encoding at least one media stream;

a packetizer for forming packets from the encoded media stream, the packets containing application data units, the application data units having a decoding order;

a packet block generator for generating at least two blocks from the packets;

an organizer for organizing the packets within a block into a first group and a second group, the first group being decodable without reception of said second group; and

an orderer for arranging the transmission order of the packets of said first group succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order, wherein the encoder is adapted to encode at least two different media types, and the organizer is adapted to organize the packets of one media type into said first group and the packets of another media type into said second group, wherein said one media type is a video stream comprising reference pictures, which are used for the prediction of other inter pictures, and non-reference pictures, which are not used for the prediction of other inter pictures, and the another media type is an audio stream, wherein the organizer is adapted to organize the packets of non-reference pictures and the packets of reference pictures into said first group such that the packets of reference pictures succeed the packets of non-reference pictures.

11. A device comprising:

an encoder for encoding at least one media stream;

a packetizer for forming packets from the encoded media stream, the packets containing application data units, the application data units having a decoding order;

a packet block generator for generating at least two blocks from the packets;

an organizer for organizing the packets within a block into a first group and a second group, the first group being decodable without reception of said second group; and

an orderer for arranging the transmission order of the packets of said first group succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order,

wherein the encoder is adapted to encode at least two different media types, the media types having been classified according to their importance, wherein the orderer is adapted to order the transmission of packets of the media types in ascending order of importance, wherein the packets of one media type are in said first group and the packets of another media type are in said second group, wherein said one media type is a video stream comprising reference pictures and non-reference pictures, and the another media type is an audio stream, wherein the packets of non-reference pictures and the packets of reference pictures are in said first group such that the packets of reference pictures succeed the packets of non-reference pictures.

12. A device comprising:

a packetizer for forming packets from at least one encoded media stream, the packets containing application data units, the application data units having a decoding order;

a packet block generator for generating at least two blocks from the packets;

an organizer for organizing the packets within a block into a first group and a second group, the first group being decodable without reception of said second group; and

an orderer for arranging the transmission order of the packets of said first group succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order, wherein the packets of one media type are in said first group and the packets of another media type are in said second group, wherein said one media type is a video stream comprising reference pictures and non-reference pictures, and the another media type is an audio stream, wherein the packets of non-reference pictures and the packets of reference pictures are in said first group such that the packets of reference pictures succeed the packets of non-reference pictures and wherein packets of said another media type succeed said packets of said first media type.

13. A device comprising:

a receiver for receiving packets including at least one encoded media stream and which packets are at least partly in a different order than the decoding order, wherein received packets of one media type are in a first group and received packets of another media type are in a second group, wherein said one media type is a video stream comprising reference pictures and non-reference pictures, and the another media type is an audio stream, wherein the packets of non-reference pictures and the packets of reference pictures are in said first group such that the packets of reference pictures succeed the packets of non-reference pictures and wherein packets of said another media type succeed said packets of said first media type;

a buffer for buffering received packets;

a reorderer for reordering received packets into the decoding order;

a depacketizer for reconstructing the encoded media information from the packets; and

a decoder for decoding media information of the packets.

14. A device according to claim 13 further comprising an error correction data decoder producing packets of the encoded media stream on the basis of received packets.

15. A device according to claim 14 comprising a common memory for the error correction data decoder and the reorderer.

16. A wireless communication device comprising:

an encoder for encoding at least one media stream;

a packetizer for forming packets from the encoded media stream, the packets containing application data units, the application data units having a decoding order;

a packet block generator for generating at least two blocks from the packets;

an organizer for organizing the packets within a block of the at least two blocks into a first group and a second group, the first group being decodable without reception of said second group; and

an orderer for arranging the transmission order of the packets of said first group succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order, wherein packets of one media type are in said first group and packets of another media type are in said second group, wherein said one media type is a video stream comprising reference pictures and non-reference pictures, and the another media type is an audio stream, wherein the packets of non-reference pictures and the packets of reference pictures are in said first group such that the packets of reference pictures succeed the packets of non-reference pictures and wherein packets of said another media type succeed said packets of said first media type.

17. A wireless communication device comprising:

a receiver for receiving packets including at least one encoded media stream and which packets are at least partly in a different order than a decoding order, wherein received packets of one media type are in a first group and received packets of another media type are in a second group, wherein said one media type is a video stream comprising reference pictures and non-reference pictures, and the another media type is an audio stream, wherein the packets of non-reference pictures and the packets of reference pictures are in said first group such that the packets of reference pictures succeed the packets of non-reference pictures and wherein packets of said another media type succeed said packets of said first media type;

a buffer for buffering received packets;

a reorderer for reordering received packets into the decoding order;

a depacketizer for reconstructing the encoded media information from the packets; and

a decoder for decoding media information of the packets.

18. A method for transmitting media information from a transmitting device to a receiving device comprising:

encoding at least one media stream, wherein said media stream comprises at least a video stream and an audio stream, the video stream comprising coded pictures, said audio stream comprising encoded audio, said encoded audio organized into a first group and the coded pictures into a second group, said coded pictures organized into a reference picture group and a non-reference picture group, the reference picture group being decodable without reception of the non-reference picture group;

forming packets from the encoded media stream, the packets containing application data units, the application data units having a decoding order;

generating at least two blocks from the packets;

organizing the packets of the at least two blocks into a first group and a second group, the first group being decodable without reception of said second group;

arranging the transmission order of the packets of said first group succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order; and

transmitting the packets to the receiving device according to the transmission order.

19. A device comprising a controller and a memory for transmitting a signal carrying encoded media information to a receiving device, said encoded media information comprising packets from at least one encoded media stream, the packets containing application data units, the application data units having a decoding order;

and at least two blocks are generated from the packets by said controller of said device running machine executable steps, wherein the packets of the signal are ordered by said controller executing said steps by;

organizing packets within a block of the at least two blocks into a first group and a second group, the first group being decodable without reception of said second group; and

arranging the transmission order of the packets of said first group succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order,

wherein said packets comprise packets of at least two different media types, wherein

the packets of one media type are in said first group and the packets of another media type are in said second group, wherein said one media type is a video stream comprising reference pictures and non-reference pictures, and the another media type is an audio stream, wherein the packets of non-reference pictures and the packets of reference pictures are in said first group such that the packets of reference pictures succeed the packets of non-reference pictures.

20. A device according to claim 19 wherein a forward error correction code rate have been selected for said first group different from a forward error correction code rate for said second group.

21. A controller comprising a software program for transmitting media information from a first device to a second device, the software program comprising machine executable steps for:

encoding a media stream, wherein said media stream comprises at least a video stream and an audio stream, the video stream comprising coded pictures, said audio stream comprising encoded audio, said encoded audio organized into a first group and the coded pictures into a second group, said coded pictures organized into a reference picture group and a non-reference picture group, the reference picture group being decodable without reception of the non-reference picture group;

forming packets from the encoded media stream, the packets containing application data units, the application data units having a decoding order;

generating at least two blocks from the packets;

organizing the packets within a block of the at least two blocks into a first group and a second group, the first group being decodable without reception of said second group;

arranging the transmission order of the packets of said first group succeeding packets of said second group such that the application data units in the transmission order are at least partly in a different order than the decoding order; and

transmitting the packets to the receiving device according to the transmission order.

22. A device comprising a controller and a memory for coding a media steam or decoding a coded media stream, comprising

a first integral group of packets and a second integral group of packets;

said first integral group of packets transmitted after said second integral group of packets;

said first integral group of packets decodable to first packets of the coded media stream;

a second integral group of packets decodable to second packets of the coded media stream;

said first packets of the coded media stream decodable to uncompressed media samples without decoding of said second packets of the coded media stream;

at least one packet in said second packets of the coded media stream succeeding in decoding order at least one packet in said first packets of the coded media stream wherein said coded media stream comprises packets of at least two different media types, wherein the packets of one media type are in said first integral group and the packets of another media type are in said second integral group, wherein said one media type is a video stream comprising reference pictures and non-reference pictures, and the another media type is an audio stream, wherein the packets of non-reference pictures and the packets of reference pictures are in said first integral group such that the packets of reference pictures succeed the packets of non-reference pictures and wherein packets of said another media type succeed said packets of said one media type.

23. A device according to claim 22 in which decoding of both said first packets of a coded media stream and said second packets of a coded media stream improves a subjective quality compared to decoding of said first packets of the coded media stream.

24. A device according to claim 22 in which decoding of both said first packets of the coded media stream and said second packets of the coded media stream increases the output sample rate compared to decoding of said first packets of the coded media stream.

25. A device according to claim 22 wherein the packets within at least one of said first integral group of packets and said second integral group of packets are in reverse decoding order.

26. A device comprising a controller and a memory for coding or decoding a signal comprising at least two coded media streams wherein

each media stream is organized into super blocks of media data;

earliest and latest media decoding time is derivative for super blocks;

super blocks do not overlap in decoding time;

super blocks are ordered within the signal in ascending order of decoding time;

super block boundaries in decoding time of one media stream are within a defined range compared to super block boundaries of another media stream; and

at least one inter-stream frame is comprised of super blocks of a different stream whose boundaries are within said defined range, wherein

there is at least a first group of data of first media type and a second group of data of second media type;

said first group of data is not interleaved with said second group of data in the signal, wherein

said first group of data precedes said second group of data if the decoded output of said first group of data is subjectively less important than the decoded output of said second group of data.

27. A receiving device comprising:

a receiver for receiving packets at least partly in an order different than a decoding order, wherein received video packets are in a first group and received audio packets are in a second group, wherein said received video packets form a video stream comprising reference pictures and non-reference pictures, and the received audio packets comprise an audio stream, wherein video packets of non-reference pictures and video packets of reference pictures are in said first group such that the packets of reference pictures succeed the packets of non-reference pictures and wherein said audio packets succeed said video packets;

a reorderer for reordering received packets into the decoding order; and

a depacketizer for reconstructing the encoded media information from the packets.