Real-time network transport protocol interface method and apparatus

• US 8,095,680 B2
• Filed: 07/10/2008
• Issued: 01/10/2012
• Est. Priority Date: 12/20/2007
• Status: Active Grant

• Current Assignee: Blackberry Limited
• Original Assignee: Telefonaktiebolaget LM Ericsson
• Inventors: Pettersson, Mattias, Gabin, Frederic
• Primary Examiner(s): Follansbee, John
• Assistant Examiner(s): TRAN, NGHI V
• Application Number:
  US12/171,179
• Publication Number:
  US 20090164655A1
• Time in Patent Office: 1,279 Days
• Field of Search: 709/220, 709/231, 709/232
• US Class Current: 709/231
• CPC Class Codes: H04L 65/1059 End-user terminal functionalities specially adapted for real-time communication
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355

Abstract

An electronic device has a multimedia framework including an integration layer and a higher-level application layer. The integration layer includes predefined media processing components having an input/output interface configurable based on the type and format of media content delivered to the device during a streaming media session and that provide predefined media processing functions to the application layer for enabling playback of the media content. A streaming media management component included in the integration layer establishes and manages the streaming media session and configures one or more content pipes used during the streaming media session to access packets received by one or more sockets. The component also maps the content pipes to the sockets and processes packets retrieved from the one or more content pipes such that the media content is provided to corresponding ones of the predefined media processing components in accordance with their configured input/output interfaces.

Claims

1. A multimedia-enabled device having a multimedia framework including an integration layer and a higher-level OpenMAX application layer, the integration layer of the multimedia framework comprising:

a plurality of OpenMAX-compliant predefined media processing components having an input/output interface configurable based on the type and format of media content delivered to the multimedia-enabled device during a streaming media session which is established responsive to the OpenMAX application layer requesting delivery of the media content in a real-time network transport protocol format, the predefined media processing components operable to provide predefined media processing functions to the OpenMAX application layer via the integration layer for enabling playback of the media content; and

an OpenMAX-compliant streaming media management component operable to;

provide an interface between the integration layer and the OpenMAX application layer so that the OpenMAX application layer has access to the predefined media processing components;

establish and manage the streaming media session;

configure one or more content pipes used during the streaming media session to access packets received by one or more sockets, the packets being encapsulated according to a real-time network transport protocol;

map the one or more content pipes to the one or more sockets; and

provide a common boundary between the content pipes and the predefined media processing components for seamlessly handling the media content delivered to the electronic device in the real-time network transport protocol format, including processing of packets retrieved from the one or more content pipes such that the media content is provided to corresponding ones of the predefined media processing components in accordance with their configured input/output interfaces;

wherein the streaming media management component comprises a first processing component operable to;

configure the one or more content pipes based on the type and format of the media content and the real-time network transport protocol used to deliver the media content;

convert synchronization information delivered to the multimedia-enabled device and timestamps extracted from the packets to new timestamp information compatible with the corresponding ones of the predefined media processing components; and

extract data from the packets in proper order with the new timestamp information; and

wherein the streaming media management component comprises a second processing component operable to;

convert the extracted data to coded media frame data based on the type and format of the media content and the real-time network transport protocol used to transmit the media content; and

output the coded media frame data in a format compatible with the configured input/output interfaces of the corresponding ones of the predefined media processing components.

2. The multimedia-enabled device of claim 1, wherein each processing component of the streaming media management component has at least one content pipe input and at least one output port configurable based on the type and format of the media content and the real-time network transport protocol used to deliver the media content.

3. The multimedia-enabled device of claim 1, wherein the first processing component is operable to output the extracted data in proper order with the new timestamp information to a local file stored at the multimedia-enabled device and the second processing component is operable to convert the local file from stored packet data to the coded media frame data for playback of the media content.

4. The multimedia-enabled device of claim 1, wherein the streaming media management component is operable to:

configure one of the content pipes in accordance with TCP; and

initiate the streaming media session via the content pipe configured in accordance with TCP.

5. The multimedia-enabled device of claim 4, wherein the streaming media management component is operable to initiate and control the streaming media session using RTSP.

6. The multimedia-enabled device of claim 4, wherein the streaming media management component is operable to configure one or more additional ones of the content pipes in accordance with UDP for accessing RTP packets received during the streaming media session.

7. The multimedia-enabled device of claim 1, wherein the streaming media management component is operable to:

configure at least one of the content pipes in accordance with UDP; and

provide quality of service feedback information related to the media content via each content pipe configured in accordance with UDP using RTCP.

8. The multimedia-enabled device of claim 1, wherein the streaming media management component is operable to de-multiplex a plurality of different media streams from the retrieved packets.

9. The multimedia-enabled device of claim 1, wherein the streaming media management component is operable to de-interleave the retrieved packets based on header information extracted from the retrieved packets, extract payload data from the de-interleaved packets and convert the extracted data to coded media frame data to a format compatible with the configured input/output interfaces of the corresponding ones of the predefined media processing components.

10. The multimedia-enabled device of claim 1, wherein the streaming media management component is operable to determine the type and format of the media content based on session description information delivered to the multimedia-enabled device or by accessing a URI link.

11. The multimedia-enabled device of claim 1, wherein the first processing component of the streaming media management component is further operable to:

input coded media frame data from one or more of the predefined media processing components operable to encode media data;

convert the coded media frame data to new packets in accordance with the real-time network transport protocol; and

output the new packets to the second processing component of the streaming media management component; and

wherein the second processing component is further operable to;

connect to one or more of the content pipes configured in accordance with UDP; and

send the new packets to the one or more of the content pipes configured in accordance with UDP for transmission via one or more of the sockets.

12. In a multimedia-enabled device having a multimedia framework including an integration layer and a higher-level OpenMAX application layer, the integration layer of the multimedia framework having a plurality of OpenMAX-compliant predefined media processing components with input/output interfaces configurable based on the type and format of media content delivered to the multimedia-enabled device during a streaming media session which is established responsive to the OpenMAX application layer requesting delivery of the media content in a real-time network transport protocol format, the predefined media processing components operable to provide predefined media processing functions to the OpenMAX application layer for enabling playback of the media content, the method comprising, within the integration layer:

providing an interface between the integration layer and the OpenMAX application layer so that the OpenMAX application layer has access to the predefined media processing components;

establishing and managing the streaming media session;

configuring one or more content pipes used during the streaming media session to access packets received by one or more sockets based on the type and format of the media content and a real-time network transport protocol used to deliver the media content;

mapping the one or more content pipes to the one or more sockets;

providing a common boundary between the content pipes and the predefined media processing components for seamlessly handling the media content delivered to the multimedia-enabled device in the real-time network transport protocol format, including processing packets extracted from the one or more content pipes such that the media content is provided to corresponding ones of the predefined media processing components in accordance with their configured input/output interfaces;

converting synchronization information delivered to the multimedia-enabled device and timestamps extracted from the packets to new timestamp information compatible with the corresponding ones of the predefined media processing components;

converting the packets to coded media frame data based on the type and format of the media content and the real-time network transport protocol used to transmit the media content; and

providing the coded media frame data in a format compatible with the configured input/output interfaces of the corresponding ones of the predefined media processing components.

13. The method of claim 12 comprising, within the integration layer:

configuring one of the content pipes in accordance with TCP; and

initiating and controlling the streaming media session via the content pipe configured in accordance with TCP.

14. The method of claim 13 comprising initiating the streaming media session within the integration layer using RTSP.

15. The method of claim 13 comprising configuring one or more additional ones of the content pipes in accordance with UDP within the integration layer for accessing RTP packets received during the streaming media session.

16. The method of claim 12 comprising, within the integration layer:

configuring at least one of the content pipes in accordance with UDP; and

providing quality of service feedback information related to the media content via each content pipe configured in accordance with UDP using RTCP.

17. The method of claim 12 comprising de-multiplexing a plurality of different media streams from the packets within the integration layer.

18. The method of claim 12 comprising, within the integration layer:

storing data extracted from the packets in proper order to a local file stored at the multimedia-enabled device; and

subsequently converting the local file to coded media frame data in a format compatible with the configured input/output interfaces of the corresponding ones of the predefined media processing components.

19. The method of claim 12 comprising, within the integration layer:

de-interleaving the packets based on header information extracted from the packets;

extracting payload data from the de-interleaved packets; and

converting the extracted data to coded media frame data in a format compatible with the configured input/output interfaces of the corresponding ones of the predefined media processing components.

20. The method of claim 12 comprising determining the type and format of the media content within the integration layer based on session description information delivered to the multimedia-enabled device or by accessing a URI link.

21. The method of claim 12 comprising, within the integration layer:

converting coded media frame data provided by one or more of the predefined media processing components to new packets in accordance with the real-time network transport protocol;

connecting to one or more of the content pipes configured in accordance with UDP; and

sending the new packets to the one or more of the content pipes configured in accordance with UPD for transmission via one or more of the sockets.