Streaming with coordination of video orientation (CVO)

• US 9,438,658 B2
• Filed: 12/01/2015
• Issued: 09/06/2016
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Intel Corporation
• Inventors: Oyman, Ozgur
• Primary Examiner(s): Gillis, Brian J
• Assistant Examiner(s): Ballard, Matthew
• Application Number:
  US14/956,130
• Publication Number:
  US 20160088053A1
• Time in Patent Office: 280 Days
• US Class Current: 1/1
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format
• CPC Class Codes: H04W 88/02 Terminal devices
• CPC Class Codes: Y02D 30/70 in wireless communication networks

Abstract

Technology to provide streaming with coordination of video orientation (CVO) is disclosed. In an example, a server can include computer circuitry configured to: receive a device capability for a client; and modify streamed content to the client based on an inclusion of a CVO attribute in the device capability.

Claims

1. An apparatus of a server operable to provide streaming with coordination of video orientation (CVO), the apparatus comprising one or more processors and memory configured to:

process, at the server, a device capability exchange message received from a client, wherein the device capability exchange message includes a CVO attribute; and

modify, at the server, content to be streamed to the client based on the CVO attribute received in the device capability exchange message, wherein the content is modified by correcting a rendering orientation of a real-time transport protocol (RTP) stream or streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content for misalignment prior to delivery to the client when the CVO attribute included in the device capability exchange message indicates that the client is not an orientation-aware terminal.

2. The apparatus of claim 1, wherein the one or more processors and memory are further configured to:

process, at the server for transmission to the client, a media presentation description (MPD) metadata file for a streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content with a CVO indication attribute provided in a codecs common attribute or element for an AdaptationSet, Representation, or SubRepresentation, wherein the codecs attribute is set to ‘

3gvo’

indicating a presence of an oriented video component and associated CVO information in the streamed DASH content; and

process, at the server for transmission to the client, a session description protocol (SDP) file for a real-time transport protocol (RTP) stream with a CVO indication via an a=extmap attribute with a CVO uniform resource name (URN) “

urn;

3gpp;

video-orientation”

representing a 2-bit granularity for a CVO or “

urn;

3gpp;

video-orientation;

6”

representing a 6-bit granularity for a Higher Granularity CVO on the CVO information contained in an RTP extension header.

3. The apparatus of claim 1, wherein the one or more processors and memory are further configured to:

store CVO data in a third generation partnership project (3GPP) file format (3GP) file using a CVOSampleEntry for a CVO timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a ‘

3gvo’

value;

or

store CVO data for a real-time transport protocol (RTP) stream in an RTP extension header.

4. The apparatus of claim 1, wherein the one or more processors and memory are further configured to:

exchange a packet switched streaming service (PSS) capability, wherein the CVO attribute includes;

a StreamingCVOCapable attribute to indicate whether the client is a CVO capable receiver of real-time transport protocol (RTP) streams,

a StreamingHighGranularityCVOCapable attribute to indicate whether the client is a Higher Granularity CVO capable receiver of the RTP streams,

a ThreeGPCVOCapable attribute to indicate whether the client is a CVO capable receiver of third generation partnership project (3GPP) file format (3GP) files, or

a ThreeGPHighGranularityCVOCapable attribute to indicate whether the client is a Higher Granularity CVO capable receiver of the 3GP files.

5. The apparatus of claim 1, wherein the one or more processors and memory are further configured to modify the streamed content by performing orientation-aware content adaptation, orientation-aware content selection, orientation-aware transcoding, or orientation-aware format conversion to correct a video orientation misalignment and ensure a content playback at the client with a correct video orientation.

6. The apparatus of claim 1, wherein the server includes a third generation partnership project (3GPP) long term evolution (LTE) packet switched streaming service (PSS) server, a dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) server, or an integrated multimedia subsystem (IMS) based PSS and multimedia broadcast and multicast service (MBMS) (IMS_PSS_MBMS) server.

7. An apparatus of a mobile terminal (MT) operable to provide coordination of video orientation (CVO) capability, the apparatus comprising one or more processors and memory configured to:

determine, at the MT, the CVO capability of the MT; and

process, at the MT, a streaming component attribute that includes the CVO capability of the MT for transmission to a server, wherein the server is configured to modify content to be streamed to the MT based on the CVO capability of the MT, wherein the server is configured to modify the content by correcting a rendering orientation of the content for misalignment when the CVO capability of the MT indicates that the MT is not an orientation-aware terminal.

8. The apparatus of claim 7, further comprising:

a transceiver configured to receive a real-time transport protocol (RTP) extension header for an RTP stream or a third generation partnership project (3GPP) file format (3GP) file for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH),

wherein the one or more processors and memory are further configured to;

parse a media presentation description (MPD) metadata file for the 3GP file for a CVO indication attribute or parse the 3GP file for embedded CVO information,

determine a orientation correction term based on the parsed CVO information and a current orientation of the MT, and

correct a rendering orientation of the HTTP stream or DASH for misalignment based on the determined orientation correction term when the MPD metadata file includes the CVO indication attribute and the MT is an orientation-aware terminal;

or

parse a session description protocol (SDP) file for the RTP stream for the CVO indication attribute or parse the RTP extension header for the RTP stream for the embedded CVO information,

determine a orientation correction term based on the parsed CVO information and current orientation of the client device, and

correct a rendering orientation of the RTP stream for misalignment based on the determined orientation correction term when the SDP file includes the CVO indication attribute and the MT is the orientation-aware terminal, wherein correcting the rendering orientation compensates for rotation or flip of an orientation.

9. The apparatus of claim 7, further comprising:

a transceiver configured to receive a media presentation description (MPD) metadata file from the server for streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content with a CVO indication attribute provided in a codecs common attribute or element for an AdaptationSet, Representation, or SubRepresentation, wherein the codecs attribute is set to ‘

3gvo’

indicating a presence of an oriented video component and an associated CVO information in the streamed DASH content,

wherein the one or more processors and memory are further configured to;

parse the MPD metadata file for the CVO indication attribute; and

modify a rendering orientation of a HTTP stream or the streamed DASH content for misalignment when the MPD metadata file includes the CVO indication attribute and the MT is an orientation-aware terminal.

10. The apparatus of claim 7, further comprising:

a transceiver configured to receive a third generation partnership project (3GPP) file format (3GP) file including a CVOSampleEntry for a CVO timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a 3gvo value, wherein the one or more processors and memory are further configured to;

parse the 3GP file for the CVOSampleEntry; and

modify a rendering orientation of a hypertext transfer protocol (HTTP) stream or streamed dynamic adaptive streaming over HTTP (DASH) content for misalignment when the 3GP file includes the CVOSampleEntry and the MT is an orientation-aware terminal.

11. The apparatus of claim 7, further comprising a transceiver configured to:

exchange a packet switched streaming service (PSS) client capability with the CVO capability of the MT, wherein the CVO capability of the MT includes;

a StreamingCVOCapable attribute to indicate whether the client is a CVO capable receiver of real-time transport protocol (RTP) streams,

a StreamingHighGranularityCVOCapable attribute to indicate whether the client is a Higher Granularity CVO capable receiver of RTP streams,

a ThreeGPCVOCapable attribute to indicate whether the client is a CVO capable receiver of third generation partnership project (3GPP) file format (3GP) files, or

a ThreeGPHighGranularityCVOCapable attribute to indicate whether the client is a Higher Granularity CVO capable receiver of 3GP files.

12. The apparatus of claim 7, wherein the one or more processors and memory are further configured to:

capture user-generated content (UGC) video with a specified orientation; and

embed CVO information for the UGC video in a third generation partnership project (3GPP) file format (3GP) file; and

process for transmission a 3GP file for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH).

13. The apparatus of claim 7, wherein the one or more processors and memory are further configured to:

capture CVO data with a specified orientation; and

store the CVO data in a third generation partnership project (3GPP) file format (3GP) file using CVOSampleEntry for a CVO timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a 3gvo value; and

process for transmission a 3GP file for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH).

14. The apparatus of claim 7, wherein the CVO capability of the MT is provided in a third generation partnership project (3GPP) long term evolution (LTE) packet switched streaming service (PSS) session, a dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) session, or an integrated multimedia subsystem (IMS) based PSS and multimedia broadcast and multicast service (MBMS) (IMS_PSS_MBMS) session.

15. The apparatus of claim 7, further comprising an orientation sensor to determine an orientation of the MT.

16. The apparatus of claim 7, wherein the MT includes a user equipment (UE) or a mobile station (MS), and the mobile terminal includes an antenna, a camera, a touch sensitive display screen, a speaker, a microphone, a graphics processor, an application processor, internal memory, or a non-volatile memory port.

17. At least one non-transitory machine readable storage medium having instructions embodied thereon for signaling coordination of video orientation (CVO) capability of a mobile terminal (MT) at a server, the instructions when executed perform the following:

processing, using one or more processors of the server, a device capability exchange message received from a client, wherein the device capability exchange message includes a CVO attribute; and

adapting, using the one or more processors of the server, content to be streamed to the client based on the CVO attribute received in the device capability exchange message, wherein the content is modified by correcting a rendering orientation of a real-time transport protocol (RTP) stream or streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content for misalignment prior to delivery to the client when the CVO attribute included in the device capability exchange message indicates that the client is not an orientation-aware terminal.

18. The at least one non-transitory machine readable storage medium of claim 17, wherein adapting the streamed content further comprises:

embedding a CVO indication attribute in a media presentation description (MPD) metadata file or a session description protocol (SDP) file when the device capability for the client includes the CVO attribute indicating that the client is an orientation-aware terminal to modify the display orientation of a stream.

19. The at least one non-transitory machine readable storage medium of claim 17, wherein adapting the streamed content further comprises:

processing, at the server for transmission to the client, a media presentation description (MPD) metadata file for a streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content with a CVO indication attribute provided in a codecs common attribute or element for an AdaptationSet, Representation, or SubRepresentation, wherein the codecs attribute is set to ‘

3gvo’

indicating a presence of an oriented video component and associated CVO information in the stream DASH content; and

processing, at the server for transmission to the client, a session description protocol (SDP) file for a real-time transport protocol (RTP) stream with a CVO indication via an a=extmap attribute with a CVO uniform resource name (URN) urn;

3gpp;

video-orientation”

representing a 2-bit granularity for a CVO or “

urn;

3gpp;

video-orientation;

6”

representing a 6-bit granularity for a Higher Granularity CVO on the CVO information contained in a RTP extension header.

20. The at least one non-transitory machine readable storage medium of claim 17, wherein adapting the streamed content further comprises:

processing, at the server, user-generated content (UGC) video received from the client, wherein the UGC video includes embedded CVO information for the UGC video in a third generation partnership project (3GPP) file format (3GP) file, wherein the 3GP file uses;

a CVOSampleEntry for a CVO timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a 3gvo value;

or

a codecs common attribute or element for an AdaptationSet, Representation, or SubRepresentation of a media presentation description (MPD), wherein the codecs attribute is set to ‘

3gvo’

indicating a presence of an oriented video component and associated CVO information in a streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content.

21. The at least one non-transitory machine readable storage medium of claim 17, wherein adapting the streamed content further comprises:

storing CVO data in a third generation partnership project (3GPP) file format (3GP) file using a CVOSampleEntry for a CVO timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a 3gvo value;

or

storing CVO data for a real-time transport protocol (RTP) stream in a RTP extension header.

22. The at least one non-transitory machine readable storage medium of claim 17, wherein processing the device capability exchange message received from a client further comprises:

processing a packet switched streaming service (PSS) client capability received from the client, wherein the CVO capability includes;

a StreamingCVOCapable attribute to indicate whether the client is a CVO capable receiver of real-time transport protocol (RTP) streams, a StreamingHighGranularityCVOCapable attribute to indicate whether the client is a Higher Granularity CVO capable receiver of RTP streams, a ThreeGPCVOCapable attribute to indicate whether the client is a CVO capable receiver of third generation partnership project (3GPP) file format (3GP) files, or a ThreeGPHighGranularityCVOCapable attribute to indicate whether the client is a Higher Granularity CVO capable receiver of 3GP files.