Streaming with coordination of video orientation (CVO)

• US 10,432,692 B2
• Filed: 08/09/2016
• Issued: 10/01/2019
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Intel Corporation
• Inventors: Oyman, Ozgur
• Primary Examiner(s): Book, Phyllis A
• Application Number:
  US15/232,331
• Publication Number:
  US 20160352799A1
• Time in Patent Office: 1,148 Days
• Field of Search: None
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format
• CPC Class Codes: H04W 88/02 Terminal devices
• CPC Class Codes: Y02D 30/70 in wireless communication networks

Abstract

Technology a client operable to receive streaming content from a server is disclosed. The client can signal a device capability exchange message for transmission to the server. The client can process streaming content received from the server. A rendering orientation of the streaming content can be corrected for misalignment at the server prior to delivery of the streaming content to the client when the device capability exchange message indicates that the client is not an orientation-aware terminal.

Claims

1. An apparatus of a client operable to receive streaming content from a server, the apparatus comprising one or more processors and memory configured to:

signal, at the client, a device capability exchange message for transmission to the server, wherein the device capability exchange message indicates that the client is not an orientation-aware terminal; and

process, at the client, streaming content received from the server, wherein a rendering orientation of the streaming content is corrected for misalignment at the server prior to delivery of the streaming content to the client when the device capability exchange message indicates that the client is not an orientation-aware terminal.

2. The apparatus of claim 1, further comprising a transceiver configured to:

transmit the device capability exchange message to the server; and

receive the streaming content from the server,

wherein the server includes a third generation partnership project (3GPP) long term evolution (LTE) packet switched streaming service (PSS) server, a dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) server, or an integrated multimedia subsystem (IMS) based PSS and multimedia broadcast and multicast service (MBMS) (IMS_PSS_MBMS) server.

3. The apparatus of claim 1, wherein the streaming content includes a real-time transport protocol (RTP) stream or streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content.

4. The apparatus of claim 1, wherein at least one of:

an orientation-aware content adaptation, an orientation-aware content selection, an orientation-aware transcoding, or an orientation-aware format conversion is performed at the server to correct a video orientation misalignment and ensure a content playback at the client with a correct video orientation.

5. The apparatus of claim 1, wherein the one or more processors and memory are further configured to:

process, at the client, a media presentation description (MPD) metadata file received from the server, wherein the MPD metadata file is for a streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content with a coordination of video orientation (CVO) indication attribute provided in a codecs common attribute or element for an AdaptationSet, Representation, or SubRepresentation, wherein the codecs attribute is set to ‘

3gvo’

indicating a presence of an oriented video component and associated CVO information in the streamed DASH content; and

process, at the client, a session description protocol (SDP) file received from the server, wherein the SDP file is for a real-time transport protocol (RTP) stream with a CVO indication via an a=extmap attribute with a CVO uniform resource name (URN) “

urn;

3gpp;

video-orientation”

representing a 2-bit granularity for a CVO or “

urn;

3gpp;

video-orientation;

6”

representing a 6-bit granularity for a Higher Granularity CVO on the CVO information contained in an RTP extension header.

6. The apparatus of claim 1, wherein:

coordination of video orientation (CVO) data is stored in a third generation partnership project (3GPP) file format (3GP) file using a CVOSampleEntry for a CVO timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a ‘

3gvo’

value;

or

CVO data is stored for a real-time transport protocol (RTP) stream in an RTP extension header.

7. The apparatus of claim 1, wherein the one or more processors and memory are further configured to:

exchange a packet switched streaming service (PSS) capability with the server, wherein the device capability exchange message includes a coordination of video orientation (CVO) attribute, and the CVO attribute includes;

a StreamingCVOCapable attribute to indicate whether the client is a CVO capable receiver of real-time transport protocol (RTP) streams,

a StreamingHighGranularityCVOCapable attribute to indicate whether the client is a Higher Granularity CVO capable receiver of the RTP streams,

a ThreeGPCVOCapable attribute to indicate whether the client is a CVO capable receiver of third generation partnership project (3GPP) file format (3GP) files, or

a ThreeGPHighGranularityCVOCapable attribute to indicate whether the client is a Higher Granularity CVO capable receiver of the 3GP files.

8. At least one non-transitory machine readable storage medium having instructions embodied thereon for sending streaming content from a server to a mobile terminal (MT), the instructions when executed perform the following:

processing, using one or more processors of the server, a device capability exchange message received from a client, wherein the device capability exchange message indicates that the client is not an orientation-aware terminal; and

adapting, using the one or more processors of the server, streaming content prior to delivery of the streaming content to the client when the device capability exchange message indicates that the client is not an orientation-aware terminal.

9. The at least one non-transitory machine readable storage medium of claim 8, wherein adapting the streaming content further comprises performing at least one of:

an orientation-aware content adaptation, an orientation-aware content selection, an orientation-aware transcoding, or an orientation-aware format conversion at the server to correct a video orientation misalignment and ensure a content playback at the client with a correct video orientation.

10. The at least one non-transitory machine readable storage medium of claim 8, wherein adapting the streaming content further comprises embedding a coordination of video orientation (CVO) indication attribute in a media presentation description (MPD) metadata file or a session description protocol (SDP) file when the device capability for the client includes the CVO attribute indicating that the client is an orientation-aware terminal to modify the display orientation of a stream.

11. The at least one non-transitory machine readable storage medium of claim 8, wherein adapting the streaming content further comprises:

processing, at the server for transmission to the client, a media presentation description (MPD) metadata file for a streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content with a coordination of video orientation (CVO) indication attribute provided in a codecs common attribute or element for an AdaptationSet, Representation, or SubRepresentation, wherein the codecs attribute is set to ‘

3gvo’

indicating a presence of an oriented video component and associated CVO information in the stream DASH content; and

processing, at the server for transmission to the client, a session description protocol (SDP) file for a real-time transport protocol (RTP) stream with a CVO indication via an a=extmap attribute with a CVO uniform resource name (URN) urn;

3gpp;

video-orientation”

representing a 2-bit granularity for a CVO or “

urn;

3gpp;

video-orientation;

6”

representing a 6-bit granularity for a Higher Granularity CVO on the CVO information contained in a RTP extension header.

12. The at least one non-transitory machine readable storage medium of claim 8, wherein adapting the streaming content further comprises:

processing, at the server, user-generated content (UGC) video received from the client, wherein the UGC video includes embedded coordination of video orientation (CVO) information for the UGC video in a third generation partnership project (3GPP) file format (3GP) file, wherein the 3GP file uses;

a CVOSampleEntry for a CVO timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a 3gvo value;

or

a codecs common attribute or element for an AdaptationSet, Representation, or SubRepresentation of a media presentation description (MPD), wherein the codecs attribute is set to ‘

3gvo’

indicating a presence of an oriented video component and associated CVO information in a streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content.

13. The at least one non-transitory machine readable storage medium of claim 8, wherein adapting the streaming content further comprises:

storing coordination of video orientation (CVO) data in a third generation partnership project (3GPP) file format (3GP) file using a CVOSampleEntry for a CVO timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a 3gvo value;

or

storing CVO data for a real-time transport protocol (RTP) stream in a RTP extension header.

14. An apparatus of a mobile terminal (MT) operable to receive streaming content from a server, the apparatus comprising one or more processors and memory configured to:

determine, at the MT, an orientation capability of the MT; and

process, at the MT, a device capability exchange message for transmission to the server that includes the orientation capability of the MT, wherein the server is configured to correct a rendering orientation of streaming content based on the orientation capability of the MT.

15. The apparatus of claim 14, wherein the one or more processors and memory are further configured to process a real-time transport protocol (RTP) extension header received from the server, wherein the RTP header extension is for an RTP stream or a third generation partnership project (3GPP) file format (3GP) file for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH).

16. The apparatus of claim 14, wherein the one or more processors and memory are further configured to process a media presentation description (MPD) metadata file received from the server, wherein the MPD meta data file is for streamed dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) content with a coordination of video orientation (CVO) indication attribute provided in a codecs common attribute or element for an AdaptationSet, Representation, or SubRepresentation, wherein the codecs attribute is set to ‘

3gvo’

indicating a presence of an oriented video component and an associated CVO information in the streamed DASH content.

17. The apparatus of claim 14, wherein the one or more processors and memory are further configured to receive a third generation partnership project (3GPP) file format (3GP) file from the server, wherein the 3GP file includes a CVOSampleEntry for a coordination of video orientation (CVO) timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a 3gvo value.

18. The apparatus of claim 14, wherein the one or more processors and memory are further configured to:

capture user-generated content (UGC) video with a specified orientation;

embed coordination of video orientation (CVO) information for the UGC video in a third generation partnership project (3GPP) file format (3GP) file; and

process for transmission a 3GP file for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH).

19. The apparatus of claim 14, wherein the one or more processors and memory are further configured to:

capture coordination of video orientation (CVO) data with a specified orientation;

store the CVO data in a third generation partnership project (3GPP) file format (3GP) file using CVOSampleEntry for a CVO timed metadata track in a sample description box of International Organization for Standardization (ISO) file format box structure, wherein CVOSampleEntry fields include a BoxHeader size or type, a Data-reference-index, or a Granularity, and wherein the BoxHeader type is configured for a 3gvo value; and

process for transmission a 3GP file for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH).

20. The apparatus of claim 14, further comprising an orientation sensor to determine an orientation of the MT.