Multimedia adaptation based on video orientation

• US 10,523,982 B2
• Filed: 08/24/2017
• Issued: 12/31/2019
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Intel Corporation
• Inventors: Oyman, Ozgur
• Primary Examiner(s): Patel, Chirag R
• Assistant Examiner(s): Hackenberg, Rachel J
• Application Number:
  US15/685,812
• Publication Number:
  US 20170353743A1
• Time in Patent Office: 859 Days
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/239 Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests hybrid fiber coaxial [HFC] networks for upstream channel allocation for video distribution H04L12/2801; flow control in data networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/25833 involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/437 Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6175 involving transmission via Internet broadcast-related systems characterised by the transmission system being the Internet H04H60/82
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/6437 Real-time Transport Protocol [RTP]
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format
• CPC Class Codes: Y02D 30/70 in wireless communication networks

Abstract

Technology for a mobile terminal (MT) operable to provide an orientation-aware processing capability is disclosed. The MT can determine video orientation capability information for the MT. The MT can process the video orientation capability information for transmission to a server via a packet switched streaming service (PSS) device capability exchange signaling between the MT and the server. The MT can access segments from the server via the PSS based on an orientation of the MT and the video orientation capability information for the MT.

Claims

1. An apparatus of a mobile terminal (MT) operable to provide an orientation-aware processing capability, the apparatus comprising one or more processors and memory configured to:

determine, at the MT, video orientation capability information for the MT;

parse a media presentation description (MPD) metadata file to identify a presence of video orientation information in a third generation partnership (3GP) file or parse the 3GP file for the video orientation information;

process, at the MT, the video orientation capability information for transmission to a server via a packet switched streaming service (PSS) device capability exchange signaling between the MT and the server; and

access, at the MT, segments from the server via the PSS based on an orientation of the MT and the video orientation capability information for the MT.

2. The apparatus of a claim 1, further comprising a transceiver configured to:

transmit the video orientation capability information to the server; and

receive the segments from the server.

3. The apparatus of claim 1, wherein the one or more processors and memory are further configured to:

determine an orientation correction term based on the video orientation information and a current orientation of the MT; and

correct a rendering orientation of a real-time transport protocol (RTP) stream for misalignment based on the orientation correction term.

4. The apparatus of claim 1, wherein the one or more processors and memory are further configured to embed video orientation information for a captured video with a specified video orientation for a specified media segment in a third generation partnership project (3GPP) file format (3GP) file as an instantiation of an International Organization for Standardization (ISO) base media file format (ISO-BMFF) file.

5. The apparatus of claim 4, wherein the video orientation information is embedded in:

a video track or real-time transport protocol (RTP) hint track of the 3GP file;

a file-level ISO-BMFF box, wherein the file-level ISO-BMFF box is a ‘

moov’

box initialization segment or a ‘

moof’

box media segment;

a timed metadata track of the ISO-BMFF;

or

a SampleEntry with video orientation parameters within a sample description box in an ISO file format box structure.

6. The apparatus of claim 4, wherein the one or more processors and memory are further configured to prepare the 3GP file to upload for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH).

7. The apparatus of claim 1, wherein the one or more processors and memory are further configured to process a video orientation attribute for transmission via the PSS device capability exchange signaling, wherein the video orientation attribute is a PSS common component that includes:

an Orientation attribute selected from a list of different video orientation states supported by a client,

a DefaultOrientation attribute that provides a default video orientation to be taken as reference for a content production or selection,

an OrientationModesCapture attribute selected from a list of different video orientation modes supported by the client for a capturing,

an OrientationModesDisplay attribute selected from a list of different video orientation modes supported by the client for a rendering and a display,

a DefaultOrientationCapture attribute that provides a default video orientation for capturing based on an installation of a camera sensor if an orientation sensor is disabled or not present,

a DefaultOrientationDisplay attribute that provides a default video orientation for rendering and display if the orientation sensor is disabled or not present,

an OrientationAngularGranularityCapture attribute selected from a number of equally-spaced angular granularity levels in a 90 degree interval at which a device sensor can detect video orientation changes during the capture,

an OrientationAngularGranularityDisplay attribute selected from a number of equally-spaced angular granularity levels in the 90 degree interval at which the device sensor can detect video orientation changes during the rendering and the display or both the capture and the rendering and display,

a NumOfCameras attribute selected from a number of cameras on a device, or

a HorizontalMirroringSupport attribute to indicate whether the device supports horizontal mirroring when the device is orientation-aware.

8. The apparatus of claim 1, wherein the one or more processors and memory are further configured to provide the video orientation capability information in a third generation partnership project (3GPP) long term evolution (LTE) packet switched streaming service (PSS) session, a dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) session, or an integrated multimedia subsystem (IMS) based PSS and multimedia broadcast and multicast service (MBMS) (IMS_PSS_MBMS) session.

9. The apparatus of claim 1, further comprising an orientation sensor operable to determine the orientation of the MT.

10. The apparatus of claim 1, wherein the one or more processors and memory are further configured to:

process a real-time transport protocol (RTP) extension header received for a RTP stream, or a third generation partnership project (3GPP) file format (3GP) file received for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH);

parse a media presentation description (MPD) metadata file to identify a presence of video orientation information in the 3GP file or parse the 3GP file for the video orientation information;

determine an orientation correction term based on the video orientation information and the orientation of the MT; and

correct a rendering orientation of the HTTP stream or DASH for misalignment based on the orientation correction term,

wherein correcting the rendering orientation compensates for rotation or flip of the orientation of the MT.

11. The apparatus of claim 1, wherein the one or more processors and memory are further configured to:

process a real-time transport protocol (RTP) extension header received for a RTP stream, or a third generation partnership project (3GPP) file format (3GP) file received for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH);

parse a session description protocol (SDP) file to identify a presence of video orientation information in the RTP stream or parse a RTP extension header for the RTP stream for the video orientation information;

determine an orientation correction term based on the video orientation information and the orientation of the MT; and

correct a rendering orientation of the RTP stream for misalignment based on the orientation correction term,

wherein correcting the rendering orientation compensates for rotation or flip of the orientation of the MT.

12. The apparatus of claim 1, wherein the one or more processors and memory are further configured to capture video with a specified video orientation for a specified media segment based on the orientation of the MT.

13. The apparatus of claim 1, wherein the video orientation capability information for the MT overrides a default video orientation capability information via profile difference (Profdiff) signaling.

14. The apparatus of claim 1, wherein the MT includes a client device, a user equipment (UE), or a mobile station (MS);

and the MT includes an antenna, a camera, a touch sensitive display screen, a speaker, a microphone, a graphics processor, an application processor, an internal memory, or a non-volatile memory port.

15. An apparatus of a server operable to process an orientation-aware capability received from a mobile terminal (MT), the apparatus comprising one or more processors and memory configured to:

process, at the server, video orientation capability information received from the MT via a packet switched streaming service (PS S) device capability exchange signaling between the server and the MT, wherein the video orientation capability information is for the MT; and

process, at the server, segments for transmission to the MT via the PSS based on an orientation of the MT and the video orientation capability information received from the MT; and

process a media presentation description (MPD) metadata file for transmission to the MT to enable the MT to parse the MPD metadata file to identify a presence of video orientation information in a third generation partnership (3GP) file or parse the 3GP file for the video orientation information.

16. The apparatus of a claim 15, further comprising a transceiver configured to:

receive the video orientation capability information from the MT; and

transmit the segments to the MT.

17. The apparatus of a claim 15, wherein the one or more processors and memory are configured to process a video orientation attribute received from the MT via the PSS device capability exchange signaling.

18. At least one non-transitory machine readable storage medium having instructions embodied thereon for providing an orientation-aware processing capability from a mobile terminal (MT), the instructions when executed by one or more processors cause the MT to perform the following:

determining, at the MT, video orientation capability information for the MT;

parsing a media presentation description (MPD) metadata file to identify a presence of video orientation information in a third generation partnership (3GP) file or parse the 3GP file for the video orientation information;

processing, at the MT, the video orientation capability information for transmission to a server via a packet switched streaming service (PSS) device capability exchange signaling between the MT and the server; and

accessing, at the MT, segments from the server via the PSS based on an orientation of the MT and the video orientation capability information for the MT.

19. The at least one non-transitory machine readable storage medium of claim 18, further comprising instructions which when executed perform the following:

determining an orientation correction term based on the video orientation information and the orientation of the MT; and

correcting a rendering orientation of a real-time transport protocol (RTP) stream for misalignment based on the orientation correction term.

20. The at least one non-transitory machine readable storage medium of claim 18, further comprising instructions which when executed perform the following:

embedding video orientation information for a captured video with a specified video orientation for a specified media segment in a third generation partnership project (3GPP) file format (3GP) file as an instantiation of an International Organization for Standardization (ISO) base media file format (ISO-BMFF) file.

21. The at least one non-transitory machine readable storage medium of claim 20, wherein the video orientation information is embedded in:

a video track or real-time transport protocol (RTP) hint track of the 3GP file;

a file-level ISO-BMFF box, wherein the file-level ISO-BMFF box is a ‘

moov’

box initialization segment or a ‘

moof’

box media segment;

a timed metadata track of the ISO-BMFF;

or

a SampleEntry with video orientation parameters within a sample description box in an ISO file format box structure.

22. The at least one non-transitory machine readable storage medium of claim 18, further comprising instructions which when executed perform the following:

preparing the 3GP file to upload for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH).

23. The at least one non-transitory machine readable storage medium of claim 18, further comprising instructions which when executed perform the following:

processing a video orientation attribute for transmission via the PSS device capability exchange signaling, wherein the video orientation attribute is a PSS common component that includes;

an Orientation attribute selected from a list of different video orientation states supported by a client,

a DefaultOrientation attribute that provides a default video orientation to be taken as reference for a content production or selection,

an OrientationModesCapture attribute selected from a list of different video orientation modes supported by the client for a capturing,

an OrientationModesDisplay attribute selected from a list of different video orientation modes supported by the client for a rendering and a display,

a DefaultOrientationCapture attribute that provides a default video orientation for capturing based on an installation of a camera sensor if an orientation sensor is disabled or not present,

a DefaultOrientationDisplay attribute that provides a default video orientation for rendering and display if the orientation sensor is disabled or not present,

an OrientationAngularGranularityCapture attribute selected from a number of equally-spaced angular granularity levels in a 90 degree interval at which a device sensor can detect video orientation changes during the capture,

an OrientationAngularGranularityDisplay attribute selected from a number of equally-spaced angular granularity levels in the 90 degree interval at which the device sensor can detect video orientation changes during the rendering and the display or both the capture and the rendering and display,

a NumOfCameras attribute selected from a number of cameras on a device, or

a HorizontalMirroringSupport attribute to indicate whether the device supports horizontal mirroring when the device is orientation-aware.

24. The at least one non-transitory machine readable storage medium of claim 18, further comprising instructions which when executed perform the following:

providing the video orientation capability information in a third generation partnership project (3GPP) long term evolution (LTE) packet switched streaming service (PSS) session, a dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) session, or an integrated multimedia subsystem (IMS) based PSS and multimedia broadcast and multicast service (MBMS) (IMS_PSS_MBMS) session.

25. The at least one non-transitory machine readable storage medium of claim 18, further comprising instructions which when executed perform the following:

processing a real-time transport protocol (RTP) extension header received for a RTP stream, or a third generation partnership project (3GPP) file format (3GP) file received for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH);

parsing a media presentation description (MPD) metadata file to identify a presence of video orientation information in the 3GP file or parse the 3GP file for the video orientation information;

determining an orientation correction term based on the video orientation information and the orientation of the MT; and

correcting a rendering orientation of the HTTP stream or DASH for misalignment based on the orientation correction term,

wherein correcting the rendering orientation compensates for rotation or flip of the orientation of the MT.

26. The at least one non-transitory machine readable storage medium of claim 19, further comprising instructions which when executed perform the following:

processing a real-time transport protocol (RTP) extension header received for a RTP stream, or a third generation partnership project (3GPP) file format (3GP) file received for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH);

parsing a session description protocol (SDP) file to identify a presence of video orientation information in the RTP stream or parse a RTP extension header for the RTP stream for the video orientation information;

determining an orientation correction term based on the video orientation information and the orientation of the MT; and

correcting a rendering orientation of the RTP stream for misalignment based on the orientation correction term,

wherein correcting the rendering orientation compensates for rotation or flip of the orientation of the MT.

27. The at least one non-transitory machine readable storage medium of claim 18, further comprising instructions which when executed perform the following:

capturing video with a specified video orientation for a specified media segment based on the orientation of the MT.

28. The at least one non-transitory machine readable storage medium of claim 18, wherein the video orientation capability information for the MT overrides a default video orientation capability information via profile difference (Profdiff) signaling.