Multimedia adaptation based on video orientation

• US 9,762,938 B2
• Filed: 10/28/2013
• Issued: 09/12/2017
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Intel Corporation
• Inventors: Oyman, Ozgur
• Primary Examiner(s): Patel, Chirag R
• Assistant Examiner(s): Hackenberg, Rachel J
• Application Number:
  US14/125,595
• Publication Number:
  US 20150256861A1
• Time in Patent Office: 1,415 Days
• Field of Search: None
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/239 Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests hybrid fiber coaxial [HFC] networks for upstream channel allocation for video distribution H04L12/2801; flow control in data networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/25833 involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/437 Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6175 involving transmission via Internet broadcast-related systems characterised by the transmission system being the Internet H04H60/82
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/6437 Real-time Transport Protocol [RTP]
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format
• CPC Class Codes: Y02D 30/70 in wireless communication networks

Abstract

Technology for multimedia adaptation based on video orientation is disclosed. In an example, a server can include computer circuitry configured to: receive a device capability from a client device, wherein the device capability includes video orientation capability; and perform video orientation-aware multimedia adaptation based on the device'"'"'s video orientation capability.

Claims

1. A mobile terminal (MT) for providing orientation-aware processing capability, comprising

a processor configured to:

determine a video orientation capability of the MT;

parse a media presentation description (MPD) metadata file to identify a presence of video orientation information in the third generation partnership (3GP) file or parse the 3GP file for embedded video orientation information;

determine an orientation correction term based on parsed video orientation information and a current orientation of the mobile terminal, and

correct a rendering orientation of a real-time transport protocol (RTP) stream for misalignment based on the determined orientation correction term; and

,

a transceiver to transmit video orientation capability information to a server via a packet switched streaming service (PSS) device capability exchange signaling to enable the mobile terminal to;

access segments from the server via the PSS based on one or more of an orientation of the MT and the video orientation capability of the MT.

2. The mobile terminal of claim 1, wherein the processor is further configured to embed video orientation information for a captured video with a specified video orientation for a specified media segment in a third generation partnership project (3GPP) file format (3GP) file as an instantiation of an International Organization for Standardization (ISO) base media file format (ISO-BMFF) file.

3. The mobile terminal of claim 2, wherein the video orientation information is embedded in:

a video track or real-time transport protocol (RTP) hint track of the 3GP file;

a file-level ISO-BMFF box, wherein the file-level ISO-BMFF box is a ‘

moov’

box initialization segment or a ‘

moof’

box media segment;

a timed metadata track of the ISO-BMFF;

or

a SampleEntry with video orientation parameters within the sample description box in an ISO file format box structure.

4. The mobile terminal of claim 2, wherein:

the transceiver is further configured to;

upload the 3GP file for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH).

5. The mobile terminal of claim 1, wherein:

the transceiver is further configured to;

transmit a video orientation attribute via a packet switched streaming service (PSS) device capability exchange signaling, wherein the video orientation attribute is a PSS common component that includes;

an Orientation attribute selected from a list of different video orientation states supported by the client,

a DefaultOrientation attribute that provides a default video orientation to be taken as reference for a content production or selection,

an OrientationModesCapture attribute selected from a list of different video orientation modes supported by the client for a capturing,

an OrientationModesDisplay attribute selected from a list of different video orientation modes supported by the client for a rendering and a display,

a DefaultOrientationCapture attribute that provides a default video orientation for capturing based on the installation of the camera sensor if an orientation sensor is disabled or not present,

a DefaultOrientationDisplay attribute that provides a default video orientation for rendering and display if the orientation sensor is disabled or not present,

an OrientationAngularGranularityCapture attribute selected from a number of equally-spaced angular granularity levels in a 90 degree interval at which a device sensor can detect video orientation changes during the capture,

an OrientationAngularGranularityDisplay attribute selected from a number of equally-spaced angular granularity levels in the 90 degree interval at which the device sensor can detect video orientation changes during the rendering and the display or both the capture and the rendering and display,

a NumOfCameras attribute selected from a number of cameras on a device, or

a HorizontalMirroringSupport attribute to indicate whether the device supports horizontal mirroring when the device is orientation-aware.

6. The mobile terminal of claim 1, wherein the video orientation capability information is provided in a third generation partnership project (3GPP) long term evolution (LTE) packet switched streaming service (PSS) session, a dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) session, or an integrated multimedia subsystem (IMS) based PSS and multimedia broadcast and multicast service (MBMS) (IMS_PSS_MBMS) session.

7. The mobile terminal of claim 1, further comprising:

an orientation sensor to determine an orientation of the mobile terminal.

8. The mobile terminal of claim 1, wherein:

the transceiver is further configured to;

receive a real-time transport protocol (RTP) extension header for a RTP stream, or receive a third generation partnership project (3GPP) file format (3GP) file for a hypertext transfer protocol (HTTP) stream or dynamic adaptive streaming over HTTP (DASH); and

the processor is further configured to;

parse the media presentation description (MPD) metadata file to identify a presence of video orientation information in the 3GP file or parse the 3GP file for embedded video orientation information,

determine the orientation correction term based on the parsed video orientation information and a current orientation of the mobile terminal, and

correct a rendering orientation of the HTTP stream or DASH for misalignment based on the determined orientation correction term;

or

parse a session description protocol (SDP) file to identify a presence of video orientation information in the RTP stream or parse a RTP extension header for the RTP stream for the embedded video orientation information,

determine the orientation correction term based on the parsed video orientation information and the current orientation of the mobile terminal, and

correct the rendering orientation of the RTP stream for misalignment based on the determined orientation correction term,

wherein correcting the rendering orientation compensates for rotation or flip of the orientation of the MT.

9. The mobile terminal of claim 1, wherein the mobile terminal (MT) includes a client device, a user equipment (UE), or a mobile station (MS);

and the mobile terminal includes an antenna, a camera, a touch sensitive display screen, a speaker, a microphone, a graphics processor, an application processor, internal memory, or a non-volatile memory port.

10. The mobile terminal of claim 1, wherein a mobile terminal'"'"'s current video orientation capability information overrides a default video orientation capability information via profile difference (Profdiff) signaling.

11. The mobile terminal of claim 1, wherein the processor is further configured to instruct the mobile terminal to capture video with a specified video orientation for a specified media segment based on a device orientation.