Device orientation capability exchange signaling and server adaptation of multimedia content in response to device orientation

• US 9,357,272 B2
• Filed: 12/19/2012
• Issued: 05/31/2016
• Est. Priority Date: 08/03/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Intel Corporation
• Inventors: Oyman, Ozgur
• Primary Examiner(s): Jean, Frantz
• Application Number:
  US13/719,402
• Publication Number:
  US 20140040959A1
• Time in Patent Office: 1,259 Days
• Field of Search: 709/203, 709/217, 709/219, 709/220, 709/231, 725/62
• US Class Current: 1/1
• CPC Class Codes: H04B 17/27 for locating or positioning the transmitter
• CPC Class Codes: H04L 41/0659 by isolating or reconfiguring faulty entities
• CPC Class Codes: H04L 41/0836 to enhance reliability, e.g. reduce downtime
• CPC Class Codes: H04L 43/0811 by checking connectivity
• CPC Class Codes: H04L 45/306 Route determination based on the nature of the carried application
• CPC Class Codes: H04L 65/1016 IP multimedia subsystem [IMS]
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/303 Terminal profiles
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/25825 involving client display capabilities, e.g. screen resolution of a mobile phone optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/25833 involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055
• CPC Class Codes: H04N 21/26258 for generating a list of items to be played back in a given order, e.g. playlist, or scheduling item distribution according to such list retrieval of multimedia data based on playlists G06F16/40
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6131 : involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 16/18 : Network planning tools
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 24/02 : Arrangements for optimising operational condition
• CPC Class Codes: H04W 24/04 : Arrangements for maintaining operational condition
• CPC Class Codes: H04W 36/14 : Reselecting a network or an air interface
• CPC Class Codes: H04W 4/026 : using orientation information, e.g. compass
• CPC Class Codes: H04W 4/70 : Services for machine-to-machine communication [M2M] or machine type communication [MTC]
• CPC Class Codes: H04W 40/02 : Communication route or path selection, e.g. power-based or shortest path routing
• CPC Class Codes: H04W 40/20 : based on geographic position or location
• CPC Class Codes: H04W 40/246 : Connectivity information discovery
• CPC Class Codes: H04W 48/14 : using user query or user detection
• CPC Class Codes: H04W 48/18 : Selecting a network or a communication service
• CPC Class Codes: H04W 76/10 : Connection setup
• CPC Class Codes: H04W 76/11 : Allocation or use of connection identifiers
• CPC Class Codes: H04W 76/14 : Direct-mode setup
• CPC Class Codes: H04W 76/15 : Setup of multiple wireless link connections
• CPC Class Codes: H04W 76/16 : Involving different core network technologies, e.g. a packet-switched [PS] bearer in combination with a circuit-switched [CS] bearer
• CPC Class Codes: H04W 76/28 : Discontinuous transmission [DTX]; Discontinuous reception [DRX]
• CPC Class Codes: H04W 88/06 : adapted for operation in multiple networks or having at least two operational modes, e.g. multi-mode terminals
• CPC Class Codes: Y02D 30/70 : in wireless communication networks

Abstract

Capability exchange signaling techniques provide orientation sensor information from UEs to network servers. The orientation sensor information describes a device'"'"'s support for orientation sensor capabilities, or a current orientation state of the device. Based on such information, a multimedia server provides different encoded versions of multimedia content for different device orientation modes supported by the device. The server may also adapt, dynamically and in real-time, media capture or transcode parameters for creating content tailored (i.e., optimized) for the device'"'"'s current orientation mode, or for its various intermediate orientation states and spatial positions.

Claims

1. A user equipment (UE) configured to wirelessly communicate with a remote media content server and initiate capability exchange signaling, the UE comprising:

an orientation sensor;

a display device;

a radio-frequency transceiver;

a processor operably coupled to the radio-frequency transceiver; and

a memory operably coupled to the processor, the memory storing instructions that configure the processor to;

transmit, via the radio-frequency transceiver, information that provides device orientation capabilities of the UE to the remote media content server or a device profile server, the device orientation capabilities of the UE including one or more predetermined orientation positions that correspond to orientation information provided by the orientation sensor;

transmit to the remote media content server a message indicating a current orientation state and including a request for video content for display with the display device based on the device orientation capabilities of the UE;

receive from the remote media content server a session description protocol (SDP) file for delivery of the video content selected from a group comprising;

streaming the video content according to a real-time streaming protocol (RTSP),

communicating the video content using dynamic adaptive streaming over HTTP (DASH), and

downloading the video content over HTTP or File Delivery over Unidirectional Transport (FLUTE) protocol; and

receive from the remote media content server the video content that is identified in the SDP file and selected by the remote media content server based on the current orientation state of the UE.

2. The UE of claim 1, in which the instructions further configure the processor to:

receive from the remote media content server a manifest file describing multiple versions of the video content configured for playback on the display device according to orientation states indicated in the device orientation capabilities of the UE.

3. The UE of claim 2, in which the multiple versions of the video content are encoded based on device orientation capabilities for playback on mobile device displays, and in which the instructions further configure the processor to:

select, for streaming or download delivery, based on the current orientation state of the UE, a version of the video content described in the manifest file.

4. The UE of claim 3, in which the manifest file comprises a media presentation description (MPD) file, and in which the instructions further configure the processor to:

provide the information that indicates the device orientation capabilities of the UE to the remote media content server with a hypertext transfer protocol (HTTP) POST message;

request from the media content server the MPD file via an HTTP GET message;

receive from the remote media content server the MPD file; and

request the selected version of the video content from the remote media content server using dynamic adaptive streaming over HTTP (DASH).

5. The UE of claim 1, in which the remote media content server includes an IP multimedia subsystem (IMS) based packet switched streaming service, and in which the instructions further configure the processor to:

transmit in a session initiation protocol (SIP) SUBSCRIBE message to an Internet Protocol Multimedia Core Network subsystem during service discovery the information that provides device orientation capabilities of the UE.

6. A method of controlling delivery of streaming or download of multimedia content from a server to a user equipment (UE) client over a wireless network, the method comprising:

providing to the server, via capability exchange signaling, information indicative of device orientation capability of the client to interpret video orientation information from the server according to device orientation modes supported at the client;

receiving from the server, in response to the information, a metadata file identifying a version of the multimedia content that is available for streaming or downloading to the client and is configured for an orientation mode of a display device of the client;

determining the orientation mode of the display device of the client; and

streaming or downloading via the wireless network, the version of the multimedia content that is identified in the metadata file and selected based on a physical orientation of the client, wherein the wireless network employs at least one of packet switched streaming service (PSS), integrated multimedia subsystems (IMS) based PSS, and multimedia broadcast and multicast service MBMS (IMS_PSS_MBMS).

7. The method of claim 6, in which the client comprises a mobile device and the information includes mobile device identity information that identifies the mobile device by model.

8. The method of claim 7, in which the server comprises a PSS server, and in which the step of providing comprises:

initiating a device capability profile exchange between the PSS server and a device profile server that stores device capability profiles of multiple mobile device models, a device capability profile including a list of predetermined device orientation modes supported by the client.

9. The method of claim 6, further comprising:

dynamically switching between streams of the multimedia content based on changes to the physical orientation of the client.

10. The method of claim 6, in which the orientation mode includes at least one of a landscape mode, a portrait mode, and a seascape mode.

11. The method of claim 6, in which the metadata file comprises a session description protocol (SDP) file for delivery of the multimedia content selected from a group comprising:

streaming the multimedia content according to a real-time streaming protocol (RTSP),

delivering the multimedia content using dynamic adaptive streaming over HTTP (DASH), and

downloading the multimedia content over HTTP or File Delivery over Unidirectional Transport (FLUTE) protocol.

12. The method of claim 6, in which the metadata file comprises a media presentation description (MPD) file for streaming the multimedia content according to dynamic adaptive streaming over HTTP (DASH).

13. One or more non-transitory computer-readable media having instructions that, when executed, cause a media content server to:

receive, via a communication network, a device orientation attribute that includes a list of device orientation positions supported by a client terminal of a wireless communication network, the device orientation positions relevant for streaming or download of video content over a transport protocol supported by a packet switched streaming service application on the client terminal;

adapt video content based on the device orientation attribute through one or more processes including content selection, capture, compression, and transcoding operations;

generate a session description or metadata file to establish a streaming or download session based on the device orientation attribute; and

deliver the adapted video content and the generated session description or metadata file to the client terminal.

14. The one or more non-transitory computer-readable media of claim 13, in which the list of device orientation positions includes one or more indication selected from a group comprising of video display device orientation modes, video capture device orientation modes, video display device orientation angles, video capture device orientation angles, a boolean-valued indicator having values indicating whether the client terminal is responsive to changes in orientation, default device orientation for rendering and display, default device orientation for capture, number of equally-spaced angular granularity levels in a 90 degree interval at which an orientation sensor of the client terminal is configured to detect orientation changes during both video capture and display, number of cameras associated with the client terminal, and horizontal mirroring support.

15. The one or more non-transitory computer-readable media of claim 13, wherein the list comprises a list of one or more orientation positions that respectively correspond to one or more device orientation modes supported by the client terminal.

16. The one or more non-transitory computer-readable media of claim 13, wherein the transport protocol comprises a real-time transport protocol (RTP).

17. The one or more non-transitory computer-readable media of claim 13, wherein the transport protocol comprises a hypertext transport protocol (HTTP).

18. The one or more non-transitory computer-readable media of claim 13, wherein the session description or metadata file is a session description protocol (SDP) file for delivery of the video content selected from a group comprising:

streaming the video content according to a real-time streaming protocol (RTSP),

streaming the video content using dynamic adaptive streaming over HTTP (DASH), and

downloading the video content over HTTP or File Delivery over Unidirectional Transport (FLUTE) protocol, or a media presentation description (MPD) file for DASH.

19. The one or more non-transitory computer-readable media of claim 13, wherein the instructions, when executed, further cause the media content server to:

transcode the video content or convert a format of the video content based on the device orientation attribute.

20. The one or more non-transitory computer-readable media of claim 13, wherein the device orientation attribute is received from a device profile server via the communication network, and wherein the client terminal comprises a user equipment (UE).

21. A user equipment (UE) configured for coordination of video orientation in a third generation partnership project (3GPP) network, the UE comprising:

a network interface to communicate with a packet-switched streaming service (PSS) server through a radio access network (RAN) of the 3GPP network; and

a PSS client to;

signal, through the network interface, a capability of the PSS client to interpret video orientation information;

receive, through the network interface from the PSS server, a session description protocol (SDP) file, the SDP file including current video orientation information indicating a current orientation of video received from the PSS server; and

rotate the video received from the PSS server based on the current orientation indicated in the current video orientation information.

22. The UE of claim 21, wherein the PSS client is further configured to execute a capability exchange with the PSS server, the capability exchange including at least one of the capability of the PSS client to interpret the video orientation information and a capability of the client to display the video based on the video orientation information.

23. The UE of claim 22, wherein the capability exchange further includes an indication of a granularity of rotation angles provided by the UE.

24. The UE of claim 23, wherein the indication of the granularity is selected from a group comprising:

a lower granularity corresponding to 90-degree rotation modes, and

a higher granularity corresponding to a plurality of equally-spaced angular granularity levels within a 90 degree interval.

25. The UE of claim 21, wherein the PSS client is further configured to horizontally flip the video received from the PSS server based on the current orientation indicated in the current video orientation information.

26. The UE of claim 21, wherein the PSS client is further configured to rotate the video before horizontally flipping the video.

27. The UE of claim 21, further comprises one or antennas for wireless communications with at least one of a wireless communication device and a wireless network.

28. The UE of claim 21, further comprising at least one of a speaker, a microphone, a connection port, a keyboard, a touch screen, a battery, and a memory port.

29. One or more non-transitory computer-readable media having instructions that, when executed, cause a media content server to:

perform packet-switched streaming service (PSS) capability exchange with a user equipment (UE) through a third generation partnership project (3GPP) network, the PSS capability exchange specifying device capability attributes for coordination of video orientation;

select, based on the PSS capability exchange, content comprising a first orientation of an image; and

signal the first orientation of the image to the UE.

30. The one or more non-transitory computer-readable media of claim 29, wherein the instructions, when executed, further cause the media content server to:

receive an indication of a change in a current orientation of the UE;

in response to the indication of the change in the current orientation of the UE, select content at the server comprising a second orientation of the image; and

signal to the UE the second orientation of the image.

31. The one or more non-transitory computer-readable media of claim 29, wherein the instructions, when executed, further cause the media content server to:

signal the first orientation of the image to the UE in a session description protocol (SDP) file and a real-time transport protocol (RTP).

32. The one or more non-transitory computer-readable media of claim 29, wherein the instructions, when executed, further cause the media content server to:

receive, from the PSS capability exchange through the 3GPP network, a device orientation attribute that includes a list of one or more device orientation positions, supported by the UE, relevant for video that can be included in a 3GPP file format file and handled by a packet switched streaming service application on the UE; and

deliver content to the UE based on the device orientation attribute.

33. The one or more non-transitory computer-readable media of claim 32, wherein the list of one or more device orientation positions includes an indication of a default orientation mode and a current orientation mode indicated by an orientation sensor of the UE.