Interactive video conferencing

• US 9,386,275 B2
• Filed: 12/24/2014
• Issued: 07/05/2016
• Est. Priority Date: 01/06/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Intel IP Corporation (Intel Corporation)
• Inventors: Oyman, Ozgur
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US14/582,376
• Publication Number:
  US 20150195490A1
• Time in Patent Office: 559 Days
• Field of Search: 345/156, 348/14.02, 348/14.03, 348/14.05, 348/14.07, 348/14.08, 348/14.09, 348/14.12, 348/14.16, 375/240.02, 375/240.03, 375/240.24, 381/92, 382/173, 382/232, 382/239, 715/716, 709/219, 726/4
• US Class Current: 1/1
• CPC Class Codes: H04L 65/1069 Session establishment or de-establishment
• CPC Class Codes: H04L 65/1083 In-session procedures
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/4053 without floor control
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 2007/145 Handheld terminals
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/632 using a connection between clients on a wide area network, e.g. setting up a peer-to-peer communication via Internet for retrieving video segments from the hard-disk of other client devices broadcast-related systems characterised by transmission among terminal devices H04H60/80; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/6437 Real-time Transport Protocol [RTP]
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: Y02D 30/70 in wireless communication networks

Abstract

Technology for operating a video conferencing application at a local user equipment (UE) that supports an interactive zooming feature is disclosed. A local UE can define a region of interest (ROI) within a field of view at a remote UE. The local UE can communicate the ROI to the remote UE via a real-time transport control protocol (RTCP) feedback message, wherein the RTCP feedback message directs the remote UE to capture video within the ROI and encode the video within the ROI. The local UE can receive the encoded video within the ROI from the remote UE, the encoded video including regions within the ROI and excluding regions outside the ROI. The local UE can receive actual transmitted ROI corresponding to the encoded video from the remote UE via an RTP header extension. The encoded video within the ROI can be rendered for display at the local UE.

Claims

1. A local user equipment (UE) operable to perform video conferencing with a remote UE, the local UE having one or more processors configured to:

define a region of interest (ROI) within a field of view at the remote UE;

communicate the ROI to the remote UE via a real-time transport control protocol (RTCP) feedback message that directs the remote UE to capture video within the ROI and encode the video within the ROI;

receive the encoded video within the ROI from the remote UE, the encoded video including regions within the ROI and excluding regions outside the ROI, the encoded video including the regions within the ROI at an increased zoom level while substantially maintaining a defined quality level;

receive actual transmitted ROI corresponding to the encoded video from the remote UE via a real-time transport protocol (RTP) header extension, the RTP header extension being included in RTP packets that contain the encoded video; and

provide the encoded video within the ROI for rendering and display at the local UE.

2. The local UE of claim 1, wherein the one or more processors are further configured to receive a session description protocol (SDP) offer message from the remote UE indicating that the remote UE supports ROI-based video capturing, ROI-based video encoding and ROI-based video transmissions.

3. The local UE of claim 1, wherein the one or more processors are further configured to send a session description protocol (SDP) answer message acknowledging that the local UE supports ROI signaling via RTCP feedback messages, ROI-based video decoding and ROI-based video reception.

4. The local UE of claim 2, wherein the SDP offer message includes an attribute of a=rtcp-fb that is associated with a parameter 3gpp-roi-zoom, thereby indicating that the remote UE supports ROI-based video capturing, ROI-based video encoding and ROI-based video transmissions, wherein the remote UE is configured to process the RTCP feedback messages received from the local UE in order to obtain ROI information.

5. The local UE of claim 3, wherein the SDP answer message includes an attribute of a=rtcp-fb that is associated with a parameter 3gpp-roi-zoom, thereby indicating that the local UE supports ROI signaling via RTCP feedback messages, ROI-based video decoding and ROI-based video reception, wherein the local UE is configured to generate RTCP feedback messages and send the RTCP feedback messages to the remote UE in order to deliver ROI information.

6. The local UE of claim 2, wherein the SDP offer message includes an attribute a=extmap that is associated with a parameter urn:

3gpp-roi-actual, thereby indicating that the remote UE supports ROI-based video capturing, ROI-based video encoding and ROI-based video transmissions, wherein the remote UE is configured to signal the actual transmitted ROI corresponding to the encoded video in an RTP header extension received at the local UE.

7. The local UE of claim 3, wherein the SDP answer message includes an attribute a=extmap that is associated with a parameter urn:

3gpp-roi-actual, thereby indicating that the local UE supports ROI signaling via RTCP feedback messages, ROI-based video decoding and ROI-based video reception, wherein the local UE is configured to receive the actual transmitted ROI corresponding to the encoded video in an RTP header extension from the remote UE.

8. The local UE of claim 1, wherein the RTCP feedback message includes a position and a size that define the ROI within the field of view at the remote UE.

9. The local UE of claim 1, wherein the one or more processors are further configured to communicate the ROI to the remote UE via the RTCP feedback message, wherein the remote UE captures video within the ROI and only encodes the video within the ROI.

10. The local UE of claim 1, wherein the one or more processors are further configured to operate a video conferencing application with the remote UE that supports a ROI-based interactive zooming feature.

11. The local UE of claim 1, wherein the RTCP feedback message includes an indication of one or more predefined ROIs within the field of view at the remote UE.

12. The local UE of claim 1, wherein the RTP header extension includes a position and a size that define the actual transmitted ROI corresponding to the encoded video received from the remote UE.

13. A local user equipment (UE) operable to perform video conferencing with a remote UE, the local UE having one or more processors configured to:

define a region of interest (ROI) within a field of view at the remote UE;

communicate the ROI to the remote UE via a real-time transport control protocol (RTCP) feedback message that directs the remote UE to capture video within the ROI and encode the video within the ROI;

receive the encoded video within the ROI from the remote UE;

receive actual transmitted ROI corresponding to the encoded video from the remote UE via a real-time transport protocol (RTP) header extension carried in RTP packets containing the encoded video; and

render the encoded video within the ROI for display at the local UE.

14. The local UE of claim 13, wherein the RTCP feedback message that is communicated to the remote UE includes a position and a size that define the ROI, the position and the size defining the ROI,

wherein the position comprises two position values indicative of a corner position of the ROI within the field of view at the remote UE, wherein the two position values are normalized to indicate relative position in two dimensions,

wherein the size comprises two size values indicative of a size of the ROI within the field of view at the remote UE, wherein the two size values are normalized to indicate relative size in two dimensions.

15. The local UE of claim 13, wherein the RTCP feedback message includes an indication of one or more predefined ROIs within the field of view at the remote UE.

16. The local UE of claim 13, wherein:

the ROI includes a variable number of pixels; and

the encoded video received from the remote UE has a variable refresh rate that is inversely proportional to the number of pixels in the ROI.

17. The local UE of claim 13, wherein:

the ROI includes a fixed number of pixels; and

the encoded video received from the remote UE has a constant refresh rate.

18. The local UE of claim 13, wherein the RTP header extension that is communicated to the local UE includes a position and a size that define the actual transmitted ROI corresponding to the encoded video received from the remote UE, the position and the size defining the ROI,

wherein the position comprises two position values indicative of a corner position of the ROI within the field of view at the remote UE, wherein the two position values are normalized to indicate relative position in two dimensions,

wherein the size comprises two size values indicative of a size of the ROI within the field of view at the remote UE, wherein the two size values are normalized to indicate relative size in two dimensions.

19. A method for operating a video conferencing application at a local user equipment (UE) that supports an interactive zooming feature, the method comprising:

defining, at the local UE, a region of interest (ROI) within a field of view at a remote UE, using one or more processors of the local UE;

communicating the ROI to the remote UE via a real-time transport control protocol (RTCP) feedback message, wherein the RTCP feedback message directs the remote UE to capture video within the ROI and encode the video within the ROI, using the one or more processors of the local UE;

receiving the encoded video within the ROI from the remote UE, the encoded video including regions within the ROI and excluding regions outside the ROI, the encoded video including the regions within the ROI at an increased zoom level while substantially maintaining a defined quality level, using the one or more processors of the local UE;

receiving actual transmitted ROI corresponding to the encoded video from the remote UE via a real-time transport protocol (RTP) header extension carried in RTP packets containing the encoded video, using the one or more processors of the local UE; and

rendering the encoded video within the ROI for display at the local UE via the video conferencing application, using the one or more processors of the local UE.

20. The method of claim 19, further comprising receiving a session description protocol (SDP) offer message from the remote UE indicating that the remote UE supports ROI-based video capturing, ROI-based video encoding and ROI-based video transmissions, wherein the SDP offer message includes an attribute of a=rtcp-fb that is associated with a parameter 3gpp-roi-zoom, wherein the remote UE is configured to process the RTCP feedback message received from the local UE in order to obtain ROI information.

21. The method of claim 19, further comprising sending a session description protocol (SDP) answer message acknowledging that the local UE supports ROI signaling via RTCP feedback messages, ROI-based video decoding and ROI-based video reception, wherein the SDP answer message includes an attribute of a=rtcp-fb that is associated with a parameter 3gpp-roi-zoom, wherein the local UE is configured to generate the RTCP feedback messages and send the RTCP feedback messages to the remote UE in order to deliver ROI information.

22. The method of claim 19, wherein:

the SDP offer message includes an attribute a=extmap that is associated with a parameter urn;

3gpp-roi-actual, thereby indicating that the remote UE supports ROI-based video capturing, ROI-based video encoding and ROI-based video transmissions, wherein the remote UE is configured to signal the actual transmitted ROI corresponding to the encoded video in an RTP header extension received at the local UE; and

the SDP answer message includes an attribute a=extmap that is associated with a parameter urn;

3gpp-roi-actual, thereby indicating that the local UE supports ROI signaling via RTCP feedback messages, ROI-based video decoding and ROI-based video reception, wherein the local UE is configured to receive the actual transmitted ROI corresponding to the encoded video in an RTP header extension from the remote UE.

23. The method of claim 19, wherein:

the RTCP feedback message includes an indication of one or more pre-defined ROIs within the field of view at the remote UE; and

the RTCP feedback message includes a position and a size that define the ROI within the field of view at the remote UE.

24. The method of claim 23, wherein:

the position comprises two position values indicative of a corner position of the ROI within the field of view at the remote UE, wherein the two position values are normalized to indicate relative position in two dimensions; and

the size comprises two size values indicative of a size of the ROI within the field of view at the remote UE, wherein the two size values are normalized to indicate relative size in two dimensions.

25. The method of claim 19, wherein:

the RTP header extension includes a position and a size that define the actual transmitted ROI corresponding to the encoded video received from the remote UE;

the position comprises two position values indicative of a corner position of the ROI within the field of view at the remote UE, wherein the two position values are normalized to indicate relative position in two dimensions; and

the size comprises two size values indicative of a size of the ROI within the field of view at the remote UE, wherein the two size values are normalized to indicate relative size in two dimensions.