User input back channel for wireless displays

• US 10,135,900 B2
• Filed: 01/05/2012
• Issued: 11/20/2018
• Est. Priority Date: 01/21/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Raveendran, Vijayalakshmi R., Huang, Xiaolong, Wang, Xiaodong, Shaukat, Fawad
• Primary Examiner(s): Yao, Kwang B
• Assistant Examiner(s): Patel, Hardikkumar D
• Application Number:
  US13/344,273
• Publication Number:
  US 20130003623A1
• Time in Patent Office: 2,511 Days
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/756 adapting media to device capabilities

Abstract

As part of a communication session, a wireless source device can transmit audio and video data to a wireless sink device, and the wireless sink device can transmit user inputs received at the wireless sink device back to the wireless source device. In this manner, a user of the wireless sink device can control the wireless source device and control the content that is being transmitted from the wireless source device to the wireless sink device. As part of transmitting user input to the source device, the sink device can generate a data packet with a data packet header that includes a timestamp flag indicating the presence of a timestamp field, and the source device can process the data packet based on a timestamp in the timestamp field.

Claims

1. A method of transmitting user input data from a wireless sink device to a wireless source device, the method comprising:

receiving, by the wireless sink device from the wireless source device, video data comprising a plurality of frames of video data, wherein each frame of the plurality of frames of video data comprises a respective timestamp;

in response to a user input event, obtaining user input data by the wireless sink device while displaying the plurality of frames of video data, wherein the user input data comprises coordinate data that identifies a location associated with the user input event, wherein the location is a location in a frame of video data being displayed by the wireless sink device when the user input data was obtained;

responsive to obtaining the user input data, generating a data packet header comprising a timestamp field that specifies one of the respective timestamps, the one of the respective timestamps being a timestamp of the frame video data being displayed by the wireless sink device when the user input data was obtained;

responsive to obtaining the user input data and generating the data packet header, generating a data packet comprising the data packet header and payload data, wherein the payload data comprises the coordinate data; and

transmitting the data packet from the wireless sink device to the wireless source device via a user input back channel (UIBC).

2. The method of claim 1, wherein the timestamp is based on multimedia data that was generated by the wireless source device and transmitted to the wireless sink device.

3. The method of claim 1, further comprising:

setting a timestamp flag of the data packet header to indicate that the timestamp field is present in the data packet header.

4. The method of claim 1, further comprising:

generating a second data packet header, wherein a timestamp flag of the second data packet header indicates a timestamp field for the second data packet header is not present in the second data packet header;

generating a second data packet comprising the second data packet header, wherein the second data packet header does not contain a timestamp field.

transmitting the second data packet to the wireless source device.

5. The method of claim 1, wherein the data packet header is an application layer packet header.

6. The method of claim 1, wherein the data packet is to control audio data or video data of the wireless source device.

7. The method of claim 1, wherein the data packet header further comprises a field to identify an input category of the user input data.

8. The method of claim 7, wherein the input category is selected from the group consisting of a generic command and an operating system (OS) specific command.

9. The method of claim 1, wherein the data packet header further comprises a field identifying a version of a protocol being used for communication.

10. The method of claim 1, wherein the data packet header further comprises a length field to indicate a length of the data packet.

11. The method of claim 10, wherein the length is identified in units of 16 bits.

12. The method of claim 1, wherein the data packet is transmitted using transmission control protocol and internet protocol (TCP/IP).

13. The method of claim 3, wherein the timestamp flag is a flag and the timestamp field is a 16-bit field.

14. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

a receiver unit configured to receive video data comprising a plurality of frames of video data, wherein each frame of the plurality of frames of video data comprises a respective timestamp;

a memory storing instructions;

one or more processors configured to execute the instructions, wherein upon execution of the instructions the one or more processors cause the wireless sink device to;

in response to a user input event, obtain user input data while displaying the frames of the video data received from the wireless source device, wherein the user input data comprises coordinate data that identifies a location associated with the user input event, wherein the location is a location in a frame of video data being displayed by the wireless sink device when the user input data was obtained;

responsive to obtaining the user input data, generate a data packet header comprising a timestamp field that specifies one of the respective timestamps, the one of the respective timestamps being a timestamp of the frame of video data being displayed by the wireless sink device when the user input data was obtained; and

;

responsive to obtaining the user input data and generating the data packet header, generate a data packet comprising the data packet header and payload data, wherein the payload data comprises the coordinate data; and

a transmitter unit to transmit the data packet to the wireless source device via a user input back channel (UIBC).

15. The wireless sink device of claim 14, wherein the timestamp is based on multimedia data that was generated by the wireless source device and transmitted to the wireless sink device.

16. The wireless sink device of claim 14, wherein the one or more processors further cause the wireless sink device to:

set a timestamp flag of the data packet header to indicate that the timestamp field is present in the data packet header.

17. The wireless sink device of claim 14, wherein the one or more processors further cause the wireless sink device to:

generate a second data packet header, wherein a timestamp flag of the second data packet header indicates a timestamp field for the second data packet header is not present in the second data packet header;

generate a second data packet comprising the second data packet header, wherein the second data packet header does not contain a timestamp field.

transmit the second data packet to the wireless source device.

18. The wireless sink device of claim 14, wherein the data packet header is an application layer packet header.

19. The wireless sink device of claim 14, wherein the data packet is to control audio data or video data of the wireless source device.

20. The wireless sink device of claim 14, wherein the data packet header further comprises a field to identify an input category of the user input data.

21. The wireless sink device of claim 20, wherein the input category is selected from the group consisting of a generic command and an operating system (OS) specific command.

22. The wireless sink device of claim 14, wherein the data packet header further comprises a field identifying a version of a protocol being used for communication.

23. The wireless sink device of claim 14, wherein the data packet header further comprises a length field to indicate a length of the data packet.

24. The wireless sink device of claim 23, wherein the length is identified in units of 16 bits.

25. The wireless sink device of claim 14, wherein the data packet is transmitted using transmission control protocol and internet protocol (TCP/IP).

26. The wireless sink device of claim 16, wherein the timestamp flag is a 1-bit flag and the timestamp field is a 16-bit field.

27. A non-transitory computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to:

receive a plurality of frames of video data from a wireless source device, wherein each frame of the plurality of frames of video data comprises a respective timestamp;

in response to a user input event, obtain user input data at a wireless sink device while causing the frames of the video data to be displayed, wherein the user input data comprises coordinate data that identifies a location associated with the user input event, wherein the location is a location in a frame of video data being displayed by the wireless sink device when the user input data was obtained;

responsive to obtaining the user input data and generating the data packet header, generate a data packet header comprising a timestamp field that specifies one of the respective timestamps, the one of the respective timestamps being a timestamp of the frame of video data being displayed by the wireless sink device when the user input data was obtained;

generate a data packet comprising the data packet header and payload data, wherein the payload data comprises the coordinate data; and

transmit the data packet from the wireless sink device to the wireless source device via a user input back channel (UIBC).

28. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

means for receiving a plurality of frames of video data from the wireless source device, wherein each frame of the plurality of frames of video data comprises a respective timestamp;

means for obtaining user input data, in response to a user input event, while displaying the frames of the video data, wherein the user input data comprises coordinate data that identifies a location associated with the user input event, wherein the location is a location in a frame of video data being displayed by the wireless sink device when the user input data was obtained;

means for generating a data packet header responsive to obtaining the user input data, the data packet header comprising a timestamp field that specifies one of the respective timestamps, the one of the respective timestamps being a timestamp of the frame of video data being displayed by the wireless sink device when the user input data was obtained;

means for generating a data packet responsive to obtaining the user input data and generating the data packet header, the data packet comprising the data packet header and payload data, wherein the payload data comprises the coordinate data; and

means for transmitting the data packet from the wireless sink device to the wireless source device via a user input back channel (UIBC).

29. A method of receiving user input data from a wireless sink device by a wireless source device, the method comprising:

transmitting, by the wireless source device to the wireless sink device, video data comprising a plurality of frames of video data, wherein each frame of the plurality of frames of video data comprises a respective timestamp;

receiving, by the wireless source device and from the wireless sink device via a user input back channel (UIBC), a data packet comprising a data packet header and payload data including the user input data, wherein the data packet header comprises a timestamp field that specifies one of the respective timestamps, the one of the respective timestamps being a timestamp of a frame of the video data being displayed by the wireless sink device when the user input data was obtained;

based on the timestamp in the timestamp field, identifying the frame of the video data being displayed by the wireless sink device when the user input data was obtained;

processing the data packet to determine coordinate data for the user input data; and

processing, by the wireless source device, the coordinate data based on content of the frame of the video data being displayed by the wireless sink device when the user input data was obtained.

30. The method of claim 29, further comprising:

generating, by the wireless source device, the timestamp based on multimedia data;

transmitting, by the wireless source device to the wireless sink device, the multimedia data and the timestamp, wherein the frame of video data comprises the multimedia data.

31. The method of claim 29, further comprising:

determining, by the wireless source device based on a timestamp flag of the data packet header, that the data packet header includes the timestamp.

32. The method of claim 31, wherein the timestamp flag is a 1-bit flag and the timestamp field is a 16-bit field.

33. The method of claim 29, wherein the timestamp is associated with a frame of video data being displayed at the wireless sink device when the user input data was obtained.

34. The method of claim 33, further comprising:

adding, by the wireless source device, the timestamp to the frame of video data; and

transmitting, by the wireless source device to the wireless sink device and prior to receiving the data packet, the frame of video data including the timestamp.

35. The method of claim 29, wherein processing the payload data based on the timestamp comprises comparing the timestamp of the timestamp field to a current timestamp for a current frame of video data being transmitted by the wireless source device to the wireless sink device.

36. The method of claim 35, wherein processing the payload data based on the timestamp further comprises performing a command described in the payload data in response to a time difference between the timestamp and the current timestamp being less than a threshold value.

37. The method of claim 35, wherein processing the payload data based on the timestamp further comprises not performing a command described in the payload data in response to a time difference between the timestamp and the current timestamp being greater than a threshold value.

38. The method of claim 29, wherein the data packet header is an application layer packet header.

39. The method of claim 29, wherein the data packet is to control audio data or video data of the wireless source device.

40. The method of claim 29, wherein the data packet header further comprises a field to identify an input category of the user input data.

41. The method of claim 40, wherein the input category is selected from the group consisting of a generic command and an operating system (OS) specific command.

42. The method of claim 29, wherein the data packet header further comprises a field identifying a version of a protocol being used for communication.

43. The method of claim 29, wherein the data packet header further comprises a length field to indicate a length of the data packet.

44. The method of claim 43, wherein the length is identified in units of 16 bits.

45. The method of claim 29, wherein the data packet is transmitted using transmission control protocol and internet protocol (TCP/IP).

46. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

a transmitter unit to transmit a plurality of frames of video data to the wireless sink device, wherein each frame of the plurality of frames of video data comprises a respective timestamp;

a receiver unit to receive, from the wireless sink device via a user input back channel (UIBC), a data packet comprising a data packet header and payload data including the user input data, wherein the data packet header comprises a timestamp field that specifies one of the respective timestamps, the one of the respective timestamps being a timestamp of a frame from the plurality of frames of the video data being displayed by the wireless sink device when the user input data was obtained;

a memory storing instructions; and

one or more processors configured to execute the instructions, wherein upon execution of the instructions the one or more processors cause the wireless source device to;

based on the timestamp in the timestamp field, identifying the frame of the video data being displayed by the wireless sink device when the user input data was obtained;

processing the data packet to determine coordinate data for the user input data; and

process the coordinate data based on content of the frame of the video data being displayed by the wireless sink device when the user input data was obtained.

48. The wireless source device of claim 46, wherein the one or more processors further cause the wireless source device to:

determine, based on a timestamp flag of the data packet header, that the data packet header includes the timestamp.

49. The wireless source device of claim 48, wherein the timestamp flag is a 1-bit flag and the timestamp field is a 16-bit field.

50. The wireless source device of claim 46, wherein the timestamp is associated with a frame of video data being displayed at the wireless sink device when the user input data was obtained.

51. The wireless source device of claim 50, wherein the one or more processors further cause the wireless source device to:

add the timestamp to the frame of video data; and

transmit, to the wireless sink device and prior to receiving the data packet, the frame of video data including the timestamp.

52. The wireless source device of claim 46, wherein to process the payload data based on the timestamp the one or more processors further cause the wireless source device to compare the timestamp of the timestamp field to a current timestamp for a current frame of video data being transmitted by the wireless source device to the wireless sink device.

53. The wireless source device of claim 52, wherein to process the payload data based on the timestamp the one or more processors further cause the wireless source device to perform a command described in the payload data in response to a time difference between the timestamp and the current timestamp being less than a threshold value.

54. The wireless source device of claim 52 wherein to process the payload data based on the timestamp the one or more processors further cause the wireless source device to not perform a command described in the payload data in response to a time difference between the timestamp and the current timestamp being greater than a threshold value.

55. The wireless source device of claim 46, wherein the data packet header is an application layer packet header.

56. The wireless source device of claim 46, wherein the data packet is to control audio data or video data of the wireless source device.

57. The wireless source device of claim 46, wherein the data packet header further comprises a field to identify an input category of the user input data.

58. The wireless source device of claim 57, wherein the input category is selected from the group consisting of a generic command and an operating system (OS) specific command.

59. The wireless source device of claim 46, wherein the data packet header further comprises a field identifying a version of a protocol being used for communication.

60. The wireless source device of claim 46, wherein the data packet header further comprises a length field to indicate a length of the data packet.

61. The wireless source device of claim 60, wherein the length is identified in units of 16 bits.

62. The wireless source device of claim 54, wherein the data packet is transmitted using transmission control protocol and internet protocol (TCP/IP).

47. The wireless source device 46, wherein the one or more processors further cause the wireless source device to:

generate the timestamp based on multimedia data;

transmit, to the wireless sink device, the multimedia data and the timestamp, wherein the frame of video data comprises the multimedia data.

63. A non-transitory computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to:

transmit a plurality of frames of video data to a wireless sink device, wherein each frame of the plurality of frames of video data comprises a respective timestamp;

receive, from the wireless sink device via a user input back channel (UIBC), a data packet comprising a data packet header and payload data including the user input data, wherein the data packet header comprises a timestamp field that specifies one of the respective timestamps, the one of the respective timestamps being a timestamp of a frame from the plurality of frames of the video data being displayed by a wireless sink device when the user input data was obtained;

based on the timestamp in the timestamp field, identifying the frame of the video data being displayed by the wireless sink device when the user input data was obtained;

processing the data packet to determine coordinate data for the user input data; and

process the coordinate data based on content of the frame of the video data being displayed by the wireless sink device when the user input data was obtained.

64. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

means for transmitting a plurality of frames of video data to the wireless sink device, wherein each frame of the plurality of frames of video data comprises a respective timestamp;

means for receiving, by the wireless source device and from the wireless sink device via a user input back channel (UIBC), a data packet comprising a data packet header and payload data including user input data, wherein the data packet header comprises a timestamp field that specifies one of the respective timestamps, the one of the respective timestamps being a timestamp of a frame from the plurality of frames of the video data being displayed by the wireless sink device when the user input data was obtained;

means for identifying the frame of the video data being displayed by the wireless sink device when the user input data was obtained based on the timestamp in the timestamp field;

means for processing the data packet to determine coordinate data for the user input data; and

means for processing the coordinate data based on content of the frame of the video data being displayed by the wireless sink device when the user input data was obtained.