User input back channel for wireless displays

• US 9,787,725 B2
• Filed: 01/05/2012
• Issued: 10/10/2017
• Est. Priority Date: 01/21/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Huang, Xiaolong, Raveendran, Vijayalakshmi R., Wang, Xiaodong, Shaukat, Fawad
• Primary Examiner(s): Yao, Kwang B
• Assistant Examiner(s): NGO, NGUYEN HOANG
• Application Number:
  US13/344,253
• Publication Number:
  US 20130003621A1
• Time in Patent Office: 2,105 Days
• Field of Search: 370230, 370236, 370252, 370310, 370349, 370389, 370392, 370471, 370474
• CPC Class Codes: H04L 65/00 Network arrangements, protocols or services for supporting real-time applications in data packet communication real-time or near real-time messaging, e.g. instant messaging [IM] H04L51/04; selective video distribution H04N21/00
• CPC Class Codes: H04L 69/24 Negotiation of communication capabilities
• CPC Class Codes: H04W 99/00 Subject matter not provided for in other groups of this subclass

Abstract

As part of a communication session, a wireless source device can transmit audio and video data to a wireless sink device, and the wireless sink device can transmit user input data received at the wireless sink device back to the wireless source device. In this manner, a user of the wireless sink device can control the wireless source device and control the content that is being transmitted from the wireless source device to the wireless sink device. The user input data transmitted by the wireless sink device can be input data obtained at a third party device and forwarded to the wireless source device.

Claims

1. A method of transmitting user input data from a wireless sink device to a wireless source device, the method comprising:

outputting, by the wireless sink device and for display, video data received from the wireless source device;

obtaining, by the wireless sink device and from an external device, user input data captured while a particular frame of the video data is displayed;

generating, by the wireless sink device, a data packet header for a data packet, wherein the data packet header includes;

a three-bit version field that indicates a version of a communication protocol;

an eight-bit reserved field;

a four-bit input category field;

a sixteen-bit length field that indicates a length of the data packet; and

a timestamp flag that indicates whether or not the data packet header includes a sixteen-bit timestamp field;

including, by the wireless sink device and where the timestamp flag indicates that the data packet header includes the timestamp field, a representation of a timestamp of the particular frame of video data in the timestamp field of the data packet header;

generating, by the wireless sink device, payload data comprising the user input data;

generating, by the wireless sink device, the data packet comprising the data packet header and the payload data; and

transmitting, by the wireless sink device, the data packet to the wireless source device.

2. The method of claim 1, further comprising:

negotiating capabilities of the external device, with the wireless sink device, via a series of messages.

3. The method of claim 1, further comprising:

as part of establishing a communication session between the wireless source device and the wireless sink device, transmitting an identifier of the external device from the wireless sink device to the wireless source device.

4. The method of claim 1, further comprising:

as part of establishing a communication session between the wireless source device and the wireless sink device, receiving an identifier of the external device from the wireless source device.

5. The method of claim 1, wherein a value of the input category field is set to indicate the payload data comprises forwarded user input data.

6. The method of claim 1, wherein the payload data comprises an identifier of the external device.

7. The method of claim 6, wherein the identifier of the external device is selected from the group consisting of:

an IP address of the external device, and a domain name of the external device.

8. The method of claim 6, wherein the identifier is generated by the wireless source device and transmitted to the wireless sink device.

9. The method of claim 1, wherein the external device is another wireless sink device.

10. The method of claim 1, wherein the external device is an input device communicatively coupled to the wireless sink device.

11. The method of claim 1, wherein the data packet header is an application layer packet header.

12. The method of claim 1, wherein the data packet is to control audio data or video data of the wireless source device.

13. The method of claim 1, wherein the data packet is transmitted over TCP/IP.

14. The method of claim 1, further comprising:

receiving, by the wireless sink device and using Wi-Fi, the video data from the wireless source device.

15. The method of claim 14, wherein transmitting the data packet to the wireless source device comprises transmitting, by the wireless sink device and using Wi-Fi, the data packet to the wireless source device.

16. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

a memory;

one or more processors; and

at least one module executable by the one or more processors to;

output, for display, video data received from the wireless source device;

obtain, from an external device, user input data captured while a particular frame of the video data is displayed;

generate a data packet header for a data packet, wherein the data packet header includes;

a three-bit version field that indicates a version of a communication protocol;

an eight-bit reserved field;

a four-bit input category field;

a sixteen-bit length field that indicates a length of the data packet; and

a timestamp flag that indicates whether or not the data packet header includes a sixteen-bit timestamp field;

include, where the timestamp flag indicates that the data packet header includes the timestamp field, a representation of a timestamp of the particular frame of video data in the timestamp field of the data packet header;

generate payload data comprising the user input data;

generate the data packet comprising the data packet header and the payload data; and

transmit, to the wireless source device, the data packet.

17. The wireless sink device of claim 16, wherein the at least one module is further executable by the one or more processors to:

negotiate capabilities of the external device, with the wireless sink device, via a series of messages.

18. The wireless sink device of claim 16, wherein the at least one module is further executable by the one or more processors to:

as part of establishing a communication session between the wireless source device and the wireless sink device, transmit, to the wireless source device, an identifier of the external device.

19. The wireless sink device of claim 16, wherein the at least one module is further executable by the one or more processors to:

as part of establishing a communication session between the wireless source device and the wireless sink device, receive an identifier of the external device from the wireless source device.

20. The wireless sink device of claim 16, wherein a value of the input category field is set to indicate the payload data comprises forwarded user input data.

21. The wireless sink device of claim 16, wherein the payload data comprises an identifier of the external device.

22. The wireless sink device of claim 21, wherein the identifier of the external device is selected from the group consisting of:

an IP address of the external device, and a domain name of the external device.

23. The wireless sink device of claim 21, wherein the identifier is generated by the wireless source device and transmitted to the wireless sink device.

24. The wireless sink device of claim 16, wherein the external device is another wireless sink device.

25. The wireless sink device of claim 16, wherein the external device is an input device communicatively coupled to the wireless sink device.

26. The wireless sink device of claim 16, wherein the data packet header is an application layer packet header.

27. The wireless sink device of claim 16, wherein the data packet is to control audio data or video data of the wireless source device.

28. The wireless sink device of claim 16, wherein the data packet is transmitted over TCP/IP.

29. The wireless sink device of claim 16, wherein, to generate the data packet comprising the data packet header and the payload data, the at least one module is executable by the one or more processors to:

generate the data packet with a header that includes the data packet header and a payload that includes the payload data but does not include the input category field.

30. A non-transitory computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to perform a method of transmitting user input data from a wireless sink device to a wireless source device, the method comprising:

outputting, for display, video data received from the wireless source device;

obtaining, from an external device, user input data captured while a particular frame of the video data is displayed;

generating a data packet header for a data packet, wherein the data packet header includes;

a three-bit version field that indicates a version of a communication protocol;

an eight-bit reserved field;

a four-bit input category field;

a sixteen-bit length field that indicates a length of the data packet; and

a timestamp flag that indicates whether or not the data packet header includes a sixteen-bit timestamp field;

including, where the timestamp flag indicates that the data packet header includes the timestamp field, a representation of a timestamp of the particular frame of video data in the timestamp field of the data packet header;

generating payload data comprising the user input data;

generating the data packet comprising the data packet header and the payload data; and

transmitting the data packet to the wireless source device.

31. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

means for outputting, for display, video data received from the wireless source device;

means for obtaining, from an external device, user input data captured while a particular frame of the video data is displayed;

means for generating a data packet header, wherein the data packet header includes;

a three-bit version field that indicates a version of a communication protocol;

an eight-bit reserved field;

a four-bit input category field;

a sixteen-bit length field that indicates a length of the data packet; and

a timestamp flag that indicates whether or not the data packet header includes a sixteen-bit timestamp field;

means for including a representation of a timestamp of the particular frame of video data in the timestamp field of the data packet header in response to the timestamp flag indicating that the data packet header includes the timestamp field;

means for generating payload data comprising the user input data;

means for generating the data packet comprising the data packet header and the payload data; and

means for transmitting the data packet to the wireless source device.

32. A method of receiving user input data from a wireless sink device at a wireless source device, the method comprising:

outputting, by the wireless source device and to the wireless sink device, video data;

receiving, by the wireless source device, a data packet comprising a data packet header and payload data from the wireless sink device;

parsing the data packet header for;

a three-bit version field that indicates a version of a communication protocol;

a four-bit input category field that identifies an input category of a user input command represented by the payload data;

a sixteen-bit length field that indicates a length of the data packet; and

a timestamp flag that indicates whether or not the data packet header includes a timestamp field;

in response to the timestamp flag indicating that the data packet header does include the timestamp field, parsing the data packet header to identify, based on a representation of a timestamp included in the timestamp field, a frame of the video data that was displayed while the user input command was captured, wherein the data packet header includes an eight-bit reserved field; and

processing the payload data based on the identified frame of the video data.

33. The method of claim 32, further comprising:

negotiating capabilities of the external device, with the wireless source device, via a series of messages.

34. The method of claim 32, further comprising:

as part of establishing a communication session between the wireless source device and the wireless sink device, transmitting an identifier of the external device to the wireless source device.

35. The method of claim 32, further comprising:

as part of establishing a communication session between the wireless source device and the wireless sink device, receiving an identifier of the external device from the wireless source device.

36. The method of claim 32, wherein a value of the input category field is set to indicate the payload data comprises forwarded user input data.

37. The method of claim 35, wherein the identifier of the external device is selected from the group consisting of:

an IP address of the external device, and a domain name of the external device.

38. The method of claim 35, wherein the identifier is generated by the wireless source device and transmitted to the wireless sink device.

39. The method of claim 32, wherein the external device is another wireless sink device.

40. The method of claim 32, wherein the external device is an input device communicatively coupled to the wireless sink device.

41. The method of claim 32, wherein the data packet header is an application layer packet header.

42. The method of claim 32, wherein the data packet is to control audio data or video data of the wireless source device.

43. The method of claim 32, wherein the data packet is transmitted over TCP/IP.

44. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

a transport unit configured to receive a data packet comprising a data packet header and payload data from a wireless sink device;

a memory;

one or more processors;

at least one module executable by the one or more processors to;

output, to a wireless sink device, video data;

receive, from wireless sink device, a data packet comprising a data packet header and payload data;

parse the data packet header for;

a three-bit version field that indicates a version of a communication protocol;

a four-bit input category field that identifies an input category of a user input command represented by the payload data;

a sixteen-bit length field that indicates a length of the data packet; and

a timestamp flag that indicates whether or not the data packet header includes a timestamp field;

in response to the timestamp flag indicating that the data packet header does include the timestamp field, parse the data packet header to identify, based on a representation of a timestamp included in the timestamp field, a frame of the video data that was displayed while the user input command was captured, wherein the data packet header includes an eight-bit reserved field; and

process the payload data based on the identified frame of the video data.

45. The wireless source device of claim 44, wherein the at least one module is further executable by the one or more processors to:

negotiate capabilities of the external device, with the wireless source device, via a series of messages.

46. The wireless source device of claim 44, wherein the at least one module is further executable by the one or more processors to:

as part of establishing a communication session between the wireless source device and the wireless sink device, receive, from the wireless sink device, an identifier of the external device.

47. The wireless source device of claim 44, wherein the at least one module is further executable by the one or more processors to:

as part of establishing a communication session between the wireless source device and the wireless sink device, receive, from the wireless source device, an identifier of the external device.

48. The wireless source device of claim 47, wherein the identifier of the external device is selected from the group consisting of:

an IP address of the external device, and a domain name of the external device.

49. The wireless source device of claim 47, wherein the identifier is generated by the wireless source device and transmitted to the wireless sink device.

50. The wireless source device of claim 44, wherein a value of the input category field is set to indicate the payload data comprises forwarded user input data.

51. The wireless source device of claim 44, wherein the external device is another wireless sink device.

52. The wireless source device of claim 44, wherein the external device is an input device communicatively coupled to the wireless sink device.

53. The wireless source device of claim 44, wherein the data packet header is an application layer packet header.

54. The wireless source device of claim 44, wherein the data packet is to control audio data or video data of the wireless source device.

55. The wireless source device of claim 44, wherein the data packet is transmitted over TCP/IP.

56. A non-transitory computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to perform a method of receiving user input data from a wireless sink device at a wireless source device, the method comprising:

outputting, to the wireless sink device, video data;

receiving, from the wireless sink device, a data packet comprising a data packet header and payload data;

parsing the data packet header for;

a three-bit version field that indicates a version of a communication protocol;

a four-bit input category field that identifies an input category of a user input command represented by the payload data;

a sixteen-bit length field that indicates a length of the data packet; and

a timestamp flag that indicates whether or not the data packet header includes a timestamp field;

in response to the timestamp flag indicating that the data packet header does include the timestamp field, parsing the data packet header to identify, based on a representation of a timestamp included in the timestamp field, a frame of the video data that was displayed while the user input command was captured, wherein the data packet header includes an eight-bit reserved field; and

processing the payload data based on the identified frame of the video data.

57. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

means for outputting, by the wireless source device and to the wireless sink device, video data;

means for receiving, from the wireless sink device, a data packet comprising a data packet header and payload data;

means for parsing the data packet header for;

a three-bit version field that indicates a version of a communication protocol;

a four-bit input category field that identifies an input category of a user input command represented by the payload data;

a sixteen-bit length field that indicates a length of the data packet; and

a timestamp flag that indicates whether or not the data packet header includes a timestamp field;

means for parsing, in response to the timestamp flag indicating that the data packet header does include the timestamp field, the data packet header to identify, based on a representation of a timestamp included in the timestamp field, a frame of the video data that was displayed while the user input command was captured, wherein the data packet header includes an eight-bit reserved field; and

means for processing the payload data based on the identified frame of the video data.