User input back channel for wireless displays

• US 8,964,783 B2
• Filed: 01/05/2012
• Issued: 02/24/2015
• Est. Priority Date: 01/21/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Huang, Xiaolong, Raveendran, Vijayalakshmi R., Wang, Xiaodong, Shaukat, Fawad
• Primary Examiner(s): Yao, Kwang B
• Assistant Examiner(s): NGO, NGUYEN HOANG
• Application Number:
  US13/344,262
• Publication Number:
  US 20130003622A1
• Time in Patent Office: 1,146 Days
• Field of Search: 370/230, 370/236, 370/252, 370/310, 370/349, 370/389, 370/392, 370/471, 370/474
• US Class Current: 370/474
• CPC Class Codes: H04L 65/1095 Inter-network session transfer or sharing
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]

Abstract

As part of a communication session, a wireless source device can transmit audio and video data to a wireless sink device, and the wireless sink device can transmit user inputs received at the wireless sink device back to the wireless source device. In this manner, a user of the wireless sink device can control the wireless source device and control the content that is being transmitted from the wireless source device to the wireless sink device. As part of transmitting user input to the source device, the sink device can generate a data packet that identifies a user input category for the user input, and the source device can process the data packet based on the user input category.

Claims

1. A method of transmitting user input data from a wireless sink device to a wireless source device, the method comprising:

performing capability negotiation to determine one or more categories of user input supported by the wireless source device, wherein performing the capability negotiation comprises;

transmitting, by the wireless sink device and to the wireless source device a real-time streaming protocol (RTSP) message that indicates a first set of one or more categories of user input supported by the sink device;

receiving, by the wireless sink device and from the wireless source device, an RTSP message that indicates a second set of one or more user input categories supported by the sink device and with which the wireless source device is compatible; and

transmitting, by the wireless sink device and to the wireless source device, a message confirming one of the second set of supported user input categories is enabled;

after performing the capability negotiation, obtaining user input data at the wireless sink device;

generating a data packet header, wherein the data packet header comprises a field to identify the enabled input category of the user input data;

generating a data packet comprising the data packet header and payload data, wherein the enabled input category identifies how the payload data is formatted; and

transmitting the data packet to the wireless source device.

2. The method of claim 1, further comprising:

categorizing the user input data to determine the input category of the user input data.

3. The method of claim 1, wherein the payload data describes the user input data.

4. The method of claim 1, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

5. The method of claim 1, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

6. The method of claim 1, wherein the data packet header is an application layer packet header.

7. The method of claim 1, wherein the data packet is to control audio data or video data of the wireless source device.

8. The method of claim 1, wherein the input category is a generic command.

9. The method of claim 1, wherein the input category is selected from the group consisting of a forwarded command and an operating system (OS) specific command.

10. The method of claim 1, wherein the data packet header further comprises a field identifying if an optional timestamp entry is present.

11. The method of claim 1, wherein the data packet header further comprises a field for a timestamp entry.

12. The method of claim 1, wherein the data packet header further comprises a field identifying a version of a protocol being used for communication.

13. The method of claim 1, wherein the data packet is transmitted over TCP/IP.

14. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

a memory storing instructions;

one or more processors configured to execute the instructions, wherein upon execution of the instructions the one or more processors cause;

performing capability negotiation to determine one or more categories of user input supported by the wireless source device, wherein to perform the capability negotiation, the one or more processors further cause;

transmitting, by the wireless sink device and to the wireless source device a real-time streaming protocol (RTSP) message that indicates a first set of one or more user input supported by the sink device;

receiving, by the wireless sink device and from the wireless source device, an RTSP message that indicates a second set of user input categories supported by the sink device and with which the wireless source device is compatible; and

transmitting, by the wireless sink device, and to the wireless source device, a message confirming one of the second set of supported user input categories is enabled;

after performing the capability negotiation, obtaining user input data at the wireless sink device;

generating a data packet header, wherein the data packet header comprises a field to identify the enabled input category of the user input data;

generating a data packet comprising the data packet header and payload data, wherein the enabled input category identifies how the payload data is formatted; and

a transport unit to transmit the data packet to the wireless source device.

15. The wireless sink device of claim 14, wherein upon execution of the instructions the one or more processors further cause:

categorizing the user input data to determine the input category of the user input data.

16. The wireless sink device of claim 14, wherein the payload data describes the user input data.

17. The wireless sink device of claim 14, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

18. The wireless sink device of claim 14, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

19. The wireless sink device of claim 14, wherein the data packet header is an application layer packet header.

20. The wireless sink device of claim 14, wherein the data packet is to control audio data or video data of the wireless source device.

21. The wireless sink device of claim 14, wherein the input category is a generic command.

22. The wireless sink device of claim 14, wherein the input category is selected from the group consisting of a forwarded command and an operating system (OS) specific command.

23. The wireless sink device of claim 14, wherein the data packet header further comprises a field identifying if an optional timestamp entry is present.

24. The wireless sink device of claim 14, wherein the data packet header further comprises a field for a timestamp entry.

25. The wireless sink device of claim 14, wherein the data packet header further comprises a field identifying a version of a protocol being used for communication.

26. The wireless sink device of claim 14, wherein the data packet is transmitted over TCP/IP.

27. A non-transitory computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to transmit user input data from a wireless sink device to a wireless source device, wherein the instructions that cause the one or more processors to transmit the user input data from the wireless sink device to the wireless source device further cause the one or more processors to:

perform capability negotiation to determine one or more categories of user input supported by the wireless source device, wherein the instructions that cause the at one or more processors to perform the capability negotiation further cause the one or more processors to;

transmit, by the wireless sink device and to the wireless source device a real-time streaming protocol (RTSP) message that indicates a first set of one or more user input supported by the sink device;

receive, by the wireless sink device and from the wireless source device, an RTSP message that indicates a second set of one or more user input categories supported by the sink device and with which the wireless source device is compatible; and

transmit, by the wireless sink device and to the wireless source device, a message confirming one of the second set of supported user input categories is enabled;

after performing the capability negotiation obtain user input data at the wireless sink device;

generate a data packet header, wherein the data packet header comprises a field to identify the enabled input category of the user input data;

transmit a data packet comprising the data packet header and payload data, wherein the enabled input category identifies how the payload data is formatted; and

transmit the data packet to the wireless source device.

28. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

means for performing capability negotiation to determine supported categories user input of the wireless source device before obtaining user input data at the wireless sink device, wherein the means for performing the capability negotiation comprises;

means for transmitting, by the wireless sink device and to the wireless source device a real-time streaming protocol (RTSP) message that indicates a first set of one or more user input categories supported by the sink device;

means for receiving, by the wireless sink device and from the wireless source device, an RTSP message that indicates a second set of one or more user input categories supported by the sink device and with which the wireless source device is compatible; and

means for transmitting, by the wireless sink device and to the wireless source device, a message confirming one of the second set of supported user input categories is enabled;

means for obtaining user input data at the wireless sink device after performing the capability negotiation;

means for generating a data packet header, wherein the data packet header comprises a field to identify the enabled input category of the user input data;

means for generating a data packet comprising the data packet header and payload data, wherein the enabled input category identifies how the payload data is formatted; and

means for transmitting the data packet to the wireless source device.

29. A method of receiving user input data from a wireless sink device at a wireless source device, the method comprising:

performing capability negotiation to determine one or more categories of user input supported by the wireless source device, wherein performing the capability negotiation comprises;

receiving, by the wireless source device and from the wireless sink device, a real-time streaming protocol (RTSP) message that indicates a first set of one or more user input categories supported by the sink device;

transmitting, by the wireless source device and to the wireless sink device, an RTSP message that indicates a second set of one or more user input categories supported by the sink device and with which the wireless source device is compatible; and

receiving, by the wireless source device and from the wireless sink device, a message confirming one of the second set of supported user input categories is enabled;

after performing the capability negotiation, receiving a data packet comprising a data packet header and payload data;

parsing the data packet header to determine the enabled input category associated with user input data contained in the payload data; and

processing the payload data based on the determined input category, wherein the input category identifies how the payload data is formatted.

30. The method of claim 29, wherein the data packet header is an application layer packet header.

31. The method of claim 29, wherein the data packet is to control audio data or video data of the wireless source device.

32. The method of claim 29, wherein the input category is selected from the group consisting of a forwarded command and an operating system (OS) specific command.

33. The method of claim 29, wherein the input category is a generic command.

34. The method of claim 29, further comprising:

parsing the data packet header to determine if an optional timestamp entry is present.

35. The method of claim 29, further comprising:

parsing the data packet header to determine a timestamp identifying a frame of video data being displayed at the time the user input data was obtained by the wireless sink device; and

processing the payload data based on the determined input category.

36. The method of claim 29, further comprising:

parsing the data packet header to determine a version of a protocol being used for communication.

37. The method of claim 29, wherein the data packet is transmitted over TCP/IP.

38. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

a transport unit to receive a data packet comprising a data packet header and payload data;

a memory storing instructions;

one or more processors configured to execute the instructions, wherein upon execution of the instructions the one or more processors cause;

performing capability negotiation to determine one or more categories of user input supported by the wireless source device, wherein to perform the capability negotiation, the one or more processors further cause;

receiving, by the wireless source device and from the wireless sink device, a real-time streaming protocol (RTSP) message that indicates a first set of one or more user input categories supported by the sink device;

transmitting, by the wireless source device and to the wireless sink device, an RTSP message that indicates a second set of one or more user input supported by of the sink device and with which the wireless source device is compatible; and

receiving, by the wireless source device and from the wireless sink device, a message confirming one of the second set of supported user input categories is enabled;

after performing the capability negotiation, parsing the data packet header to determine an input category associated with user input data contained in the payload data; and

processing the payload data based on the determined input category, wherein the input category identifies how the payload data is formatted.

39. The wireless source device of claim 38, wherein the data packet header is an application layer packet header.

40. The wireless source device of claim 38, wherein the data packet is to control audio data or video data of the wireless source device.

41. The wireless source device of claim 38, wherein the input category is selected from the group consisting of a forwarded command and an operating system (OS) specific command.

42. The wireless source device of claim 38, wherein the input category is a generic command.

43. The wireless source device of claim 38, wherein upon execution of the instructions the one or more processors further cause:

parsing the data packet header to determine if an optional timestamp entry is present.

44. The wireless source device of claim 38, wherein upon execution of the instructions the one or more processors further cause:

parsing the data packet header to determine a timestamp identifying a frame of video data being displayed at the time the user input data was obtained by the wireless sink device; and

processing the payload data based on the determined input category.

45. The wireless source device of claim 38, wherein upon execution of the instructions the one or more processors further cause:

parsing the data packet header to determine a version of a protocol being used for communication.

46. The wireless source device of claim 38, wherein the data packet is transmitted over TCP/IP.

47. A non-transitory computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to receive user input data from a wireless source device at a wireless sink device, wherein the instructions that cause the one or more processors to receive the user input data from the wireless source device further cause the one or more processors to:

perform capability negotiation to determine one or more categories of user input supported by the wireless source device, wherein the instructions that cause the one or more processors to perform the capability negotiation further cause the one or more processors to;

receive, by the wireless source device and from the wireless sink device, a real-time streaming protocol (RTSP) message that indicates a first set of one or more user input categories supported by the sink device;

transmit, by the wireless source device and to the wireless sink device, an RTSP message that indicates a second set of one or more user input categories supported by the sink device and with which the wireless source device is compatible;

receive, by the wireless source device and from the wireless sink device, a message confirming one of the second set of supported user input categories is enabled;

after performing the capability negotiation, receive a data packet comprising a data packet header and payload data;

parse the data packet header to determine the enabled input category associated with user input data contained in the payload data; and

process the payload data based on the determined input category, wherein the input category identifies how the payload data is formatted.

48. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

means for performing capability negotiation to determine one or more categories of user input supported by the wireless source device, wherein the means for performing the capability negotiation comprises;

means for receiving, by the wireless source device and from the wireless sink device, a real-time streaming protocol (RTSP) message that indicates a first set of one or more user input supported by the sink device;

means for transmitting, by the wireless source device and to the wireless sink device, an RTSP message that indicates a second set of one or more user input categories supported by the sink device and with which the wireless source device is compatible; and

means for receiving, by the wireless source device and from the wireless sink device, a message confirming one of the second set of supported user input data categories is enabled;

means for receiving a data packet comprising a data packet header and payload data after performing the capability negotiation;

means for parsing the data packet header to determine the enabled input category associated with user input data contained in the payload data; and

means for processing the payload data based on the determined input category, wherein the input category identifies how the payload data is formatted.