User input back channel for wireless displays

• US 9,413,803 B2
• Filed: 01/05/2012
• Issued: 08/09/2016
• Est. Priority Date: 01/21/2011
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Huang, Xiaolong, Raveendran, Vijayalakshmi R., Wang, Xiaodong, Shaukat, Fawad
• Primary Examiner(s): Snyder, Steven
• Application Number:
  US13/344,394
• Publication Number:
  US 20130009873A1
• Time in Patent Office: 1,678 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355

Abstract

As part of a communication session, a wireless source device can transmit audio and video data to a wireless sink device, and the wireless sink device can transmit user input data received at the wireless sink device back to the wireless source device. In this manner, a user of the wireless sink device can control the wireless source device and control the content that is being transmitted from the wireless source device to the wireless sink device.

Claims

1. A method of transmitting user input data from a wireless sink device to a wireless source device, the method comprising:

obtaining the user input data at the wireless sink device;

generating payload data describing the user input data, wherein the payload data comprises an input type field to identify an input type;

generating a data packet comprising a data packet header and the payload data, wherein the data packet header comprises an input category field;

setting a value of the input category field to indicate that the payload data comprises a generic input comprising generic information elements defined in a protocol being executed by both the wireless source device and the wireless sink device, wherein the generic information elements are not formatted based on a device that obtained the user input data, wherein setting the value of the input category field comprises selecting the value from a list of supported input categories, the list of supported input categories comprising the value indicating the generic input and a value indicating a human interface device command, wherein the value indicating the human interface device command indicates the data packet comprises payload data formatted based on a type of human interface device; and

transmitting the data packet to the wireless source device.

2. The method of claim 1, wherein the input type is selected from a group consisting of pressing a mouse button, releasing a mouse button, touching a display, releasing a display, moving a mouse cursor, moving a touch on the display, pressing a key down, releasing a key, zooming, vertical scrolling, horizontal scrolling, and rotating.

3. The method of claim 1, wherein the payload data comprises a length field identifying the length of an input description.

4. The method of claim 3, wherein the length of the input description is identified in units of octets.

5. The method of claim 1, wherein the payload data comprises a describe field to describe details of the user input data.

6. The method of claim 1, wherein the payload data comprises a first input type field, a first length field, a first describe field, a second input type field, a second length field, and a second describe field, wherein a length value contained in the first length field identifies an end of the first describe field and a beginning of the second input type field.

7. The method of claim 1, wherein the payload data identifies a number of pointer events, and for each pointer event identifies an x-coordinate and a y-coordinate corresponding to coordinates at which the pointer event occurred.

8. The method of claim 7, wherein the x-coordinate and y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

9. The method of claim 1, wherein the payload data comprises an American Standard Code For Information Interchange (ASCII) code corresponding to a key down event.

10. The method of claim 1, wherein the payload data comprises an x-coordinate value, a y-coordinate value, and an indication of a zoom factor.

11. The method of claim 1, wherein the payload data comprises an indication of an amount to horizontally scroll.

12. The method of claim 11, wherein the amount to horizontally scroll is in units of pixels and is based on a display resolution of the wireless source device.

13. The method of claim 1, wherein the payload data comprises an indication of an amount to vertically scroll.

14. The method of claim 13, wherein the amount to vertically scroll is in units of pixels and is based on a display resolution of the wireless source device.

15. The method of claim 1, wherein the payload data comprises an indication of an amount of a rotation.

16. The method of claim 15, wherein the amount of the rotation is in units of radians.

17. The method of claim 1, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

18. The method of claim 1, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

19. The method of claim 1, wherein the data packet header is an application layer packet header.

20. The method of claim 1, wherein the data packet is to control audio data or video data of the wireless source device.

21. The method of claim 1, wherein the data packet is transmitted using a Transmission Control Protocol/Internet Protocol (TCP/IP) protocol.

22. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

an input device configured to obtain user input data at the wireless sink device;

one or more processors configured to;

generate payload data describing the user input data, wherein the payload data comprises an input type field to identify an input type;

generate a data packet comprising a data packet header and the payload data, wherein the data packet header comprises an input category field;

set a value of the input category field to indicate that the payload data comprises a generic input comprising generic information elements defined in a protocol being executed by both the wireless source device and the wireless sink device,

wherein the generic information elements are not formatted based on a device that obtained the user input data,

wherein to set the value of the input category field, the one or more processors are further configured to select the value from a list of supported input categories,

the list of supported input categories comprising the value indicating the generic input and a value indicating a human interface device command,

wherein the value indicating the human interface device command indicates the data packet comprises payload data formatted based on a type of human interface device; and

a transport unit to transmit the data packet to the wireless source device.

23. The wireless sink device of claim 22, wherein the input type is selected from a group consisting of pressing a mouse button, releasing a mouse button, touching a display, releasing a display, moving a mouse cursor, moving a touch on the display, pressing a key down, releasing a key, zooming, vertical scrolling, horizontal scrolling, and rotating.

24. The wireless sink device of claim 22, wherein the payload data comprises a length field identifying the length of an input description.

25. The wireless sink device of claim 24, wherein the length of the input description is identified in units of octets.

26. The wireless sink device of claim 22, wherein the payload data comprises a describe field to describe details of the user input data.

27. The wireless sink device of claim 22, wherein the payload data comprises a first input type field, a first length field, a first describe field, a second input type field, a second length field, and a second describe field, wherein a length value contained in the first length field identifies an end of the first describe field and a beginning of the second input type field.

28. The wireless sink device of claim 22, wherein the payload data identifies a number of pointer events, and for each pointer event identifies an x-coordinate and a y-coordinate corresponding to coordinates at which the pointer event occurred.

29. The wireless sink device of claim 28, wherein the x-coordinate and y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

30. The wireless sink device of claim 22, wherein the payload data comprises an American Standard Code For Information Interchange (ASCII) code corresponding to a key down event.

31. The wireless sink device of claim 22, wherein the payload data comprises an x-coordinate value, a y-coordinate value, and an indication of a zoom factor.

32. The wireless sink device of claim 22, wherein the payload data comprises an indication of an amount to horizontally scroll.

33. The wireless sink device of claim 32, wherein the amount to horizontally scroll is in units of pixels and is based on a display resolution of the wireless source device.

34. The wireless sink device of claim 22, wherein the payload data comprises an indication of an amount to vertically scroll.

35. The wireless sink device of claim 34, wherein the amount to vertically scroll is in units of pixels and is based on a display resolution of the wireless source device.

36. The wireless sink device of claim 22, wherein the payload data comprises an indication of an amount of a rotation.

37. The wireless sink device of claim 36, wherein the amount of the rotation is in units of radians.

38. The wireless sink device of claim 22, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

39. The wireless sink device of claim 22, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

40. The wireless sink device of claim 22, wherein the data packet header is an application layer packet header.

41. The wireless sink device of claim 22, wherein the data packet is to control audio data or video data of the wireless source device.

42. The wireless sink device of claim 22, wherein the data packet is transmitted using a Transmission Control Protocol/Internet Protocol (TCP/IP) protocol.

43. A computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to:

obtain user input data at the wireless sink device;

generate payload data describing the user input data, wherein the payload data comprises an input type field to identify an input type;

generate a data packet comprising a data packet header and the payload data, wherein the data packet header comprises an input category field;

set a value of the input category field to indicate that the payload data comprises a generic input comprising generic information elements defined in a protocol being executed by both the wireless source device and the wireless sink device, wherein the generic information elements are not formatted based on a device that obtained the user input data, wherein to set the value of the input category field, the instructions cause the one or more processors to select the value from a list of supported input categories, the list of supported input categories comprising the value indicating the generic input and a value indicating a human interface device command, wherein the value indicating the human interface device command indicates the data packet comprises payload data formatted based on a type of human interface device; and

transmit the data packet to the wireless source device.

44. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

means for obtaining user input data at the wireless sink device;

means for generating payload data describing the user input data, wherein the payload data comprises an input type field to identify an input type;

means for generating a data packet comprising a data packet header and the payload data, wherein the data packet header comprises an input category field;

means for setting a value of the input category field to indicate that the payload data comprises a generic input comprising generic information elements defined in a protocol being executed by both the wireless source device and the wireless sink device, wherein the generic information elements are not formatted based on a device that obtained the user input data, wherein the means for setting the value of the input category field comprises means for selecting the value from a list of supported input categories, the list of supported input categories comprising the value indicating the generic input and a value indicating a human interface device command, wherein the value indicating the human interface device command indicates the data packet comprises payload data formatted based on a type of human interface device; and

means for transmitting the data packet to the wireless source device.

45. A method of receiving input data from a wireless sink device at a wireless source device, the method comprising:

receiving from the wireless sink device a data packet, wherein the data packet comprises a data packet header and payload data, wherein the payload data comprises data describing details of a user input;

parsing an input category field of the data packet header to determine that the payload data comprises a generic input, the generic input comprising generic information elements defined in a protocol being executed by both the wireless source device and the wireless sink device, wherein the generic information elements are not formatted based on a device that obtained the user input data, wherein a value for the input category field is selected from a list of supported input categories comprising a value indicating the generic input and a value indicating a human interface device command, wherein the value indicating the human interface device command indicates the data packet comprises payload data formatted based on a type of human interface device;

in response to determining that the payload data comprises the generic information elements, parsing the data packet to determine an input type value in an input type field in the payload data;

processing the data describing details of the user input based on the input type value.

46. The method of claim 45, wherein the input type is selected from a group consisting of pressing a mouse button, releasing a mouse button, touching a display, releasing a display, moving a mouse cursor, moving a touch on the display, pressing a key down, releasing a key, zooming, vertical scrolling, horizontal scrolling, and rotating.

47. The method of claim 45, wherein the payload data comprises a length field identifying the length of an input description.

48. The method of claim 47, wherein the length of the input description is identified in units of octets.

49. The method of claim 45, wherein the payload data comprises a describe field comprising the data describing the details of the user input.

50. The method of claim 45, wherein the payload data comprises a first input type field, a first length field, a first describe field, a second input type field, a second length field, and a second describe field, wherein a length value contained in the first length field identifies an end of the first describe field and a beginning of the second input type field.

51. The method of claim 45, wherein the payload data identifies a number of pointer events, and for each pointer event identifies an x-coordinate and a y-coordinate corresponding to coordinates at which the pointer event occurred.

52. The method of claim 51, wherein the x-coordinate and y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

53. The method of claim 45, wherein the payload data comprises an American Standard Code For Information Interchange (ASCII) code corresponding to a key down event.

54. The method of claim 45, wherein the payload data comprises an x-coordinate value, a y-coordinate value, and an indication of a zoom factor.

55. The method of claim 45, wherein the payload data comprises an indication of an amount to horizontally scroll.

56. The method of claim 55, wherein the amount to horizontally scroll is in units of pixels and is based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

57. The method of claim 45, wherein the payload data comprises an indication of an amount to vertically scroll.

58. The method of claim 57, wherein the amount to vertically scroll is in units of pixels and is based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

59. The method of claim 45, wherein the payload data comprises an indication of an amount of a rotation.

60. The method of claim 59, wherein the amount of the rotation is in units of radians.

61. The method of claim 45, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

62. The method of claim 45, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

63. The method of claim 45, wherein the data packet header is an application layer packet header.

64. The method of claim 45, wherein the data packet is to control audio data or video data of the wireless source device.

65. The method of claim 45, wherein the data packet is transmitted using a Transmission Control Protocol/Internet Protocol (TCP/IP) protocol.

66. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

a transport unit to receive from the wireless sink device a data packet, wherein the data packet comprises a data packet header and payload data, wherein the payload data comprises data describing details of a user input;

a memory to store the data packet;

one or more processors configured to;

parse an input category field of the data packet header to determine that the payload data comprises a generic input comprising generic information elements defined in a protocol being executed by both the wireless source device and the wireless sink device, wherein the generic information elements are not formatted based on a device that obtained the user input data, wherein a value for the input category field is selected from a list of supported input categories comprising a value indicating the generic input and a value indicating a human interface device command, wherein the value indicating the human interface device command indicates the data packet comprises payload data formatted based on a type of human interface device;

in response to determining that the payload data comprises the generic information elements, parse the data packet to determine an input type value in an input type field in the payload data;

process the data describing details of the user input based on the input type value.

67. The wireless source device of claim 66, wherein the input type is selected from a group consisting of pressing a mouse button, releasing a mouse button, touching a display, releasing a display, moving a mouse cursor, moving a touch on the display, pressing a key down, releasing a key, zooming, vertical scrolling, horizontal scrolling, and rotating.

68. The wireless source device of claim 66, wherein the payload data comprises a length field identifying the length of an input description.

69. The wireless source device of claim 68, wherein the length of the input description is identified in units of octets.

70. The wireless source device of claim 66 wherein the payload data comprises a describe field comprising the data describing the details of the user input.

71. The wireless source device of claim 66, wherein the payload data comprises a first input type field, a first length field, a first describe field, a second input type field, a second length field, and a second describe field, wherein a length value contained in the first length field identifies an end of the first describe field and a beginning of the second input type field.

72. The wireless source device of claim 66, wherein the payload data identifies a number of pointer events, and for each pointer event identifies an x-coordinate and a y-coordinate corresponding to coordinates at which the pointer event occurred.

73. The wireless source device of claim 72, wherein the x-coordinate and y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

74. The wireless source device of claim 66, wherein the payload data comprises an American Standard Code For Information Interchange (ASCII) code corresponding to a key down event.

75. The wireless source device of claim 66, wherein the payload data comprises an x-coordinate value, a y-coordinate value, and an indication of a zoom factor.

76. The wireless source device of claim 66, wherein the payload data comprises an indication of an amount to horizontally scroll.

77. The wireless source device of claim 76, wherein the amount to horizontally scroll is in units of pixels and is based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

78. The wireless source device of claim 66, wherein the payload data comprises an indication of an amount to vertically scroll.

79. The wireless source device of claim 78, wherein the amount to vertically scroll is in units of pixels and is based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

80. The wireless source device of claim 66, wherein the payload data comprises an indication of an amount of a rotation.

81. The wireless source device of claim 80, wherein the amount of the rotation is in units of radians.

82. The wireless source device of claim 66, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

83. The wireless source device of claim 66, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

84. The wireless source device of claim 66, wherein the data packet header is an application layer packet header.

85. The wireless source device of claim 66, wherein the data packet is to control audio data or video data of the wireless source device.

86. The wireless source device of claim 66, wherein the data packet is transmitted using a Transmission Control Protocol/Internet Protocol (TCP/IP) protocol.

87. A computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to:

receive from the wireless sink device a data packet, wherein the data packet comprises a data packet header and payload data, wherein the payload data comprises data describing details of a user input;

parse an input category field of the data packet header to determine that the payload data comprises a generic input comprising generic information elements defined in a protocol being executed by both the wireless source device and the wireless sink device, wherein the generic information elements are not formatted based on a device that obtained the user input data, wherein a value for the input category field is selected from a list of supported input categories comprising a value indicating the generic input and a value indicating a human interface device command, wherein the value indicating the human interface device command indicates the data packet comprises payload data formatted based on a type of human interface device;

in response to determining that the payload data comprises the generic information elements, parse the data packet to determine an input type value in an input type field in the payload data;

process the data describing details of the user input based on the input type value.

88. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

means for receiving from the wireless sink device a data packet, wherein the data packet comprises a data packet header and payload data, wherein the payload data comprises data describing details of a user input;

means for parsing an input category field of the data packet header to determine that the payload data comprises a generic input comprising generic information elements defined in a protocol being executed by both the wireless source device and the wireless sink device, wherein the generic information elements are not formatted based on a device that obtained the user input data, wherein a value for the input category field is selected from a list of supported input categories comprising a value indicating the generic input and a value indicating a human interface device command, wherein the value indicating the human interface device command indicates the data packet comprises payload data formatted based on a type of human interface device;

means for parsing the data packet to determine an input type value in an input type field in the payload data in response to determining that the payload data comprises the generic information elements; and

means for processing the data describing details of the user input based on the input type value.