USER INPUT BACK CHANNEL FOR WIRELESS DISPLAYS

• US 20130009873A1
• Filed: 01/05/2012
• Published: 01/10/2013
• Est. Priority Date: 01/21/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Huang, Xiaolong, Raveendran, Vijayalakshmi R., Wang, Xiaodong, Shaukat, Fawad
• Granted Patent: US 9,413,803 B2
• Time in Patent Office: Days
• US Class Current: 345/163
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355

Abstract

As part of a communication session, a wireless source device can transmit audio and video data to a wireless sink device, and the wireless sink device can transmit user input data received at the wireless sink device back to the wireless source device. In this manner, a user of the wireless sink device can control the wireless source device and control the content that is being transmitted from the wireless source device to the wireless sink device.

Claims

1. A method of transmitting user input data from a wireless sink device to a wireless source device, the method comprising:

obtaining user input data at the wireless sink device;

generating payload data describing the user input data;

generating a data packet comprising a data packet header and the payload data;

transmitting the data packet to the wireless source device.

2. The method of claim 1, wherein the payload data comprises an input type field to identify an input type.

3. The method of claim 2, wherein the input type is selected from a group consisting of pressing a mouse button, releasing a mouse button, touching a display, releasing a display, moving a mouse cursor, moving a touch on the display, pressing a key down, releasing a key, zooming, vertical scrolling, horizontal scrolling, and rotating.

4. The method of claim 1, wherein the payload data comprises a length field identifying the length of an input description.

5. The method of claim 4, wherein the length of the input description is identified in units of octets.

6. The method of claim 1, wherein the payload data comprises a describe field to describe details of the user input data.

7. The method of claim 1, wherein the payload data comprises a first input type field, a first length field, a first describe field, a second input type field, a second length field, and a second describe field, wherein a length value contained in the first length field identifies an end of the first describe field and a beginning of the second input type field.

8. The method of claim 1, wherein the payload data identifies a number of pointer events, and for each pointer event identifies an x-coordinate and a y-coordinate corresponding to coordinates at which the pointer event occurred.

9. The method of claim 8, wherein the x-coordinate and y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

10. The method of claim 1, wherein the payload data comprises an ASCII code corresponding to a key down event.

11. The method of claim 1, wherein the payload data comprises an ASCII code corresponding to a key down event.

12. The method of claim 1, wherein the payload data comprises an x-coordinate value, a y-coordinate value, and an indication of a zoom factor.

13. The method of claim 1, wherein the payload data comprises an indication of an amount to horizontally scroll.

14. The method of claim 13, wherein the amount to horizontally scroll is in units of pixels and is based on a display resolution of the wireless source device.

15. The method of claim 1, wherein the payload data comprises an indication of an amount to vertically scroll.

16. The method of claim 15, wherein the amount to vertically scroll is in units of pixels and is based on a display resolution of the wireless source device.

17. The method of claim 1, wherein the payload data comprises an indication of an amount of a rotation.

18. The method of claim 17, wherein the amount of the rotation is in units of radians.

19. The method of claim 1, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

20. The method of claim 1, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

21. The method of claim 1, wherein the data packet header is an application layer packet header.

22. The method of claim 1, wherein the data packet is to control audio data or video data of the wireless source device.

23. The method of claim 1, wherein the data packet is transmitted over TCP/IP.

24. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

a memory storing instructions;

one or more processors configured to execute the instructions, wherein upon execution of the instructions the one or more processors cause;

obtaining user input data at the wireless sink device;

generating payload data describing the user input data;

generating a data packet comprising a data packet header and the payload data;

a transport unit to transmit the data packet to the wireless source device.

25. The wireless sink device of claim 24, wherein the payload data comprises an input type field to identify an input type.

26. The wireless sink device of claim 25, wherein the input type is selected from a group consisting of pressing a mouse button, releasing a mouse button, touching a display, releasing a display, moving a mouse cursor, moving a touch on the display, pressing a key down, releasing a key, zooming, vertical scrolling, horizontal scrolling, and rotating.

27. The wireless sink device of claim 24, wherein the payload data comprises a length field identifying the length of an input description.

28. The wireless sink device of claim 27, wherein the length of the input description is identified in units of octets.

29. The wireless sink device of claim 24, wherein the payload data comprises a describe field to describe details of the user input data.

30. The wireless sink device of claim 24, wherein the payload data comprises a first input type field, a first length field, a first describe field, a second input type field, a second length field, and a second describe field, wherein a length value contained in the first length field identifies an end of the first describe field and a beginning of the second input type field.

31. The wireless sink device of claim 24, wherein the payload data identifies a number of pointer events, and for each pointer event identifies an x-coordinate and a y-coordinate corresponding to coordinates at which the pointer event occurred.

32. The wireless sink device of claim 31, wherein the x-coordinate and y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

33. The wireless sink device of claim 24, wherein the payload data comprises an ASCII code corresponding to a key down event.

34. The wireless sink device of claim 24, wherein the payload data comprises an ASCII code corresponding to a key down event.

35. The wireless sink device of claim 24, wherein the payload data comprises an x-coordinate value, a y-coordinate value, and an indication of a zoom factor.

36. The wireless sink device of claim 24, wherein the payload data comprises an indication of an amount to horizontally scroll.

37. The wireless sink device of claim 36, wherein the amount to horizontally scroll is in units of pixels and is based on a display resolution of the wireless source device.

38. The wireless sink device of claim 24, wherein the payload data comprises an indication of an amount to vertically scroll.

39. The wireless sink device of claim 38, wherein the amount to vertically scroll is in units of pixels and is based on a display resolution of the wireless source device.

40. The wireless sink device of claim 24, wherein the payload data comprises an indication of an amount of a rotation.

41. The wireless sink device of claim 40, wherein the amount of the rotation is in units of radians.

42. The wireless sink device of claim 24, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

43. The wireless sink device of claim 24, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

44. The wireless sink device of claim 24, wherein the data packet header is an application layer packet header.

45. The wireless sink device of claim 24, wherein the data packet is to control audio data or video data of the wireless source device.

46. The wireless sink device of claim 24, wherein the data packet is transmitted over TCP/IP.

47. A computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to perform a method of transmitting user input data from a wireless sink device to a wireless source device, the method comprising:

obtaining user input data at the wireless sink device;

generating payload data describing the user input data;

generating a data packet comprising a data packet header and the payload data;

transmitting the data packet to the wireless source device.

48. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

means for obtaining user input data at the wireless sink device;

means for generating payload data describing the user input data;

means for generating a data packet comprising a data packet header and the payload data;

means for transmitting the data packet to the wireless source device.

49. A method of receiving input data from a wireless sink device at a wireless source device, the method comprising:

receiving from the wireless sink device a data packet, wherein the data packet comprises a data packet header and payload data, wherein the payload data comprises data describing details of a user input;

parsing the data packet to determine an input type value in an input type field in the payload data;

processing the data describing details of the user input based on the input type value.

50. The method of claim 49, wherein the input type is selected from a group consisting of pressing a mouse button, releasing a mouse button, touching a display, releasing a display, moving a mouse cursor, moving a touch on the display, pressing a key down, releasing a key, zooming, vertical scrolling, horizontal scrolling, and rotating.

51. The method of claim 49, wherein the payload data comprises a length field identifying the length of an input description.

52. The method of claim 51, wherein the length of the input description is identified in units of octets.

53. The method of claim 49, wherein the payload data comprises a describe field comprising the data describing the details of the user input.

54. The method of claim 49, wherein the payload data comprises a first input type field, a first length field, a first describe field, a second input type field, a second length field, and a second describe field, wherein a length value contained in the first length field identifies an end of the first describe field and a beginning of the second input type field.

55. The method of claim 49, wherein the payload data identifies a number of pointer events, and for each pointer event identifies an x-coordinate and a y-coordinate corresponding to coordinates at which the pointer event occurred.

56. The method of claim 55, wherein the x-coordinate and y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

57. The method of claim 49, wherein the payload data comprises an ASCII code corresponding to a key down event.

58. The method of claim 49, wherein the payload data comprises an ASCII code corresponding to a key down event.

59. The method of claim 49, wherein the payload data comprises an x-coordinate value, a y-coordinate value, and an indication of a zoom factor.

60. The method of claim 49, wherein the payload data comprises an indication of an amount to horizontally scroll.

61. The method of claim 60, wherein the amount to horizontally scroll is in units of pixels and is based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

62. The method of claim 49, wherein the payload data comprises an indication of an amount to vertically scroll.

63. The method of claim 62, wherein the amount to vertically scroll is in units of pixels and is based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

64. The method of claim 49, wherein the payload data comprises an indication of an amount of a rotation.

65. The method of claim 64, wherein the amount of the rotation is in units of radians.

66. The method of claim 49, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

67. The method of claim 49, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

68. The method of claim 49, wherein the data packet header is an application layer packet header.

69. The method of claim 49, wherein the data packet is to control audio data or video data of the wireless source device.

70. The method of claim 49, wherein the data packet is transmitted over TCP/IP.

71. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

a transport unit to receive from the wireless sink device a data packet, wherein the data packet comprises a data packet header and payload data, wherein the payload data comprises data describing details of a user input;

a memory storing instructions;

one or more processors configured to execute the instructions, wherein upon execution of the instructions the one or more processors cause;

parsing the data packet to determine an input type value in an input type field in the payload data;

processing the data describing details of the user input based on the input type value.

72. The wireless source device of claim 71, wherein the input type is selected from a group consisting of pressing a mouse button, releasing a mouse button, touching a display, releasing a display, moving a mouse cursor, moving a touch on the display, pressing a key down, releasing a key, zooming, vertical scrolling, horizontal scrolling, and rotating.

73. The wireless source device of claim 71, wherein the payload data comprises a length field identifying the length of an input description.

74. The wireless source device of claim 73, wherein the length of the input description is identified in units of octets.

75. The wireless source device of claim 71 wherein the payload data comprises a describe field comprising the data describing the details of the user input.

76. The wireless source device of claim 71, wherein the payload data comprises a first input type field, a first length field, a first describe field, a second input type field, a second length field, and a second describe field, wherein a length value contained in the first length field identifies an end of the first describe field and a beginning of the second input type field.

77. The wireless source device of claim 71, wherein the payload data identifies a number of pointer events, and for each pointer event identifies an x-coordinate and a y-coordinate corresponding to coordinates at which the pointer event occurred.

78. The wireless source device of claim 77, wherein the x-coordinate and y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

79. The wireless source device of claim 71, wherein the payload data comprises an ASCII code corresponding to a key down event.

80. The wireless source device of claim 71, wherein the payload data comprises an ASCII code corresponding to a key down event.

81. The wireless source device of claim 71, wherein the payload data comprises an x-coordinate value, a y-coordinate value, and an indication of a zoom factor.

82. The wireless source device of claim 71, wherein the payload data comprises an indication of an amount to horizontally scroll.

83. The wireless source device of claim 82, wherein the amount to horizontally scroll is in units of pixels and is based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

84. The wireless source device of claim 71, wherein the payload data comprises an indication of an amount to vertically scroll.

85. The wireless source device of claim 84, wherein the amount to vertically scroll is in units of pixels and is based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

86. The wireless source device of claim 71, wherein the payload data comprises an indication of an amount of a rotation.

87. The wireless source device of claim 86, wherein the amount of the rotation is in units of radians.

88. The wireless source device of claim 71, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

89. The wireless source device of claim 71, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

90. The wireless source device of claim 71, wherein the data packet header is an application layer packet header.

91. The wireless source device of claim 71, wherein the data packet is to control audio data or video data of the wireless source device.

92. The wireless source device of claim 71, wherein the data packet is transmitted over TCP/IP.

93. A computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to perform a method of receiving user input data from a wireless sink device, the method comprising:

receiving from the wireless sink device a data packet, wherein the data packet comprises a data packet header and payload data, wherein the payload data comprises data describing details of a user input;

parsing the data packet to determine an input type value in an input type field in the payload data;

processing the data describing details of the user input based on the input type value.

94. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

means for receiving from the wireless sink device a data packet, wherein the data packet comprises a data packet header and payload data, wherein the payload data comprises data describing details of a user input;

means for parsing the data packet to determine an input type value in an input type field in the payload data; and

means for processing the data describing details of the user input based on the input type value.