User input back channel from a wireless sink device to a wireless source device for multi-touch gesture wireless displays

• US 9,065,876 B2
• Filed: 01/05/2012
• Issued: 06/23/2015
• Est. Priority Date: 01/21/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Huang, Xiaolong, Raveendran, Vijayalakshmi R., Wang, Xiaodong, Shaukat, Fawad
• Primary Examiner(s): PEYTON, TAMMARA R
• Application Number:
  US13/344,377
• Publication Number:
  US 20130009887A1
• Time in Patent Office: 1,265 Days
• Field of Search: 710 8- 10
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G09G 2370/10 Use of a protocol of communication by packets in interfaces along the display data pipeline
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/752 adapting media to network capabilities

Abstract

As part of a communication session, a wireless source device can transmit audio and video data to a wireless sink device, and the wireless sink device can transmit user input data received at the wireless sink device back to the wireless source device. In this manner, a user of the wireless sink device can control the wireless source device and control the content that is being transmitted from the wireless source device to the wireless sink device. The input data received at the wireless sink device can be a multi-touch gesture.

Claims

1. A method of transmitting user input data from a wireless sink device to a wireless source device, the method comprising:

receiving video data, at the wireless sink device, from the wireless source device;

rendering the video data at the wireless sink device;

obtaining user input data for a multi-touch gesture at the wireless sink device;

generating a packet header;

generating payload data, wherein the payload data associates user input data for a first touch input event with a first pointer identification and user input data for a second touch input event with a second pointer identification;

generating a data packet comprising the packet header and the payload data; and

transmitting the data packet to the wireless source device, wherein the data packet transmitted to the wireless source device controls the video data being received at the wireless sink device from the wireless source device.

2. The method of claim 1, wherein the payload data comprises a field to identify a number of pointers, wherein each pointer is associated with a unique pointer identification.

3. The method of claim 1, wherein the number of pointers is greater than two.

4. The method of claim 1, wherein the first touch input event and the second touch input event occur at substantially the same time.

5. The method of claim 1, wherein the first touch input event is selected from a group consisting of touching down on an input device, touching up from the input device, and moving a touch on the input device.

6. The method of claim 1, wherein the payload data comprises:

a first touch down input associated with the first pointer identification;

a first touch move input associated with the first pointer identification;

a first touch up input associated with the first pointer identification;

a second touch down input associated with the second pointer identification;

a second touch move input associated with the second pointer identification;

a second touch up input associated with the second pointer identification.

7. The method of claim 1, wherein the data packet header comprises a field to identify an input category of the user input data, and the field identifies a generic input.

8. The method of claim 1, wherein the payload data comprises a length field identifying the length of an input description.

9. The method of claim 8, wherein the length of the input description is identified in units of octets.

10. The method of claim 1, wherein the payload data comprises a describe field to describe details of the user input data.

11. The method of claim 1, wherein the payload data identifies:

a first x-coordinate and a first y-coordinate corresponding to coordinates at which the first touch input event occurred;

a second x-coordinate and a y-coordinate corresponding to coordinates at which the second touch input event occurred.

12. The method of claim 11, wherein the first x-coordinate, first y-coordinate, second x-coordinate, and second y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

13. The method of claim 1, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

14. The method of claim 1, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

15. The method of claim 1, wherein the data packet header is an application layer packet header.

16. The method of claim 1, wherein the data packet is to control audio data or video data of the wireless source device.

17. The method of claim 1, wherein the data packet is transmitted over TCP/IP.

18. A wireless sink device configured to transmit user input data to a wireless source device, the wireless sink device comprising:

a memory storing instructions;

one or more processors configured to execute the instructions, wherein upon execution of the instructions the one or more processors cause the wireless sink device to;

receive video data, at the wireless sink device, from the wireless source device;

render the video data at the wireless sink device;

obtain user input data for a multi-touch gesture at the wireless sink device;

generate a packet header;

generate payload data, wherein the payload data associates user input data for a first touch input event with a first pointer identification and user input data for a second touch input event with a second pointer identification; and

generate a data packet comprising the packet header and the payload data;

a transport unit configured to transmit the data packet to the wireless source device, wherein the data packet transmitted to the wireless source device controls the video data being received at the wireless sink device from the wireless source device.

19. The device of claim 18, wherein the payload data comprises a field to identify a number of pointers, wherein each pointer is associated with a unique pointer identification.

20. The device of claim 18, wherein the number of pointers is greater than two.

21. The device of claim 18, wherein the first touch input event and the second touch input event occur at substantially the same time.

22. The device of claim 18, wherein the first touch input event is selected from a group consisting of touching down on an input device, touching up from the input device, and moving a touch on the input device.

23. The device of claim 18, wherein the payload data comprises:

a first touch down input associated with the first pointer identification;

a first touch move input associated with the first pointer identification;

a first touch up input associated with the first pointer identification;

a second touch down input associated with the second pointer identification;

a second touch move input associated with the second pointer identification;

a second touch up input associated with the second pointer identification.

24. The device of claim 18, wherein the data packet header comprises a field to identify an input category of the user input data, and the field identifies a generic input.

25. The device of claim 18, wherein the payload data comprises a length field identifying the length of an input description.

26. The device of claim 25, wherein the length of the input description is identified in units of octets.

27. The device of claim 18, wherein the payload data comprises a describe field to describe details of the user input data.

28. The device of claim 18, wherein the payload data identifies:

a first x-coordinate and a first y-coordinate corresponding to coordinates at which the first touch input event occurred;

a second x-coordinate and a y-coordinate corresponding to coordinates at which the second touch input event occurred.

29. The device of claim 28, wherein the first x-coordinate, first y-coordinate, second x-coordinate, and second y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

30. The device of claim 18, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

31. The device of claim 18, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

32. The device of claim 18, wherein the data packet header is an application layer packet header.

33. The device of claim 18, wherein the data packet is to control audio data or video data of the wireless source device.

34. The device of claim 18, wherein the data packet is transmitted over TCP/IP.

35. A non-transitory computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to perform a method of transmitting user input data from a wireless sink device to a wireless source device, the method comprising:

receiving video data, at the wireless sink device, from the wireless source device;

rendering the video data at the wireless sink device;

obtaining user input data for a multi-touch gesture at the wireless sink device;

generating a packet header;

generating payload data, wherein the payload data associates user input data for a first touch input event with a first pointer identification and user input data for a second touch input event with a second pointer identification;

generating a data packet comprising the packet header and the payload data; and

transmitting the data packet to the wireless source device, wherein the data packet transmitted to the wireless source device controls the video data being received at the wireless sink device from the wireless source device.

36. A wireless sink device configured to transmit user input to a wireless source device, the wireless sink device comprising:

receiving video data, at the wireless sink device, from the wireless source device;

rendering the video data at the wireless sink device;

means for obtaining user input data for a multi-touch gesture at the wireless sink device;

means for generating a packet header;

means for generating payload data, wherein the payload data associates user input data for a first touch input event with a first pointer identification and user input data for a second touch input event with a second pointer identification;

means for generating a data packet comprising the packet header and the payload data;

means for transmitting the data packet to the wireless source device, wherein the data packet transmitted to the wireless source device controls the video data being received at the wireless sink device from the wireless source device.

37. A method of receiving user input data from a wireless sink device at a wireless source device, the method comprising:

receiving a data packet comprising a data packet header and payload data from the wireless sink device;

parsing the payload data to identify;

user input data for a first touch input event with a first pointer identification, and

user input data for a second touch input event with a second pointer identification;

interpreting the user input data for the first touch input event and the user input data for

the second touch input event as a multi-touch gesture; and

transmitting video data to the wireless sink device based on the received data packet.

38. The method of claim 37, wherein the payload data comprises a field to identify a number of pointers, wherein each pointer is associated with a unique pointer identification.

39. The method of claim 37, wherein the number of pointers is greater than two.

40. The method of claim 37, wherein the first touch input event and the second touch input event occur at substantially the same time at the wireless sink device.

41. The method of claim 37, wherein the first touch input event is selected from a group consisting of touching down on an input device, touching up from the input device, and moving a touch on the input device.

42. The method of claim 37, wherein the payload data comprises:

a first touch down input event associated with the first pointer identification;

a first touch move input event associated with the first pointer identification;

a first touch up input event associated with the first pointer identification;

a second touch down input event associated with the second pointer identification;

a second touch move input event associated with the second pointer identification;

a second touch up input event associated with the second pointer identification; and

,

wherein the method further comprises;

interpreting the first touch down input event, first touch move input event, and first touch up input event as a first touch input event sequence;

interpreting the second touch down input event, second touch move input event, and second touch up input event as a second touch input event sequence;

interpreting the first touch input event sequence and the second touch input event sequence as the multi-touch gesture.

43. The method of claim 37, wherein the data packet header comprises a field to identify an input category of the user input data, and the field identifies a generic input.

44. The method of claim 37, wherein the payload data comprises a length field identifying the length of an input description.

45. The method of claim 44, wherein the length of the input description is identified in units of octets.

46. The method of claim 37, wherein the payload data comprises a describe field to describe details of the user input data.

47. The method of claim 37, wherein the payload data identifies:

a first x-coordinate and a first y-coordinate corresponding to coordinates at which the first touch input event occurred;

a second x-coordinate and a second y-coordinate corresponding to coordinates at which the second touch input event occurred.

48. The method of claim 47, wherein the first x-coordinate, first y-coordinate, second x-coordinate, and second y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

49. The method of claim 37, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

50. The method of claim 37, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

51. The method of claim 37, wherein the data packet header is an application layer packet header.

52. The method of claim 37, wherein the data packet is to control audio data or video data of the wireless source device.

53. The method of claim 37, wherein the data packet is transmitted over TCP/IP.

54. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

a transport unit to receive a data packet comprising a data packet header and payload data from the wireless sink device;

a memory storing instructions;

one or more processors configured to execute the instructions, wherein upon execution of the instructions the one or more processors cause;

parsing the payload data to identify;

user input data for a first touch input event with a first pointer identification, and

user input data for a second touch input event with a second pointer identification;

interpreting the user input data for the first touch input event and the user input data for the second touch input event as a multi-touch gesture; and

transmitting video data to the wireless sink device based on the received data packet.

55. The wireless source device of claim 54, wherein the payload data comprises a field to identify a number of pointers, wherein each pointer is associated with a unique pointer identification.

56. The wireless source device of claim 54, wherein the number of pointers is greater than two.

57. The wireless source device of claim 54, wherein the first touch input event and the second touch input event occur at substantially the same time at the wireless sink device.

58. The wireless source device of claim 54, wherein the first touch input event is selected from a group consisting of touching down on an input device, touching up from the input device, and moving a touch on the input device.

59. The wireless source device of claim 54, wherein the payload data comprises:

a first touch down input event associated with the first pointer identification;

a first touch move input event associated with the first pointer identification;

a first touch up input event associated with the first pointer identification;

a second touch down input event associated with the second pointer identification;

a second touch move input event associated with the second pointer identification;

a second touch up input event associated with the second pointer identification; and

,

wherein the method further comprises;

interpreting the first touch down input event, first touch move input event, and first touch up input event as a first touch input event sequence;

interpreting the second touch down input event, second touch move input event, and second touch up input event as a second touch input event sequence;

interpreting the first touch input event sequence and the second touch input event sequence as the multi-touch gesture.

60. The wireless source device of claim 54, wherein the data packet header comprises a field to identify an input category of the user input data, and the field identifies a generic input.

61. The wireless source device of claim 54, wherein the payload data comprises a length field identifying the length of an input description.

62. The wireless source device of claim 61, wherein the length of the input description is identified in units of octets.

63. The wireless source device of claim 54, wherein the payload data comprises a describe field to describe details of the user input data.

64. The wireless source device of claim 54, wherein the payload data identifies:

a first x-coordinate and a first y-coordinate corresponding to coordinates at which the first touch input event occurred;

a second x-coordinate and a second y-coordinate corresponding to coordinates at which the second touch input event occurred.

65. The wireless source device of claim 64, wherein the first x-coordinate, first y-coordinate, second x-coordinate, and second y-coordinate are based on a negotiated resolution of a video stream between the wireless sink device and the wireless source device.

66. The wireless source device of claim 54, wherein obtaining the user input data comprises capturing the user input data through an input device of the wireless sink device.

67. The wireless source device of claim 54, wherein obtaining the user input data comprises receiving forwarded user input data from another wireless sink device.

68. The wireless source device of claim 54, wherein the data packet header is an application layer packet header.

69. The wireless source device of claim 54, wherein the data packet is to control audio data or video data of the wireless source device.

70. The wireless source device of claim 54, wherein the data packet is transmitted over TCP/IP.

71. A non-transitory computer-readable storage medium storing instructions that upon execution by one or more processors cause the one or more processors to perform a method of receiving user input data from a wireless sink device at a wireless source device, the method comprising:

receiving a data packet comprising a data packet header and payload data from the wireless sink device;

parsing the payload data to identify;

user input data for a first touch input event with a first pointer identification, and

user input data for a second touch input event with a second pointer identification;

interpreting the user input data for the first touch input event and the user input data for the second touch input event as a multi-touch gesture; and

transmitting video data to the wireless sink device based on the received data packet.

72. A wireless source device configured to receive user input data from a wireless sink device, the wireless source device comprising:

means for receiving a data packet comprising a data packet header and payload data from the wireless sink device;

means for parsing the payload data to identify;

user input data for a first touch input event with a first pointer identification, and

user input data for a second touch input event with a second pointer identification;

means for interpreting the user input data for the first touch input event and the user input data for the second touch input event as a multi-touch gesture; and

means for transmitting video data to the wireless sink device based on the received data packet.