Multi-touch remoting

• US 9,727,227 B2
• Filed: 07/28/2011
• Issued: 08/08/2017
• Est. Priority Date: 07/28/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Saul, Elton, Meister, Benjamin, Keymer, Daniel
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Heffington, John M
• Application Number:
  US13/193,565
• Publication Number:
  US 20130031482A1
• Time in Patent Office: 2,203 Days
• Field of Search: 715740, 715836
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: H04L 65/1101 Session protocols

Abstract

An invention is disclosed for using multi-touch input in a remote presentation session. In embodiments of the invention, a client computer is configured to locally receive both mouse and keyboard input, and multi-touch input. Where the client computer receives mouse or keyboard input, it sends this input to a remote presentation session server via a TCP connection. Where the client computer receives multi-touch input, it sends this input to the remote presentation session server via a UDP connection. The server computer processes input received from the client computer, via either the TCP or UDP connection, and sends the client a graphical result of performing processing corresponding to that received input.

Claims

1. A method for remoting multi-touch input in a remote presentation session between a remote presentation client computer and a remote presentation server computer that serves the remote presentation session comprising graphical output from a user session, comprising:

establishing the remote presentation session connection with the remote presentation server computer, wherein the remote presentation session connection comprises a Transmission Control Protocol (TCP) connection and a User Datagram Protocol (UDP) connection, the TCP connection being used to receive graphical presentation output from the user session running on the remote presentation server computer;

receiving, by the client computer, multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer;

sending an indication of the multi-touch input to a process executing in the user session of the server computer via the UDP connection;

receiving via the TCP connection graphical presentation output generated by the server computer performing processing corresponding to the multi-touch input;

receiving, by the client computer, a second input indicative of mouse, keyboard, or pen input made locally to the client computer; and

sending an indication of the second input to a process executing in system space of the server computer via the TCP connection, the process executing in system space injecting the second input into the user session, the user session performing processing corresponding to the second input.

2. The method of claim 1, wherein the multi-touch input comprises a plurality of frames of multi-touch data, further comprising:

determining that the multi-touch input indicated by two consecutive frames of the plurality of frames is identical; and

wherein sending an indication of the multi-touch input comprises;

sending an indication of only one of the two consecutive frames.

3. The method of claim 2, wherein sending an indication of the input to a process executing in the user session of the server computer, the server computer performing processing corresponding to the input comprises:

the server computer injecting a keep alive indication into the user session in response to determining that a period of time since a time indicated by a frame of the plurality of frames is above a threshold.

4. The method of claim 1, wherein the multi-touch input comprises a plurality of contacts, and further comprising:

mapping each contact from a first address space to a second address space, the second address space being smaller than the first address space.

5. The method of claim 4, further comprising:

in response to determining that the number of contacts in the plurality of contacts is greater than a maximum number of allowed contacts, determining to send the server only multi-touch input corresponding to a number of contacts equal to the maximum number of allowed contacts.

6. The method of claim 1, further comprising:

establishing, by the client computer, a remote presentation session with the server computer, the server computer disabling graphical multi-touch feedback on the server computer.

7. The method of claim 1, wherein the multi-touch input comprises a plurality of frames of multi-touch data, further comprising:

marking each frame of the plurality of frames with one of the relative timestamps indicative of a time at which the frame of input was made by the user.

8. The method of claim 7, wherein each relative timestamp is indicative of a time at which the frame of input was made by the user relative to when the most recent frame of the plurality of frames was made by the user or the oldest frame of the plurality of frames was made by the user.

9. The method of claim 8, further comprising marking each frame of the plurality of frames with one of the relative timestamps by replacing an absolute timestamp of each frame of the plurality of frames with the relative timestamp.

10. The method of claim 1, wherein receiving, by the client computer, multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer comprises:

receiving an indication of a pressure with which the user multi-touched the multi-touch-sensitive input device.

11. A system for remoting multi-touch input in a remote presentation session between a remote presentation client computer and a remote presentation server computer that serves the remote presentation session comprising graphical output from a user session, comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system to at least;

establish a remote presentation session connection with the server computer, wherein the remote presentation session connection comprises a reliable connection established using a reliable data transport protocol and an unreliable connection established using an unreliable data transport protocol, the reliable connection being used to receive graphical presentation output from the user session running on the server computer;

receive multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer;

send an indication of the multi-touch input to a process executing in the user session of the server computer via the unreliable connection; and

receive via the reliable connection graphical presentation output generated by the server computer performing processing corresponding to the multi-touch input.

12. The system of claim 11, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

receive a second input indicative of mouse, keyboard, or pen input made locally to the client computer; and

send an indication of the second input to the server computer via the reliable connection, the server computer performing processing corresponding to the second input, wherein the reliable connection is a Transmission Control Protocol (TCP) connection.

13. The system of claim 12, wherein the processor-executable instructions, that when executed on the processor, cause the system to at least send an indication of the second input to the server computer via the reliable connection further cause the system to at least:

send the indication of the second input to a process executing in system space of the server computer, the process executing in system space injecting the second input into the user session.

14. The system of claim 11, wherein the processor-executable instructions, that when executed on the processor, cause the system to at least receive multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer further cause the system to at least:

receive an indication of an orientation with which the user multi-touched the multi-touch-sensitive input device.

15. The system of claim 11, wherein the processor-executable instructions, that when executed on the processor, cause the system to at least receive multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer further cause the system to at least:

receive an indication of a rectangular area formed by a contact point of the user on the multi-touch-sensitive input device.

16. The system of claim 11, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

send an indication that the system supports multi-touch input, the server computer initializing the process in the user session in response to receiving the indication that the client computer supports multi-touch input.

17. The system of claim 11, wherein the multi-touch input comprises a plurality of frames, and wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

determine a field of each frame of the plurality of frames not needed by the server computer to process the multi-touch input; and

remove the field from each frame.

18. The system of claim 11, wherein the multi-touch input comprises a plurality of frames, and wherein at least one frame of the plurality of frames includes a field, and wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

determine that the field of the at least one frame represents the same value as the field of a frame possessed by the server computer; and

remove the field from the at least one frame before sending the indication of the input to the process executing in the user session of the server computer.

19. The system of claim 11, wherein the multi-touch input comprises a plurality of frames, and wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

determine that the multi-touch input indicated by two consecutive frames of the plurality of frames is identical; and

wherein the instructions that, when executed on the processor, cause the system to at least send an indication of the multi-touch input further cause the system to at least;

send an indication of only one of the two consecutive frames.

20. A computer-readable storage medium for remoting multi-touch input in a remote presentation session with a remote presentation server computer that serves the remote presentation session comprising graphical output from a user session, bearing computer-readable instructions that, when executed on a computer, cause the computer to perform operations comprising:

establishing the remote presentation session with the remote presentation server computer, wherein the remote presentation session comprises a reliable connection established using a reliable data transport protocol and an unreliable connection established using an unreliable data transport protocol, the reliable connection being used to receive graphical presentation output from the user session running on the remote presentation server computer;

receiving multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer;

sending an indication of the multi-touch input to the server computer via the unreliable connection of the remote presentation session; and

receiving, via the reliable connection of the remote presentation session, graphical presentation output generated by the server computer performing processing corresponding to the multi-touch input.