Multi-Touch Remoting

• US 20130031482A1
• Filed: 07/28/2011
• Published: 01/31/2013
• Est. Priority Date: 07/28/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Saul, Elton, Meister, Benjamin, Keymer, Daniel
• Granted Patent: US 9,727,227 B2
• Time in Patent Office: Days
• US Class Current: 715/740
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: H04L 65/1101 Session protocols

Abstract

An invention is disclosed for using multi-touch input in a remote presentation session. In embodiments of the invention, a client computer is configured to locally receive both mouse and keyboard input, and multi-touch input. Where the client computer receives mouse or keyboard input, it sends this input to a remote presentation session server via a TCP connection. Where the client computer receives multi-touch input, it sends this input to the remote presentation session server via a UDP connection. The server computer processes input received from the client computer, via either the TCP or UDP connection, and sends the client a graphical result of performing processing corresponding to that received input.

Claims

1. A method for remoting multi-touch input in a remote presentation session between a remote presentation client computer and a remote presentation server computer that serves the remote presentation session comprising graphical output from a user session, comprising:

receiving, by the client computer, multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer;

sending an indication of the multi-touch input to a process executing in the user session of the server computer via a User Datagram Protocol (UDP) connection, the server computer performing processing corresponding to the multi-touch input;

establishing, by the client computer, a Transmission Control Protocol (TCP) connection with the server computer;

receiving, by the client computer, a second input indicative of mouse, keyboard, or pen input made locally to the client computer; and

sending an indication of the second input to a process executing in system space of the server computer via the TCP connection, the process executing in system space injecting the second input into the user session, the user session performing processing corresponding to the second input.

2. The method of claim 1, wherein the multi-touch input comprises a plurality of frames of multi-touch data, and further comprising:

determining that the multi-touch input indicated by two consecutive frames of the plurality of frames is identical; and

wherein sending an indication of the multi-touch input comprises;

sending an indication of only one of the two consecutive frames.

3. The method of claim 2, wherein sending an indication of the input to a process executing in the user session of the server computer, the server computer performing processing corresponding to the input comprises:

the server computer injecting a keep alive indication into the user session in response to determining that a period of time since a time indicated by a frame of the plurality of frames is above a threshold.

4. The method of claim 1, wherein the multi-touch input comprises a plurality of contacts, and further comprising:

mapping each contact from a first address space to a second address space, the second address space being smaller than the first address space.

5. The method of claim 4, further comprising:

in response to determining that the number of contacts in the plurality of contacts is greater than a maximum number of allowed contacts, determining to send the server only multi-touch input corresponding to a number of contacts equal to the maximum number of allowed contacts.

6. The method of claim 1, further comprising:

establishing, by the client computer, a remote presentation session with the server computer, the server computer disabling graphical multi-touch feedback on the server computer.

7. The method of claim 1, wherein the multi-touch input comprises a plurality of frames, and further comprising:

marking each frame of the plurality of frames with a relative timestamp indicative of a time at which the frame of input was made by the user.

8. The method of claim 7, wherein marking each frame of the plurality of frames with a relative timestamp comprises:

marking each frame of the plurality of frames with a relative timestamp indicative of a time at which the frame of input was made by the user relative to when the most recent frame of the plurality of frames was made by the user or the oldest frame of the plurality of frames was made by the user.

9. The method of claim 8, wherein marking each frame of the plurality of frames with a relative timestamp comprises:

replacing an absolute timestamp of each frame with the relative timestamp.

10. The method of claim 1, wherein receiving, by the client computer, multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer comprises:

receiving an indication of a pressure with which the user multi-touched the multi-touch-sensitive input device.

11. A system for remoting multi-touch input in a remote presentation session between a remote presentation client computer and a remote presentation server computer that serves the remote presentation session comprising graphical output from a user session, comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system to at least;

establish a remote presentation session connection with the server computer;

receive multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer; and

send an indication of the multi-touch input to a process executing in the user session of the server computer via the UDP connection, the server computer performing processing corresponding to the input.

12. The system of claim 11, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

establish a Transmission Control Protocol (TCP) connection with the server computer;

receive a second input indicative of mouse, keyboard, or pen input made locally to the client computer;

send an indication of the second input to the server computer via the TCP connection, the server computer performing processing corresponding to the second input.

13. The system of claim 11, wherein the processor-executable instructions, that when executed on the processor, cause the system to at least send an indication of the second input to the server computer via the TCP connection further cause the system to at least:

send the indication of the second input to a process executing in system space of the server computer, the process executing in system space injecting the second input into the user session.

14. The system of claim 11, wherein the processor-executable instructions, that when executed on the processor, cause the system to at least receive multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer further cause the system to at least:

receive an indication of an orientation with which the user multi-touched the multi-touch-sensitive input device.

15. The system of claim 11, wherein the processor-executable instructions, that when executed on the processor, cause the system to at least receive multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer further cause the system to at least:

receive an indication of a rectangular area formed by a contact point of the user on the multi-touch-sensitive input device.

16. The system of claim 11, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

send an indication that the system supports multi-touch input, the server computer initializing the process in the user session in response to receiving the indication that the client computer supports multi-touch input.

17. The system of claim 11, wherein the input comprises a plurality of frames, and wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

determine a field of each frame of the plurality of frames not needed by the server computer to process the multi-touch input; and

remove the field from each frame.

18. The system of claim 11, wherein the multi-touch input comprises a frame, the frame having a field, and wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

determine that the field of the frame represents the same value as the field of a frame possessed by the server computer; and

remove the field from the frame before sending the indication of the input to the process executing in the user session of the server computer.

19. The system of claim 11, wherein the multi-touch input comprises a plurality of frames of multi-touch data, and wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

determine that the multi-touch input indicated by two consecutive frames of the plurality of frames is identical; and

wherein the instructions that, when executed on the processor, cause the system to at least send an indication of the multi-touch input further cause the system to at least;

send an indication of only one of the two consecutive frames.

20. A computer-readable storage medium for remoting multi-touch input in a remote presentation session with a remote presentation server computer that serves the remote presentation session comprising graphical output from a user session, bearing computer-readable instructions that, when executed on a computer, cause the computer to perform operations comprising:

establishing a remote presentation session with the remote presentation server computer;

receiving multi-touch input indicative of a user multi-touching a multi-touch-sensitive input device made locally to the client computer, the multi-touch input comprising a plurality of frames, each frame comprising an absolute timestamp representing an absolute time at which multi-touch input corresponding to the frame was made;

replacing the absolute timestamp of each frame with a relative timestamp, the relative timestamp representing a time at which multi-touch input corresponding to the frame was made relative to the oldest frame of the plurality of frames or to the newest frame of the plurality of frames; and

sending an indication of the multi-touch input to the server computer via the remote presentation session, the server computer performing processing corresponding to the multi-touch input, and sending a graphical result of performing processing corresponding to the multi-touch input to the computer.