Remote Server Environment

• US 20110314093A1
• Filed: 06/21/2011
• Published: 12/22/2011
• Est. Priority Date: 06/22/2010
• Status: Active Grant

• Current Assignee: Splashtop Inc.
• Original Assignee: Splashtop Inc.
• Inventors: Sheu, Philip, Cheng, Yu Qing, Ha, Robert, Deng, Thomas, Hsiao, Ching-Hsu, Chen, Kay, Lee, Mark
• Granted Patent: US 8,539,039 B2
• Time in Patent Office: Days
• US Class Current: 709/203
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G09G 2360/08 Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally

Abstract

A system in which computer content is encoded at low latency and distributed to one or more client devices and in which user gestures on the client device are translated into a digital representation of the computer'"'"'s native input format, thereby allowing the client device to control the computer. A method of capturing screen data on a desktop system, packaging the captured content, streaming the content to one or more client device, decoding the content on the client-side, displaying the decoded content on the client device, accepting user input, transmitting the user input to the desktop system, translating the input into a native format, and controlling the desktop system using the translated user input.

Claims

1. A method of viewing and controlling a computer display remotely on a client device comprising:

configuring a computer with at least one processing module and at least one input device configured for receiving user input of a first type;

modifying the computer resolution to reduce the encoding effort and scaling for displaying on the client;

capturing at least one computer image stream at a first frame rate using said at least one processing module;

packaging said at least one image stream for transmission to at least one client device using said at least one processing module;

encoding a packaged image stream, on a frame-by-frame basis, using said at least one processing module;

transmitting an encoded image stream to a client device using said processing module and a computer network interface via at least one network;

receiving a transmitted image stream on said at least one client device using a client network interface;

decoding a received image stream, on a frame-by-frame basis, using a decoding module on said at least one client device;

displaying a decoded image stream at said first frame rate on a display of said at least one client device;

receiving at least one client input of a second input type on said at least one client device;

translating said at least one client input of a second input type into a translated input of a first input type, thereby forming at least one control instruction

transmitting said at least one client input of a second input type from at least one client device to said computer via said at least one network interface and via said at least one network;

receiving said at least one client input of a second input type by said computer network interface; and

controlling said computer using said at least one control instruction.

2. The method of claim 1, wherein the step of translating is performed by said at least one client device.

3. The method of claim 1, wherein the step of translating is performed by said at least one processing module.

4. The method of claim 1, wherein the step of configuring a computer with at least one processing module comprises configuring said computer with a hardware computer processor with an embedded graphics controller.

5. The method of claim 1, wherein the step of configuring a computer with at least one processing module comprises configuring said computer with a hardware computer processor coupled with a hardware graphics accelerator.

6. The method of claim 1, wherein the step of configuring a computer with at least one processing module comprises configuring said computer with a hardware computer processor coupled with a dedicated hardware encoder.

7. The method of claim 6, wherein the step of configuring a computer with a hardware computer processor coupled with a dedicated hardware encoder comprises coupling said hardware computer processor with a hardware encoder is integral to said computer.

8. The method of claim 6, wherein the step of configuring a computer with a hardware computer processor coupled with a dedicated hardware encoder comprises coupling said hardware computer processor with a hardware encoder peripheral to said computer.

9. The method of claim 5, wherein the step of capturing at least one computer image stream comprises capturing said computer image stream using a graphics driver of said graphics accelerator and a screen capture API recognizable by said computer processor.

10. The method of claim 1,

wherein the step of configuring a computer with at least one processing module comprises further comprises configuring said computer with a multi-core or multi-thread processor,

wherein the step decoding a received image stream, on a frame-by-frame basis, using a decoding module further comprises decoding using a multi-core decoding module,

wherein the step of capturing at least one computer image stream comprises:

dividing said computer image into a plurality of regional frames;

capturing each regional frame of said plurality of regional frames individually using said at least one processing module, thereby forming a plurality of display image threads;

wherein the step of encoding further comprises encoding individual display image threads with individual processing cores of said multi-core or multi-thread processor;

wherein the step of decoding a received image stream further comprises;

decoding individually encoded display image threads by individual processing cores of said multi-core or multi-thread client processor; and

combining decoded display threads for display on said display of said at least one client device.

11. The method of claim 1, wherein the step of packaging said image stream for transmission further comprises:

scaling said image stream to at least one aspect ratio native to said at least one client device;

converting a color image space to a native color image space native to said at least one client device; and

combining and multiplexing a scaled and color-converted image stream with at least one time-stamped audio packet.

12. The method of claim 1, wherein the step of decoding a received image stream comprises software decoding.

13. The method of claim 1, wherein the step of decoding a received image stream comprises decoding using a peripheral hardware decoder coupled with said at least one client device.

14. The method of claim 1,

wherein the step of receiving at least one client input of a second input type further comprises receiving, via a touch screen interface, a touch screen input;

and

wherein the step of translating said at least one client input of a second type into a translated input of a first type further comprises translating said touch screen input into a mouse pointer input.

15. The method of claim 1,

wherein the step of receiving at least one client input of a second input type further comprises receiving, via a gyroscope-controlled interface, at least one of a pitch rotation, a roll rotation, and a yaw rotation;

and

wherein the step of translating said at least one client input of a second type into a translated input of a first type further comprises translating said at least one of a pitch rotation, a roll rotation, and a yaw rotation into a mouse pointer and mouse click input.

16. The method of claim 1,

wherein the step of displaying a decoded image stream at said first frame rate on a display of said at least one client device further comprises rendering a representation of virtual controller having controls of said first input type;

and

wherein said second input type comprises virtual manipulation of controls of said first input type via said virtual controller.

17. The method of claim 1, further comprising:

capturing a plurality of computer image streams,

packaging said plurality of computer image streams for transmission to at least one client device using said at least one processing module;

encoding said plurality of computer image streams, each on a frame-by-frame basis, using said at least one processing module;

transmitting a plurality of encoded image streams to a client device using said processing module and a computer network interface via at least one network;

receiving a plurality of transmitted image streams on said at least one client device using a client network interface;

decoding a plurality of received image streams, on a frame-by-frame basis, using a decoding module on said at least one client device;

displaying a plurality of decoded image streams at said first frame rate on a display of said at least one client device.

18. A multi-modal method of viewing and controlling a computer display remotely on a client device comprising:

configuring a computer with at least one processing module;

a computer display; and

at least one input device configured for receiving user input of a first type;

capturing a computer image stream at a first frame rate using said at least one processing module;

packaging said image stream for transmission to at least one client device using said at least one processing module;

encoding a packaged image stream, on a frame-by-frame basis, using said at least one processing module;

transmitting an encoded image stream to a client device using said processing module and a computer network interface via at least one network;

receiving a transmitted image stream on said at least one client device using a client network interface;

decoding a received image stream, on a frame-by-frame basis, using a decoding module on said at least one client device;

displaying, in a first display mode, a decoded image stream at said first frame rate on a display of said at least one client device;

monitoring said at least one client device to determine if a threshold period of time lapses without a user input on said at least one client device;

determining that a threshold period has lapsed, thereby triggering a second display mode comprising;

selectively skipping frames of said received image stream, thereby building an image stream buffer; and

displaying, in said second display mode, said decoded image stream at said first frame rate minus said skipped frames on a display of said at least one client device;

receiving at least one client input of a second input type on said at least one client device, thereby triggering a reversion to said first display mode;

catching up to a current frame of said received transmitted image stream, thereby discarding said buffer;

displaying, in said first display mode, said decoded image stream at said first frame rate on a display of said at least one client device;

transmitting said at least one client input of a second input type from at least one client device to said computer via said at least one network interface and via said at least one network;

receiving said at least one client input of a second input type by said computer network interface;

translating said at least one client input of a second input type into a translated input of a first input type using said at least one using said at least one processing module, thereby forming at least one control instruction; and

controlling said computer using said at least one control instruction.

19. The multi-modal method of claim 18, wherein said threshold period ranges between 15 seconds and 2 minutes.

20. The multi-modal method of claim 18, wherein the step of catching up to a current frame of said received transmitted image stream comprising immediately discarding all buffered frame and displaying the most recent decoded image frame on a display of said at least one client device;

21. The multi-modal method of claim 18, wherein the step of catching up to a current frame of said received transmitted image stream comprising displaying one or more of the buffered frames at an increased frame rate until reaching the most recent decoded image frame.

22. A system for viewing and controlling a computer display remotely on a client device comprising:

a computer system comprising;

a processor;

at least one input device configured for receiving user input of a first type;

a screen capture module configured for modifying the display resolution and capturing at least one computer image stream at a first frame rate;

a packaging module configured for packaging said at least one image stream for transmission to at least one client device;

an encoder module configured for encoding a packaged image stream, on a frame-by-frame basis; and

a network interface operatively coupled with said processor and configured for transmitting an encoded image stream to a client device; and

a client device comprising;

a client device processor;

a receiver configured for receiving a transmitted image stream on said at least one client device;

a decoder configured for decoding a received image stream, on a frame-by-frame basis;

a client device display configured for displaying a decoded image stream at said first frame rate;

a client input system configured for receiving user input of a second type;

an input translation module configured for translating said at least one client input of a second input type into a translated input of a first input type in part or full using, thereby forming at least one control instruction; and

a transmitter configured for transmitting said at least one control instruction to said computer;

wherein said network interface of said computer system is further configured for receiving said at least one control instruction by said computer network interface;

wherein said computer system further comprises a controller configured controlling at least one application running on said computer using said at least one control instruction.

23. The system of claim 22, wherein said processor is operatively coupled with a dedicated hardware encoder containing said encoder module.

24. The system of claim 22, wherein said processor comprises a multi-core processor, wherein said decoder on said client device comprises a multi-core decoder.

25. The system of claim 24, wherein said screen capture module further comprises:

a screen divider configured for dividing said computer image into a plurality of regional frames;

a regional frame capture module configured for capturing each regional frame of said plurality of regional frames individually, thereby forming a plurality of display image threads;

wherein said multi-core processer is further configured for encoding individual display image threads with individual processing cores; and

wherein said decoder is further configured for decoding individually encoded display image threads by individual processing cores of said multi-core decoder.

26. The system of claim 22, wherein said client input system further comprises a touch screen interface, and wherein said input translation module is further configured for translating touch screen inputs into mouse pointer and mouse click inputs.

27. The system of claim 26, wherein said client device processor is further configured for rendering a virtual representation of an input device and displaying said virtual representation of an input device on said client device display, and wherein said client input system is further configured receiving user input by user virtually manipulating said virtual representation of an input device via said touch screen interface.

28. The system of claim 22, wherein said client input system further comprises a gyro-scope interface, and wherein said input translation module is further configured for translating at least one of a pitch rotation, a roll rotation, and a yaw rotation into mouse pointer and mouse click inputs.

29. A system for translating inputs comprising:

a first computer device comprising;

a processor;

a memory device;

a display configured for displaying an image stream; and

a user input system configured for receiving inputs of a first input type selected from among mouse pointer and clicks, video game controller inputs, gyroscope rotational inputs, accelerometer inputs, camera inputs, and microphone inputs;

at least one additional computer device comprising;

at least one additional processor;

at least one additional memory device;

at least one additional display configured for displaying an image stream;

at least one additional user input system configured for receiving inputs of a second input type that is different from said first input type and that is selected from among mouse pointer and clicks, gyroscope rotational inputs, accelerometer inputs, camera inputs, and microphone inputs;

an input translation module, operatively coupled with said first computer device and with said at least one additional computer device, and configured for translating said first input type into said second input type, in part or in full.

30. A system for viewing and controlling a computer display remotely on a client device comprising:

a computer system comprising;

a processor;

at least one input device configured for receiving user input of a first type;

a screen capture module configured for modifying the display resolution and capturing at least one computer image stream at a first frame rate;

a packaging module configured for packaging said at least one image stream for transmission to at least one client device;

an encoder module configured for encoding a packaged image stream, on a frame-by-frame basis; and

a network interface operatively coupled with said processor and configured for transmitting an encoded image stream to a client device; and

a client device comprising;

a client device processor;

a receiver configured for receiving a transmitted image stream on said at least one client device;

a decoder configured for decoding a received image stream, on a frame-by-frame basis;

a client device display configured for displaying a decoded image stream at said first frame rate in a first display mode;

a client input system configured for receiving user input of a second type;

an input monitor configured for monitoring said at least one client device to determine if a threshold period of time lapses without a user input on said at least one client device;

a multi-modal trigger module configured for determining that a threshold period has lapsed, thereby triggering a second display mode;

wherein, while in said second display mode, said client device processor is further configured for selectively skipping frames of said received image stream, thereby building an image stream buffer

wherein, while in said second display mode, said client device processor is further configured for displaying, said decoded image stream at said first frame rate minus said skipped frames on a display of said at least one client device;

wherein said client device processor is further configured for reverting to said first display mode upon receiving a input;

an input translation module configured for translating said at least one client input of a second input type into a translated input of a first input type in part or full using, thereby forming at least one control instruction; and

a transmitter configured for transmitting said at least one control instruction to said computer;

wherein said network interface of said computer system is further configured for receiving said at least one control instruction by said computer network interface;

wherein said computer system further comprises a controller configured controlling at least one application running on said computer using said at least one control instruction.