Remote Control of a Mobile Device

• US 20160249106A1
• Filed: 09/14/2012
• Published: 08/25/2016
• Est. Priority Date: 09/14/2012
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Appurify, Inc. (Alphabet Inc.)
• Inventors: LACHWANI, Manish, SRINIVASAN, Jay, JAIN, Rahul, PATNAIK, Pratyus
• Application Number:
  US13/619,867
• Publication Number:
  US 20160249106A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 21/234309 by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo conversion of standards in analog television systems H04N7/01
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/478 Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/482 End-user interface for program selection broadcast systems using EPGs H04H60/72

Abstract

A remote control module enables remote operation and control of a host device such as a mobile computing device. The user may employ an interface of the service to specify input events to be executed on the host device, such as touch, text, numeric, key, voice, haptic, or other types of inputs. The input events may be executed on the device through a virtualization module executing as a background process on the device. A display capture module may also execute as a background process on the host device to capture screenshots of the host device display using a hardware video encoder of the host device. The screenshots are placed in device memory. The remote control module may retrieve the screenshots from the host device memory, assemble them into a video stream, and provide the video stream to the user through the user interface as a real-time view of the host device.

Claims

1. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by at least one processor of a client device, instruct the client device to perform operations comprising:

specifying, by a user interface of the client device, at least one input event to be performed on a host device, wherein the user interface comprises an interactive control and a selection control, wherein the interactive control is configured to receive a first particular input at the client device and generate a first input event corresponding to the first particular input, and wherein the selection control is configured to allow selection of a particular input event from a list of input events and generate a second input event corresponding to the selected particular input event, wherein the client device is not configured to receive a multi-part gesture as an input, and wherein the list of input events comprises an input event corresponding to the multi-part gesture;

providing, by the client device, one or more instructions to the host device, wherein the one or more instructions perform at least the first input event and the second input event on the host device;

receiving, at the client device, a plurality of screenshots of a display of the host device generated in response to at least the first input event and the second input event; and

providing the plurality of screenshots using the user interface of the client device.

2. The non-transitory computer-readable media of claim 1, the operations further comprising:

detecting available throughput of a network connection to the client device; and

adjusting a resolution of a video stream based at least in part on the detected available throughput of the network connection, wherein the video stream comprises the plurality of screenshots.

3. The non-transitory computer-readable media of claim 1, wherein the at least one input event includes an input event for at least one of a gestural input, a touch input, an audio input, a haptic input, a key input, a numeric input, and a text input.

4. The non-transitory computer-readable media of claim 1, wherein the user interface comprises a device display, configured to display the plurality of screenshots, and wherein the interactive control is configured to receive the first particular input from within the device display.

5. The non-transitory computer-readable media of claim 1, further comprising:

setting one or more conditions for a communications network via the user interface, wherein the one or more conditions comprise a signal strength of the communications network.

6. The non-transitory computer-readable media of claim 1,

wherein providing the plurality of screenshots using the user interface of the client device comprises:

assembling the plurality of screenshots into a video stream; and

providing the video stream using the user interface of the client device.

7. The non-transitory computer-readable media of claim 6, the operations further comprising:

downsampling the video stream prior to providing the video stream to the client device, based on at least one of a device type, a device capability, or a network connection speed for the client device.

8. A system comprising:

at least one memory;

at least one processor configured to access the at least one memory; and

a remote control module, stored in the at least one memory and executed by the at least one processor, the remote control module comprising;

a user interface configured to specify at least one input event, the user interface module comprising;

a device display, configured to display a plurality of screenshots generated in response to the at least one input event,

an interactive control, configured to receive a first particular input within the device display and generate a first input event corresponding to the first particular input, and

a selection control, configured to allow selection of a particular input event from a list of input events and generate a second input event corresponding to the selected particular input event, wherein the user interface is not configured to receive a multi-part gesture as an input, and wherein the list of input events comprises an input event related to the multi-part gesture;

wherein the remote module is configured to;

provide one or more instructions to perform at least the first input event and the second input event to a host device;

receive a plurality of screenshots of a display of the host device, the plurality of screenshots generated in response to at least the first input event and the second input event; and

provide the plurality of screenshots using the user interface.

9. The system of claim 8, wherein the plurality of screenshots are generated at a pre-determined refresh rate and a pre-determined resolution.

10. The system of claim 8, where in the plurality of screenshots are encoded in a pre-determined video coding format.

11. The system of claim 8, wherein the plurality of screenshots are stored such that a second screenshot overwrites a first screenshot, and wherein the first screenshot precedes the second screenshot.

12. The system of claim 8, wherein the at least one input event is provided to remotely control at least one application.

13. A method, comprising:

specifying, by a user interface of a client device, at least one input event, wherein the user interface comprises an interactive control and a selection control, wherein the interactive control is configured to receive a first particular input at the client device and generate a first input event corresponding to the first particular input, and wherein the selection control is configured to allow selection of a particular input event from a list of input events and generate a second input event corresponding to the selected particular input event, wherein the client device is not configured to receive a multi-part gesture as an input, and wherein the list of input events comprises an input event related to the multi-part gesture;

providing, by the client device, one or more instructions to perform at least the first input event and the second input event on a host device;

receiving, at the client device, a plurality of images of display of the host device, wherein the plurality of images were generated in response to at least the first input event and the second input event; and

providing the plurality of images using the user interface of the client device.

14. The method of claim 13, further comprising:

detecting available throughput of a network connection to the client device; and

adjusting a resolution of a video stream based at least in part on the detected available throughput of the network connection, wherein the video stream comprises

the plurality of images.

15. The method of claim 13, wherein the at least one input event includes an input event for at least one of a gestural input, a touch input, an audio input, a haptic input, a key input, a numeric input, and a text input.

16. The method of claim 13,

wherein the user interface comprises a device display, configured to display the plurality of screenshots, and wherein the interactive control is configured to receive the first particular input from within the device display.

17. The method of claim 13, further comprising:

setting one or more conditions for a communications network via the user interface, wherein the one or more conditions comprise a signal strength of the communications network.

18. The method of claim 13, wherein providing the plurality of screenshots using the user interface of the client device comprises:

assembling the plurality of the images into a video stream

providing the video stream using the user interface of the client device.

19. The method of claim 13, the operations further comprising:

receiving debug information associated with the execution of the one or more instructions to perform at least the first input event and the second input event; and

providing the debug information using the user interface of the client device.

20. The method of claim 13, wherein the user interface is configured to enable performing one or more tasks related to debugging a software application.