Virtual controller for touchscreen

• US 9,937,415 B1
• Filed: 12/17/2013
• Issued: 04/10/2018
• Est. Priority Date: 12/17/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Makuch, Jason David, Surishetty, Pavan Kumar, Hunter, Kimberly Nicole, Wang, Xin
• Primary Examiner(s): GALKA, LAWRENCE STEFAN
• Application Number:
  US14/108,654
• Time in Patent Office: 1,575 Days
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/77 involving data related to game devices or game servers, e.g. configuration data, software version or amount of memory
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Described are techniques and systems for providing a virtual controller user interface. The virtual controller user interface of a source media device accepts user input and provides control data based on that input to a target media device. The target media device may process the control data, injecting input events into an operating system of the target media device. An application executing on the target media device may use these inputs.

Claims

1. A system comprising a source device configured to accept user input and a target device responsive to the user input:

the target device comprising;

a first communication interface;

a first memory storing computer-executable instructions; and

a first hardware processor communicatively coupled to the first communication interface and the first memory, the first hardware processor configured to execute the computer-executable instructions to;

establish communication with the source device;

determine an application executing on the first hardware processor of the target device;

retrieve input configuration data for executing the application including input mapping data which associates control data received from the source device with first event data, wherein;

the control data comprises data indicative of particular user inputs made using input devices of the source device; and

the first event data comprises data indicative of particular commands associated with activation of a particular control on a physical controller;

send, to the source device, application data indicating the application executing on the target device and virtual controller data used by the source device to select a virtual controller user interface to present;

receive the control data, from the source device, wherein the control data is based on user inputs from one or more virtual controls of the virtual controller user interface presented on the source device based on the virtual controller data;

generate modified input mapping data associated with the virtual controller data used to select the virtual controller user interface, the modified input mapping data used by the target device to change association of the control data with second event data for providing to the application as input for executing the application;

use the modified input mapping data to generate the second event data from the control data instead of the first event data; and

store the second event data into an input file read by an operating system executing on the first hardware processor, wherein the operating system is configured to provide the second event data to the application executing on the first hardware processor.

2. The system of claim 1, wherein the source device comprises:

a second communication interface;

a second memory storing computer-executable instructions;

a touchscreen input device; and

a second hardware processor communicatively coupled to the second communication interface, the second memory, and the touchscreen input device, the second hardware processor configured to execute the computer-executable instructions to;

receive the application data;

receive the virtual controller data;

select the virtual controller user interface based at least in part on the application data and the virtual controller data;

present the virtual controller user interface on the touchscreen input device, the virtual controller user interface comprising the one or more virtual controls configured to simulate controls available on a physical controller;

accept the user inputs comprising data indicative of one or more touches to the touchscreen input device corresponding to the one or more virtual controls;

generate the control data based on the user inputs; and

send the control data to the target device using the second communication interface.

3. The system of claim 2, wherein:

the one or more virtual controls simulate one or more joystick controllers, and

the first event data is descriptive of input using the one or more joystick controllers.

4. The system of claim 1, further comprising a server, the server comprising:

a third communication interface;

a third memory storing computer-executable instructions; and

a third hardware processor communicatively coupled to the third communication interface and the third memory, the third hardware processor configured to execute the computer-executable instructions to;

access the application as provided for distribution to the target device;

determine one or more application inputs associated with the application, the determination comprising at least one or more of;

application of a plurality of pre-determined inputs to the application during execution,

generation of decompiled code of the application and analysis of the decompiled code,

inspection of a manifest file associated with the application, or

analysis of one or more user interface elements provided by the application during execution;

generate the input configuration data based on the one or more application inputs;

store the input configuration data;

determine the virtual controller data corresponding to the input configuration data; and

store information indicative of the corresponding virtual controller data.

5. A computer-implemented method executable on a source device, the method comprising:

receiving, at the source device, input configuration data for executing an application, the input configuration data including input mapping data which associates control data received with first event data;

receiving at the source device, application data determined by a target device and indicative of the application executing on the target device;

receiving at the source device, virtual controller data associated with the application, the virtual controller data associated with a virtual controller user interface presented by the source device, the virtual controller user interface including one or more virtual controls;

generating modified input mapping data for the virtual controller user interface associated with the virtual controller data, the modified input mapping data for changing association of the control data with second event data for providing to the application as input for executing the application;

receiving one or more user inputs by way of the virtual controller user interface;

generating the control data from the one or more user inputs based on the input configuration data and the virtual controller data;

using the modified input mapping data to generate the second event data from the control data instead of the first event data; and

sending the control data and the second event data to the application executing on the target device to use in response to the control data.

6. The computer-implemented method of claim 5, further comprising:

establishing communication with the target device and using one or more networks, wherein the one or more networks are compliant with at least a portion of one or more standards including Wi-Fi, Wi-Fi Direct, Bluetooth, or Bluetooth Low Energy.

7. The computer-implemented method of claim 5, further comprising:

generating log data indicative of usage of the virtual controller user interface, wherein the log data includes one or more of;

number of active target devices,

number of active source devices,

input latency data,

data communication statistics,

control session length data,

source device operating system version, or

target device operating system version; and

sending the log data to an external device.

8. The computer-implemented method of claim 5, wherein the one or more user inputs comprise coordinates corresponding to one or more touches on a touch sensor device;

and

the generating the control data comprises receiving association data representing the one or more of the touches on the touch sensor device with a particular one of the virtual controls on the virtual controller user interface.

9. The computer-implemented method of claim 5, wherein the generating the control data from the source device based on user inputs provided at the virtual controller user interface further comprises association data representing one or more touches on images of one or more of the virtual controls presented on a touchscreen, further wherein the images are configured to simulate a physical control of a physical controller.

10. The computer-implemented method of claim 9, wherein the virtual controls are representative of one or more of:

directional pads,

action buttons, or

joysticks.

11. A computer-implemented method executable on a target device, the method comprising:

determining that an application is executing on the target device;

sending, to a source device, application data indicating the executing application and virtual controller data used by the source device to select a virtual controller user interface;

retrieving input mapping data which associates control data received from the source device with first event data;

receiving, from the source device, the control data that is based on user inputs from one or more virtual controls of the virtual controller user interface presented on the source device based on the virtual controller data;

generating modified input mapping data associated with the virtual controller data used to select the virtual controller user interface, the modified input mapping data for changing association of the control data with second event data for providing to the application as input for executing the application;

generating the second event data from the control data using the modified input mapping data; and

storing the second event data at a memory location read by an operating system, wherein the second event data stored in the memory location is processed by the target device as input to the executing application.

12. The computer-implemented method of claim 11, further comprising:

establishing communication using one or more networks, wherein the one or more networks are compliant with at least a portion of one or more standards including Wi-Fi, Wi-Fi Direct, Bluetooth, or Bluetooth Low Energy.

13. The computer-implemented method of claim 11, further comprising:

sending, to the source device, the virtual controller data configured to present the virtual controller user interface on the source device.

14. The computer-implemented method of claim 11, wherein:

the receiving the control data comprises receiving information indicative of particular user input associated with an input device of the source device; and

the generating the modified input mapping data further comprises generating the modified input mapping data that associates the change of at least one of the control data to the second event data such that a particular user input on the input device of the source device results in particular event data.

15. The computer-implemented method of claim 11, wherein the storing the second event data at the memory location further comprises storing the second event data at the memory location read by a user level input subsystem of the operating system of the target device.

16. The computer-implemented method of claim 11, the method further comprising:

establishing communication between the device and the source device using one or more networks.

17. The computer-implemented method of claim 11, the determining the executing application comprising polling the operating system to obtain the application data indicative of the executing application.

18. The computer-implemented method of claim 11, the determining the executing application comprising establishing a controller wrapper configured to initiate execution of the application and generate information indicative of the initiation.

19. The computer-implemented method of claim 11, the method further comprising:

determining the application is no longer executing; and

sending, to the source device, the application data comprising information indicative that the application is no longer executing.

20. The computer-implemented method of claim 11, further comprising:

generating log data indicative of operation of the device; and

sending, to an external device, the log data.